Repository navigation
feat(core): validate provider-specific API key formats - #64028
RealThanhNguyxn wants to merge 12 commits into
Conversation
There was a problem hiding this comment.
This works in core, but it isn't ready yet: the test it adds fails, and the Anthropic prefix can block users who route Anthropic through a proxy.
How I checked it: I ran bun test test/integration.test.ts in packages/core (16 pass, 1 fail: the new test) and tsc -b in packages/core (passes). I couldn't run the connect flow end to end here.
Public API: this adds an optional validation?: { prefix: string } to Integration.KeyMethod in packages/schema, which also appears in the generated HTTP and OpenAPI schema for integration methods. It adds the same field to IntegrationKeyMethod in the plugin API. Behaviour also changes: integration.connect.key now rejects empty or whitespace-only keys for every integration, and keys without sk-ant- for anthropic. Existing callers and plugins still compile, since the field is optional.
Concerns:
- The new test fails because of an error-message mismatch.
- The
anthropicprefix also applies whenprovider.anthropicpoints at a custombaseURL. - The prefix check runs on the untrimmed key, so a key pasted with surrounding whitespace is rejected (details inline).
- The branch includes a commit that adds a docs plan and a later one that deletes it. Please squash those away.
- Note that #64023 hasn't been accepted by a maintainer yet, so whether the schema and plugin API should gain this field is still open.
Related: #56782
| const invalid = yield* integrations.connection | ||
| .key({ integrationID: constrainedID, key: "1" }) | ||
| .pipe(Effect.flip) | ||
| expect(invalid.cause).toMatchObject({ message: 'API key must start with "sk-ant-".' }) |
There was a problem hiding this comment.
This assertion fails: the code throws API key must start with "sk-ant-" and include a value after the prefix., so bun test test/integration.test.ts reports this test as failing. Please run the test and align the message, either here or in integration.ts.
|
|
||
| // Only enforce formats that the provider documents as stable. Other integrations stay opaque. | ||
| const API_KEY_VALIDATION: Record<string, Integration.KeyMethod["validation"]> = { | ||
| anthropic: { prefix: "sk-ant-" }, |
There was a problem hiding this comment.
This applies to every anthropic key connection, including users who set provider.anthropic in config with a custom baseURL (LiteLLM, corporate gateways and other proxies). The config provider plugin only adds its own unconstrained key method when the integration doesn't exist yet, so those users keep this method and can no longer save their proxy's key. Could the constraint be skipped when the provider's endpoint is overridden, or kept out of core until maintainers agree on the approach in #64023?
| if (!input.key.trim()) { | ||
| return yield* new AuthorizationError({ cause: new Error("API key cannot be empty.") }) | ||
| } | ||
| const prefix = method.validation?.prefix |
There was a problem hiding this comment.
The empty check trims the key, but the prefix check and the stored value use the raw input.key. A key pasted with a leading space or a trailing newline is rejected with a misleading "must start with" error, and on unconstrained providers it's stored with the whitespace. Trimming once and using that value for both the check and the stored credential would avoid this.
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
There was a problem hiding this comment.
The new commits fix the failing test: bun test test/integration.test.ts now passes, and tsc -b in packages/core passes. The prefix list has grown from one provider to eleven, and two of the new entries can stop users from saving keys that work. I couldn't run the connect flow end to end here.
Public API: Integration.KeyMethod.validation (schema/OpenAPI) and the plugin API's IntegrationKeyMethod.validation change from { prefix: string } to { prefixes: string[] }. The field hasn't been released yet, so existing callers are unaffected. integration.connect.key now also rejects keys without the declared prefix for alibaba, alibaba-coding-plan-cn, deepseek, digitalocean, google, openai, groq, huggingface, openrouter and xai.
Concerns:
- The
googleentry rejectsAIzakeys, which still work. - The
openaientry makes the custombaseURL/proxy problem from the earlier thread worse. - The whitespace thread is still open.
- The test grew to an 84-line table with nested loops and special cases for some providers, and it rewrites the prefixes itself instead of using the real declarations. A wrong entry in
models-dev.tswould still pass. Please cut it down to one constrained provider (valid key, malformed key, prefix only), the empty case and one unconstrained provider.
| anthropic: { prefixes: ["sk-ant-"] }, | ||
| deepseek: { prefixes: ["sk-"] }, | ||
| digitalocean: { prefixes: ["dop_v1_", "doo_v1_"] }, | ||
| google: { prefixes: ["AQ."] }, |
There was a problem hiding this comment.
Only allowing AQ. rejects AIza… Gemini keys, and those still work. Google's forum posts say AI Studio now issues AQ. keys, but Cloud Console can still create AIza keys, and restricted ones are still accepted. Existing Gemini users couldn't reconnect, and the test explicitly asserts that AIzaSy… is rejected. Please accept both prefixes, or leave google out.
openai is often configured with a custom baseURL (Azure-style gateways, LiteLLM, corporate proxies, local OpenAI-compatible servers), and those keys usually don't start with sk-. Because of the earlier anthropic problem (the config provider keeps this key method when the integration already exists), all of those users would now be unable to connect. Can the constraint be skipped when the provider's endpoint is overridden?
| Effect.gen(function* () { | ||
| const integrations = yield* Integration.Service | ||
| const credentials = yield* Credential.Service | ||
| const providers = [ |
There was a problem hiding this comment.
This table restates every provider's prefixes instead of reading the real declarations in models-dev.ts. It mostly checks that startsWith works, and it passes even if a declared prefix is wrong, as with google. The nested loops and the special cases for digitalocean and google also make it hard to follow. Please cut it down to one constrained method (a valid key, 1, and the prefix alone), the empty case and the unconstrained case.
There was a problem hiding this comment.
The new commit addresses the open points:
- Custom
baseURL: the prefix check is turned off for that provider. I checked this with a throwaway test againstConfigProviderPlugin:proxy-tokenis rejected foranthropicwithout abaseURLand accepted with one. - Whitespace: the key is trimmed once and the trimmed value is both checked and saved.
- Google: both
AQ.andAIzakeys are accepted. - Test: it's down to the core cases.
bun test passes for test/integration.test.ts (17/17) and test/config/provider.test.ts (44/44), and tsc -b in packages/core passes. I didn't run the connect flow end to end.
Public API:
- Adds optional
validation?: { prefixes: string[] }toIntegration.KeyMethod(schema and OpenAPI) and to the plugin API'sIntegrationKeyMethod. Existing callers and plugins still work. - Behaviour change:
integration.connect.keynow trims keys and rejects empty ones for every integration. - It also rejects keys without the declared prefix for alibaba, alibaba-coding-plan-cn, anthropic, deepseek, digitalocean, google, openai, groq, huggingface, openrouter and xai, unless the provider has a custom
baseURLin config.
I found no other blocking problems. Maintainers still need to decide on #64023, including which providers should get a prefix list.
…i-key-format-validation
Declare key formats only while the api_key_format flag is on, and redeclare when it changes. Drop the DigitalOcean entry: its inference keys and account tokens share no prefix. Keep the catalog's declared format when the xAI plugin relabels the key method. Cover the catalog declarations and the custom baseURL exemption with the real plugins, and regenerate the OpenAPI document and client types.
Issue for this PR
Closes #64023
Type of change
What does this PR do?
Key methods can declare the prefixes a provider issues (
validation.prefixes).connection.keytrims the key, rejects an empty one, and rejects a key that matches none of the declared prefixes before anything is written, so App, TUI, CLI and SDK callers all get the same result. The check is local: no request is made and the key is never echoed in the error.The catalog declares formats for Alibaba Model Studio, Alibaba Coding Plan (China), Anthropic, DeepSeek, Google (
AQ.andAIza), OpenAI, Groq, Hugging Face, OpenRouter and xAI. Everything else stays opaque, including DigitalOcean, whose inference keys and account tokens share no prefix. A provider configured with a custombaseURLis exempt, because a gateway hands out its own tokens; removing thebaseURLrestores the provider's format.Guarded by the
api_key_formatflag. With the flag off no format is declared and only the empty-key check applies.How did you verify your code works?
From
packages/core:bun run test(7607 pass, 350 skip, 0 fail). New tests run the realModelsDevPlugin,XAIPluginandConfigProviderPlugin:baseURLaccepts a gateway token for that provider only, and removing it rejects the token againbun run checkpasses from the repo root.packages/protocolandpackages/clientwere regenerated.Not verified: a live key against any provider.
Screenshots / recordings
Not applicable. No UI changes.
Checklist