Skip to content

feat(core): validate provider-specific API key formats - #64028

Open
RealThanhNguyxn wants to merge 12 commits into
anomalyco:devfrom
RealThanhNguyxn:feat/provider-aware-api-key-format-validation
Open

RealThanhNguyxn wants to merge 12 commits into
anomalyco:devfrom
RealThanhNguyxn:feat/provider-aware-api-key-format-validation

Conversation

@RealThanhNguyxn

@RealThanhNguyxn RealThanhNguyxn commented Nov 15, 2026 •

Copy link
Copy Markdown

Issue for this PR

Closes #64023

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Key methods can declare the prefixes a provider issues (validation.prefixes). connection.key trims the key, rejects an empty one, and rejects a key that matches none of the declared prefixes before anything is written, so App, TUI, CLI and SDK callers all get the same result. The check is local: no request is made and the key is never echoed in the error.

The catalog declares formats for Alibaba Model Studio, Alibaba Coding Plan (China), Anthropic, DeepSeek, Google (AQ. and AIza), OpenAI, Groq, Hugging Face, OpenRouter and xAI. Everything else stays opaque, including DigitalOcean, whose inference keys and account tokens share no prefix. A provider configured with a custom baseURL is exempt, because a gateway hands out its own tokens; removing the baseURL restores the provider's format.

Guarded by the api_key_format flag. With the flag off no format is declared and only the empty-key check applies.

How did you verify your code works?

From packages/core: bun run test (7607 pass, 350 skip, 0 fail). New tests run the real ModelsDevPlugin, XAIPlugin and ConfigProviderPlugin:

  • flag on declares the formats and rejects a malformed key; flag off declares none and accepts it
  • a custom baseURL accepts a gateway token for that provider only, and removing it rejects the token again
  • a rejected key leaves the stored credential untouched

bun run check passes from the repo root. packages/protocol and packages/client were regenerated.

Not verified: a live key against any provider.

Screenshots / recordings

Not applicable. No UI changes.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@github-actions github-actions Bot added the needs:compliance This means the issue will auto-close after 2 hours. label Nov 15, 2026
@RealThanhNguyxn
RealThanhNguyxn marked this pull request as draft November 15, 2026 05:57

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works in core, but it isn't ready yet: the test it adds fails, and the Anthropic prefix can block users who route Anthropic through a proxy.

How I checked it: I ran bun test test/integration.test.ts in packages/core (16 pass, 1 fail: the new test) and tsc -b in packages/core (passes). I couldn't run the connect flow end to end here.

Public API: this adds an optional validation?: { prefix: string } to Integration.KeyMethod in packages/schema, which also appears in the generated HTTP and OpenAPI schema for integration methods. It adds the same field to IntegrationKeyMethod in the plugin API. Behaviour also changes: integration.connect.key now rejects empty or whitespace-only keys for every integration, and keys without sk-ant- for anthropic. Existing callers and plugins still compile, since the field is optional.

Concerns:

  • The new test fails because of an error-message mismatch.
  • The anthropic prefix also applies when provider.anthropic points at a custom baseURL.
  • The prefix check runs on the untrimmed key, so a key pasted with surrounding whitespace is rejected (details inline).
  • The branch includes a commit that adds a docs plan and a later one that deletes it. Please squash those away.
  • Note that #64023 hasn't been accepted by a maintainer yet, so whether the schema and plugin API should gain this field is still open.

Related: #56782

Comment thread packages/core/test/integration.test.ts Outdated
const invalid = yield* integrations.connection
.key({ integrationID: constrainedID, key: "1" })
.pipe(Effect.flip)
expect(invalid.cause).toMatchObject({ message: 'API key must start with "sk-ant-".' })

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assertion fails: the code throws API key must start with "sk-ant-" and include a value after the prefix., so bun test test/integration.test.ts reports this test as failing. Please run the test and align the message, either here or in integration.ts.

Comment thread packages/core/src/plugin/models-dev.ts Outdated

// Only enforce formats that the provider documents as stable. Other integrations stay opaque.
const API_KEY_VALIDATION: Record<string, Integration.KeyMethod["validation"]> = {
anthropic: { prefix: "sk-ant-" },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This applies to every anthropic key connection, including users who set provider.anthropic in config with a custom baseURL (LiteLLM, corporate gateways and other proxies). The config provider plugin only adds its own unconstrained key method when the integration doesn't exist yet, so those users keep this method and can no longer save their proxy's key. Could the constraint be skipped when the provider's endpoint is overridden, or kept out of core until maintainers agree on the approach in #64023?

Comment thread packages/core/src/integration.ts Outdated
if (!input.key.trim()) {
return yield* new AuthorizationError({ cause: new Error("API key cannot be empty.") })
}
const prefix = method.validation?.prefix

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The empty check trims the key, but the prefix check and the stored value use the raw input.key. A key pasted with a leading space or a trailing newline is rejected with a misleading "must start with" error, and on unconstrained providers it's stored with the whitespace. Trimming once and using that value for both the check and the stored credential would avoid this.

@RealThanhNguyxn
RealThanhNguyxn marked this pull request as ready for review November 15, 2026 06:37
@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for updating your PR! It now meets our contributing guidelines. 👍

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new commits fix the failing test: bun test test/integration.test.ts now passes, and tsc -b in packages/core passes. The prefix list has grown from one provider to eleven, and two of the new entries can stop users from saving keys that work. I couldn't run the connect flow end to end here.

Public API: Integration.KeyMethod.validation (schema/OpenAPI) and the plugin API's IntegrationKeyMethod.validation change from { prefix: string } to { prefixes: string[] }. The field hasn't been released yet, so existing callers are unaffected. integration.connect.key now also rejects keys without the declared prefix for alibaba, alibaba-coding-plan-cn, deepseek, digitalocean, google, openai, groq, huggingface, openrouter and xai.

Concerns:

  • The google entry rejects AIza keys, which still work.
  • The openai entry makes the custom baseURL/proxy problem from the earlier thread worse.
  • The whitespace thread is still open.
  • The test grew to an 84-line table with nested loops and special cases for some providers, and it rewrites the prefixes itself instead of using the real declarations. A wrong entry in models-dev.ts would still pass. Please cut it down to one constrained provider (valid key, malformed key, prefix only), the empty case and one unconstrained provider.

Comment thread packages/core/src/plugin/models-dev.ts Outdated
anthropic: { prefixes: ["sk-ant-"] },
deepseek: { prefixes: ["sk-"] },
digitalocean: { prefixes: ["dop_v1_", "doo_v1_"] },
google: { prefixes: ["AQ."] },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only allowing AQ. rejects AIza… Gemini keys, and those still work. Google's forum posts say AI Studio now issues AQ. keys, but Cloud Console can still create AIza keys, and restricted ones are still accepted. Existing Gemini users couldn't reconnect, and the test explicitly asserts that AIzaSy… is rejected. Please accept both prefixes, or leave google out.

openai is often configured with a custom baseURL (Azure-style gateways, LiteLLM, corporate proxies, local OpenAI-compatible servers), and those keys usually don't start with sk-. Because of the earlier anthropic problem (the config provider keeps this key method when the integration already exists), all of those users would now be unable to connect. Can the constraint be skipped when the provider's endpoint is overridden?

Comment thread packages/core/test/integration.test.ts Outdated
Effect.gen(function* () {
const integrations = yield* Integration.Service
const credentials = yield* Credential.Service
const providers = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table restates every provider's prefixes instead of reading the real declarations in models-dev.ts. It mostly checks that startsWith works, and it passes even if a declared prefix is wrong, as with google. The nested loops and the special cases for digitalocean and google also make it hard to follow. Please cut it down to one constrained method (a valid key, 1, and the prefix alone), the empty case and the unconstrained case.

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new commit addresses the open points:

  • Custom baseURL: the prefix check is turned off for that provider. I checked this with a throwaway test against ConfigProviderPlugin: proxy-token is rejected for anthropic without a baseURL and accepted with one.
  • Whitespace: the key is trimmed once and the trimmed value is both checked and saved.
  • Google: both AQ. and AIza keys are accepted.
  • Test: it's down to the core cases.

bun test passes for test/integration.test.ts (17/17) and test/config/provider.test.ts (44/44), and tsc -b in packages/core passes. I didn't run the connect flow end to end.

Public API:

  • Adds optional validation?: { prefixes: string[] } to Integration.KeyMethod (schema and OpenAPI) and to the plugin API's IntegrationKeyMethod. Existing callers and plugins still work.
  • Behaviour change: integration.connect.key now trims keys and rejects empty ones for every integration.
  • It also rejects keys without the declared prefix for alibaba, alibaba-coding-plan-cn, anthropic, deepseek, digitalocean, google, openai, groq, huggingface, openrouter and xai, unless the provider has a custom baseURL in config.

I found no other blocking problems. Maintainers still need to decide on #64023, including which providers should get a prefix list.

Declare key formats only while the api_key_format flag is on, and redeclare when it changes.
Drop the DigitalOcean entry: its inference keys and account tokens share no prefix.
Keep the catalog's declared format when the xAI plugin relabels the key method.
Cover the catalog declarations and the custom baseURL exemption with the real plugins, and regenerate the OpenAPI document and client types.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Add provider-aware API key format validation before saving credentials

1 participant