Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 64 additions & 8 deletions memory-bank/aiConfig.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,16 +294,21 @@ the built-in one; a `"user"` or `"default"` source yields to the stored endpoint
`resolveModels(modelsBlock, discovered, providerConnection, context?)` runs the
per-provider model pipeline: discovery gate (`discovery: "auto" | "off"`) → merge
discovered + `custom` models → apply `overrides` → `allow` filter (exclusive
allowlist) → `deny` filter (always wins) → attach routing (protocol/baseUrl). It
allowlist) → `deny` filter (always wins) → attach routing (protocol/baseUrl) →
finalize web-search capability (when a serving context is passed; see below). It
is pure and reusable independent of the catalog builder.

The optional fourth argument, `ModelResolutionContext { providerId }`, supplies
the provider identity `ResolvedConnection` deliberately lacks. The built-in
`opencode` provider uses it to recompute its product-dependent inferred protocol
under the full routing context — the discovery-time stamp knew only the provider
URL, while a canonical model-level URL override can point at the other OpenCode
product and change the documented route (see the OpenCode Protocol Routing
section below).
The optional fourth argument, `ModelResolutionContext`, supplies provider-level
context `ResolvedConnection` deliberately lacks:

- `providerId` — the built-in `opencode` provider uses it to recompute its
product-dependent inferred protocol under the full routing context — the
discovery-time stamp knew only the provider URL, while a canonical model-level
URL override can point at the other OpenCode product and change the documented
route (see the OpenCode Protocol Routing section below).
- `webSearchServing` — the provider's web-search serving context (from
`resolveWebSearchServing`). When present, each surviving model's final
`supportsWebSearch` is computed after overrides and routing are resolved.

Capacity overrides (`maxContextLength`, `maxInputTokens`, and
`maxOutputTokens`) replace the corresponding values in both the resolved flat
Expand Down Expand Up @@ -599,6 +604,56 @@ here, with three improvements over that copy: the `openai` case now derives
the DeepSeek table, and `supportsImages` is derived from media types as
described above).

### Web-search capability finalization (`web-search.ts`)

Discovery and the capability tables report _intrinsic_ model metadata (e.g.
the Mantle family rule marks documented GPT-5.4/5.5/5.6 IDs
`supportsWebSearch: true`). Whether a model may actually advertise the hosted
web-search toggle depends on the resolved serving context — effective route,
effective endpoint, provider kind, AWS region, FIPS policy — which only
exists after `resolveModels()` has applied overrides and routing.
`web-search.ts` owns that final step so every host computes the same final
`supportsWebSearch` from the same inputs:

- `resolveWebSearchServing(provider, facts)` builds the serving context
(`openai-builtin`/`gemini-builtin` with the client base URL,
`openai-custom`, `gemini-custom`, or `bedrock-mantle` with region and FIPS
flag), or `undefined` for providers outside the policy, whose models keep
their already-resolved capability unchanged. `facts`
(`WebSearchServingFacts`) must come from the host's _effective_
credentials — the catalog connection overlaid on the stored credential,
exactly what the client is built with — because a stored base URL, AWS
region, or profile the catalog lacks still changes where requests go. The
finalizer checks the endpoint as `model.resolvedBaseUrl ?? clientBaseUrl`,
matching send-time precedence. The async AWS FIPS read is the host's job
(the bridge's `resolveBedrockTransport`); an unknown FIPS flag fails
closed.
- `finalizeWebSearchCapability(model, explicit, serving)` computes the final
value. The explicit `supportsWebSearch` override is kept separate from the
discovered value until this step, so a deliberate opt-in or opt-out
survives finalization.

The policy rules, in order:

1. A non-Responses effective route never has hosted search on OpenAI or
Bedrock, regardless of any override. (OpenAI client kinds treat an
unresolved protocol as the Responses route — the provider factory builds
`OpenAIClient` in Responses mode; Bedrock treats it as the
Converse/Anthropic heuristic route, no search.)
2. Built-in OpenAI on its canonical Responses endpoint defaults to `true`;
an explicit `false` wins.
3. A redirected built-in OpenAI endpoint defaults to `false`; an explicit
`true` may opt in when the gateway genuinely serves Responses search.
4. A custom OpenAI provider defaults to `false` and requires explicit `true`
plus Responses routing.
5. Bedrock requires a documented Mantle GPT family (re-derived from the
model ID via `getBedrockMantleModelCapabilities`, so an explicit `true`
cannot manufacture eligibility for gpt-oss or unknown families), the
Mantle Responses route, a web-search-enabled region (`us-east-1`,
`us-east-2`, `us-west-2` — deliberately narrower than Mantle inference
availability), and no FIPS veto. An explicit `false` disables it; an
explicit `true` cannot bypass the service gates.

### Databricks Native Routing (`databricks-helpers.ts`, `gemini-generate-content.ts`)

Databricks fronts many vendors behind one workspace and exposes native
Expand Down Expand Up @@ -847,6 +902,7 @@ the bridge's `ModelInfo` — compatible by contract, not by import.
| `src/resolve-models.ts` | `resolveModels()` model selection + routing pipeline |
| `src/model-capabilities/*-helpers.ts` | Per-provider capability tables (moved from the bridge, ai-lib#9); `gpt6-model-profile.ts` holds shared GPT-6 traits |
| `src/model-capabilities/infer.ts` | `inferModelCapabilities()` — baseline + provider-family merge, Snowflake protocol rule |
| `src/model-capabilities/web-search.ts` | `resolveWebSearchServing()` / `finalizeWebSearchCapability()` — hosted web-search capability finalization over the resolved serving context |
| `src/index.ts` | Pure entrypoint exports |
| `src/node/paths.ts` | `AI_CONFIG_DIR`, `PROVIDERS_CONFIG_PATH`, enforced env-var name, lockfile path |
| `src/node/types.ts` | Node seam option/result types (`LoadCatalogOptions`, `ProviderCatalogChange`, `Disposable`, …) |
Expand Down
45 changes: 45 additions & 0 deletions memory-bank/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,51 @@ permissions are not needed after subscription. AWS's
zero-retention guarantee: AWS account/project retention policy is configured
separately and may still retain data according to that policy.

### Hosted web search on the OpenAI Responses transports

Direct OpenAI Responses and Bedrock Mantle Responses share one hosted-tool
seam: `mergeOpenAIWebSearchTool()` (`src/model-clients/provider-tools.ts`)
attaches the SDK's `openai.tools.webSearch(...)` provider tool under the
fixed `web_search` key, merging through `mergeProviderTools()` so a local
tool occupying that key rejects the request instead of being silently
overwritten. Both routes run the AI SDK's `OpenAIResponsesLanguageModel`, so
the same provider tool serializes correctly on either, and callers derive
`toolChoice` from the single merged record the helper returns. (The SDK
factory's return type is re-wrapped into this repo's `ai.Tool` because
`@ai-sdk/openai` and `ai` pin different `@ai-sdk/provider-utils` copies whose
unique-symbol schema brands are nominally incompatible; provider tools
serialize from `id`/`args` alone, so the re-wrap loses nothing.)

Each client vetoes an incompatible route before any network call —
`OpenAIClient` rejects `webSearchEnabled` unless the effective mode is
Responses, and `BedrockClient` rejects it unless the resolved protocol is
`openai-responses` (Mantle Chat, Converse, and Anthropic Messages carry no
hosted search). The veto is defense in depth for non-host consumers; whether
a model may advertise the toggle at all is decided upstream, at ai-config's
capability finalization seam (see `memory-bank/aiConfig.md` → "Web-search
capability finalization").

Mantle search has its own IAM namespace and egress boundary:

- Retrieval needs `bedrock-websearch:InvokeSearch` and
`bedrock-websearch:InvokeFetch` in addition to the Mantle inference
permissions above. The commonly used `AmazonBedrockFullAccess` policy
grants Search and Fetch but **not** external access. PA never probes these
actions during discovery — permission varies by identity and the invocation
error is authoritative.
- `external_web_access` controls whether a cache-miss Fetch may reach the
live external page; when off, retrieval is restricted to AWS's own web
index and cache. AWS defaults an omitted flag to `true`, which would widen
the egress boundary, so `BedrockClient` always serializes an explicit
boolean from the optional `ModelClientChatParams.bedrockExternalWebAccess`
(omission defaults to `false`). Enabling it additionally requires the
`bedrock-websearch:ExternalWebAccess` IAM action. Direct OpenAI omits the
option and keeps OpenAI's own defaults.
- AWS documents web search only for specific Mantle GPT model families and a
region allowlist narrower than Mantle inference availability; both gates
are applied at capability finalization, and FIPS vetoes the route entirely
(AWS publishes no Mantle FIPS endpoint).

### Databricks: pinned surface + native protocol dispatch

Databricks fronts many vendors behind one workspace on one of two surfaces —
Expand Down
69 changes: 68 additions & 1 deletion memory-bank/geminiInteractions.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Gemini Interactions API
description: Stateful chaining design, unsigned-reasoning filtering, the raw-usage metadata hoist, and known API gotchas for the Gemini Interactions path, contrasted with the stateless generateContent client used by gateways like Databricks.
description: Stateful chaining design, unsigned-reasoning filtering, server-side Google Search grounding, the raw-usage metadata hoist, and known API gotchas for the Gemini Interactions path, contrasted with the stateless generateContent client used by gateways like Databricks.
package: ai-provider-bridge
---

Expand Down Expand Up @@ -33,6 +33,7 @@ The two clients differ at every point their host surface differs:
| Retry | `withExpiredIdRetry()` retries once on expired-interaction errors | No retry path — there is no interaction ID to expire |
| Thinking control | `thinkingLevel` mapped from the product-level effort via per-model `INTERACTIONS_PROFILES` (`effortToWireLevel`), top-level option | `thinkingConfig`: numeric `thinkingBudget` for Gemini 2.5 variants, categorical `thinkingLevel` for 3.x, derived from ai-config's `getGeminiGenerateContentProfile` at variant granularity |
| Auth | SDK-native | `apiKey` mode uses the SDK's native `x-goog-api-key`; `authToken` (bearer gateways) uses a fetch middleware that sets `Authorization: Bearer` and strips `x-goog-api-key` |
| Search grounding | Supported — attaches the provider-defined `google_search` tool when `webSearchEnabled` (see below) | Not implemented — sends local tools only |

`sanitizeGenerateContentHistory()` is deliberately **not** a reuse of
`filterUnsignedReasoning()`: the two APIs key signed reasoning on different
Expand Down Expand Up @@ -120,6 +121,55 @@ Gemini 3.6/3.7 Flash were invisible until manually allowlisted):
`thinkingEffortLevels` (`buildGeminiModel` strips them) and the client
sends no `thinkingLevel`/`thinkingSummaries`.

## Server-Side Web Search (Google Search Grounding)

`GeminiClient.chat()` honors the per-request `webSearchEnabled` param by
attaching the SDK's provider-defined Search tool
(`provider.tools.googleSearch({})`) under the fixed `google_search` key —
the Interactions API treats that name as a tool _type_, so it cannot be
renamed. On the wire this serializes to `tools: [{ type: "google_search" }]`
on `POST /v1beta/interactions`.

- **The merge goes through `mergeProviderTools()`** (`provider-tools.ts`),
which rejects the request when a local tool already occupies the
`google_search` key rather than silently overwriting it. The same helper
serves Anthropic's `web_search`.
- **The toolset widening is absorbed inside the client.** The public
`ModelClientChatParams.tools` contract stays narrow (local
`AiToolWithJsonSchema` tools only); the merged record is wider because a
provider-defined tool has no JSON-schema input. The merged toolset is part
of the `streamArgs` the expired-interaction retry is built from, so the tool
survives the retry unchanged.
- **The client takes eligibility as a per-request flag only** — it does not
re-check model IDs. Advertisement is the catalog's job: `buildGeminiModel`
sets `supportsWebSearch` from `googleHosted && isGeminiWebSearchVerified(id)`.
Like thinking profiles, the verified list (`WEB_SEARCH_VERIFIED_MODELS` in
ai-config's `gemini-api-helpers.ts`) is **fail-closed** against fail-open
discovery: an unverified future model gets no toggle. ai-config's
`finalizeWebSearchCapability` re-derives the same predicate from the model
ID, so a `supportsWebSearch: true` model override cannot enable it. Gemini 2.5 is deliberately
excluded — its grounding bills per _prompt_ (~$35 vs ~$14 per 1,000), and
it rejects built-in Search combined with function tools (verified live
2026-09-05), which PA always sends. Custom Gemini endpoints and Vertex
stay `false`; ai-config's capability table stays provider-neutral.
- **`tool_choice` behaves differently search-only vs. search-plus-tools.**
The SDK drops `tool_choice` when the resolved tool list contains no
`function` tools (the Interactions API rejects it otherwise), so a
search-only request sends none; with local tools present the client sends
`"auto"`.
- **Stream shape** (verified live 2026-09-05 against `gemini-3.8-flash`):
the SDK emits the `google_search` tool-call with `providerExecuted: true`,
then the tool-result, then reasoning/text, with URL `source` parts arising
both from the result block (pre-text, when result entries carry `url`) and
from `text_annotation_delta` citations (post-text). Core owns rendering:
ownership follows the `providerExecuted` flag, not the tool name, and core
buffers pre-text sources until the first text delta.
- **Known limitation:** the pinned `@ai-sdk/google` records `is_error` on a
built-in result but omits it when emitting the tool-result stream part, so
a failed Search renders downstream as an empty success. Closing this needs
an upstream SDK fix; the bridge only ever sees AI SDK stream parts and
cannot recover the bit.

## Raw Usage Metadata Hoist

The SDK's interactions provider reports token usage only as `usage.raw` on
Expand Down Expand Up @@ -150,6 +200,23 @@ errors. On retry, it resends the full signature-filtered history with no
`previousInteractionId` (fresh interaction). The replacement `interactionId` persists
via the normal finish-metadata path.

- **The rejection arrives as an in-stream `error` part, not a throw.** When the
request fails (`doStream` rejects with an `APICallError`), the AI SDK emits
its leading `start` part and then an `error` part. The wrapper consumes that
part and retries; a thrown error is handled the same way.
- **Retry only while chaining and before content.** The retry fires only when
the request carried a `previousInteractionId` and nothing but `start` has been
yielded. An expired-interaction error after content passes through unchanged,
so the consumer never sees spliced partial output.
- **The retry request is built in `chat()`** from the same `streamArgs` as the
first attempt (only `messages` and `providerOptions` differ) and handed to the
wrapper as a callback, so every other argument — tools, system prompt,
`allowSystemInMessages`, `onError`, abort signal — cannot drift between the
two attempts. The retry's duplicate `start` part is dropped.
- `streamArgs` sets `onError: suppressAiSdkDefaultErrorLogging`, so a recovered
rejection does not reach stderr through the SDK's default `console.error`;
the `[GeminiClient] stream error part` diagnostic goes through the logger.

## Error Diagnostics

- `serializeGeminiError()` handles two error shapes: `APICallError` (thrown/pre-stream)
Expand Down
Loading
Loading