graphql: add discovery APIs

[jshearer]

Description:

Add createDiscover and discover(id) to GraphQL. Discovery uses the capture staged in an owned draft, or a readable live capture. Submission atomically queues the existing discovery executor with the selected inputs. The executor prepares its capture from the draft or live state and persists discovery results.

Workflow steps:

1. Create a draft. Stage an initial capture with bindings: [], or use an existing live capture.
2. Call createDiscover(draftId, captureName, dataPlane); the optional data plane must match an existing capture's data plane or be permitted by the new capture's storage mapping.
3. Poll discover(id) until status leaves QUEUED. Read logs through discover(id).logs, and inspect the resulting definitions through draft(id).specs.
4. Review and publish the draft separately.

Implementation Plan

This plan adds createDiscover and discover(id) to the GraphQL API in control-plane-api, with fields for status, errors, and logs. A discover is an asynchronous job that runs discovery on a capture definition and merges the resulting capture and collection definitions into a draft. It does not publish the draft.

The mutation uses the capture definition in the draft if one exists. Otherwise, it reads the live capture of that name, provided the caller can read it. If neither definition is available, the mutation fails. One statement inserts the discover row and its executor task through the existing database trigger. Submission leaves the draft unchanged. Clients poll for the result.

The discovers table and its row-level security policies do not change. Existing PostgREST clients can continue to use them.

What this builds on

Discovery asks the connector for potential bindings. The executor uses the discovered bindings to update the drafted capture's bindings and create or update target collection in the draft. It records the job's outcome in discovers.

The new API uses this existing process:

• The executor prepares a capture from the draft, a readable live capture, or an initial definition that it constructs. It replaces the endpoint's image and configuration with values from the discover row. It also takes update_only from that row. GraphQL requires a staged or readable live definition at submission. PostgREST callers retain all three preparation paths.
• The capture model carries secret references and the redaction salt. This API adds no separate inputs for them and does not change secret resolution.
• Before running the connector, the executor checks the connector tag, SpecEdit on the capture name, and access to the data plane. Access to the data plane requires legacy read, which includes every Viewer capability. The executor also needs the data plane's first configured HMAC key to sign requests to its connector proxy.
• A discover belongs to its draft's owner. The table has no user column, and deleting a draft deletes its discovers. Draft ownership therefore controls reads of a discover.
• The log writer associates connector logs with the row's logs_token. Discover.logs uses that token internally.

Capture configuration and discovery

CaptureDef combines two concerns today. Its endpoint, bindings, and runtime settings describe the capture itself. Its autoDiscover settings tell the control plane how discovery should modify that capture and its target collections.

The name autoDiscover makes the second concern sound exclusive to automatic discovery. The field's presence enables periodic discovery, but its flags also affect manually requested discovery. For example, manual discovery derives update_only from addNewBindings, and the discover executor uses evolveIncompatibleCollections when collection keys change. The field therefore combines whether discovery runs automatically with policy for applying discovery results.

The policy discussion considered letting manual discovery use different settings from the capture's ongoing policy. A caller might want to enable new bindings during one manual discover while leaving automatic addition disabled. We chose to keep discovery policy on the capture and have discovers use it. This follows the broader decision to treat discovery as an operation applied to a capture. It avoids a second set of endpoint, secret, and policy inputs that would duplicate parts of CaptureDef on each discover request.

This API therefore uses the selected capture's configuration and autoDiscover settings, with the defaults described below. The mutation accepts no separate configuration arguments or policy overrides. Clients that want a different discovery policy must change it in the draft's capture definition.

Renaming or aliasing autoDiscover to discovery would express this broader role: policy for discovery, whether initiated manually or automatically. The policy would remain part of CaptureDef. This API adopts that understanding while retaining the current field name and defaults. The rename or alias can follow separately.

Here, a new capture means one with no live definition, even if the draft already contains a definition.

createDiscover takes an existing draft, a capture name, and an optional data plane for this operation. It selects the definition as follows:

1. If an entry exists under that name in the draft, it must contain a capture model that deserializes as CaptureDef. A deletion, another catalog type, or a model that fails to deserialize is an error.
2. If no entry exists under that name, the mutation uses a readable live capture of the same name. Other entries in the draft do not prevent the mutation from using the live capture.
3. If neither definition is available, the mutation returns an error.

For a new capture, the client stages an initial definition with stageDraftSpecs before requesting discovery. The definition must include bindings, which may be []. For an existing capture, the client can stage edits or submit using the readable live definition. The executor persists the resulting capture after successful discovery; failed discovery does not seed a missing capture in the draft.

createDiscover derives connector_tag_id, endpoint_config, and update_only from the selected definition. The existing executor needs these columns. PostgREST clients continue to supply them directly. This API adds no columns for capture fields and does not store a complete copy of the job's inputs. Clients read the current capture and collection definitions through the draft API.

Draft contents after submission

Discover describes the job. Clients read the current definitions through the specs connection on draft(id).

When the executor prepares a capture from live state, it uses the live capture's last publication ID as expectPubId. A later publication can then detect intervening changes to that live baseline. Submission leaves existing draft entries and their publication preconditions unchanged.

The discover row fixes the endpoint and update_only at submission. The executor uses these values even if the client subsequently edits the draft. It reads the other fields, including autoDiscover.evolveIncompatibleCollections and secret references, when it loads the draft. A successful discover writes its merged definitions into the draft, replacing any intervening endpoint edits.

The executor selects its base definition when preparing execution: a staged capture takes precedence over live state. It retains its existing behavior if it finds neither definition, including constructing an initial definition when necessary.

The draft is not a historical record of the inputs to a discover. Concurrent edits and jobs can overwrite each other's changes. This API does not serialize those operations. Clients should wait for a discover to finish before editing its draft or starting another operation on it.

Client migration

Existing PostgREST clients can still request discovery without a draft entry or a live capture. The executor constructs an initial capture definition from the discover row. GraphQL clients must stage a new capture first. The UI migration must add this step wherever the UI currently relies on the executor to construct the capture.

The UI must supply its intended autoDiscover settings in that definition. The executor currently sets both flags to true when it constructs a capture. Clients that need those defaults must set them explicitly. Staging only an endpoint and empty bindings does not reproduce those settings.

When the executor constructs a capture, it also sets the draft entry's expectPubId to zero. Publication therefore fails if someone creates a live capture of that name in the meantime. To retain that protection, the client must set expectPubId to zero (0000000000000000) each time it stages the new capture. Omitting this field or passing null clears any previous precondition.

When a user edits a capture and re-enables disabled capture bindings, the UI currently forces update_only for that discover. Through this API, the capture's autoDiscover settings determine the behavior instead. The merge adds new capture bindings with disable: false unless the policy or connector requires disable: true. These changes take effect only after publication.

The UI reads discover logs through PostgREST today. It must use Discover.logs or continue to fetch logs_token from the discover row through PostgREST. The GraphQL response does not expose that token.

Schema

extend type QueryRoot {
  """
  Returns a discover visible to the caller, or null.
  """
  discover(id: Id!): Discover
}

extend type MutationRoot {
  """
  Queue discovery for a capture using its staged or live definition.
  Discovery updates the given draft with its results.
  """
  createDiscover(
    draftId: Id!
    captureName: Name!

    """
    Optional data plane name for discovery. Selected automatically when omitted.
    """
    dataPlane: String
  ): Discover!
}

type Discover {
  id: Id!
  draftId: Id!
  captureName: Name!
  dataPlaneName: String!
  status: DiscoverStatus!
  createdAt: DateTime!
  updatedAt: DateTime!

  """
  Errors currently recorded on the associated draft.
  """
  errors: [Error!]!

  """
  Discovery logs in timestamp order. Pagination is best effort, and logs
  may arrive after discovery completes.
  `first` controls page size and cannot exceed 1000 log lines.
  """
  logs(after: String, first: Int): LogLineConnection!
}

enum DiscoverStatus {
  "The discover is queued or in progress."
  QUEUED
  SUCCESS
  WRONG_PROTOCOL
  TAG_FAILED
  IMAGE_FORBIDDEN
  DISCOVER_FAILED
  NO_DATA_PLANE
  NOT_AUTHORIZED
  MERGE_FAILED
  DEPRECATED_BACKGROUND
  PULL_FAILED
}

type LogLineConnection {
  pageInfo: PageInfo!
  edges: [LogLineEdge!]!
}

type LogLineEdge {
  cursor: String!
  node: LogLine!
}

type LogLine {
  loggedAt: DateTime!
  stream: String!
  line: String!
}

Id, Name, DateTime, Error, and PageInfo reuse existing types. Error is the type that Draft.errors uses. Log cursors use the existing TimestampCursor representation of loggedAt.

createDiscover

All operations require authentication. createDiscover checks draft ownership before inspecting the capture. A draft owned by someone else gives the same "draft not found" error as a missing draft, matching the draft API.

The mutation rejects the request without committing any changes in these cases:

• The caller lacks SpecEdit on captureName.
• The draft entry is a staged deletion, has delete: true, names another catalog type, or cannot deserialize as CaptureDef. A live capture of the same name does not replace an invalid draft entry.
• The draft has no entry under captureName, and the caller cannot read a live capture of that name with CatalogRead. Missing and unreadable live captures give the same error.
• The endpoint does not identify a connector image with a tag or digest, or its configuration is not an inline JSON object. The discovers.endpoint_config column requires an object, even though CaptureDef also accepts references to configuration files.
• The image reference does not identify a known connector tag with protocol capture and job status success.
• No data plane satisfies the selection and authorization rules below. Missing and unauthorized data planes give the same error.

On acceptance, the mutation inserts the discover row. The existing trigger schedules discovery. The mutation returns the job with status QUEUED:

• If the draft has no entry under captureName, the mutation queues discovery using the readable live definition. The executor prepares its working capture when execution starts and persists successful discovery results.
• Existing draft entries do not change. The executor merges collections when discovery runs.
• Submission leaves the draft's modification time unchanged.
• The row records connector_tag_id, endpoint_config, and update_only as described below.
• Existing draft errors remain until the executor applies its outcome.

connector_tag_id identifies the connector_tags row for the endpoint's image name and tag or digest. A connector tag job requests Spec, validates the returned metadata, and updates that row. endpoint_config contains the endpoint configuration from the capture definition. Staging and submission must preserve the order and values of encrypted configuration fields so SOPS can verify the document.

A rejection leaves the draft, its errors, its modification time, and the job queue unchanged. Acceptance does not validate endpoint credentials or guarantee that the connector can connect to the external endpoint. The executor rechecks the connector tag, SpecEdit, and access to the selected data plane when it runs. It also needs the data plane's first configured HMAC key to authenticate to the connector proxy. Changes to permissions, connector metadata, or the selected data plane can therefore cause a later failure.

The executor uses the data plane named in the discover row. It does not repeat selection against the storage mapping.

Connector and merge failures do not seed a missing capture in the draft. An unrelated malformed definition in the draft can also cause discovery to fail. The executor loads the whole draft, and any errors in that loaded draft prevent it from committing the merged definitions.

Discovery policy defaults

The API derives update_only from autoDiscover. The executor derives the policy for changed keys from the capture it loads:

autoDiscover	update_only	Mark collections for reset when their keys change
Missing or null	false	No
{}	true	No
Explicit flags	!addNewBindings	evolveIncompatibleCollections

When update_only is true, new capture bindings enter the draft disabled. When it is false, the connector can still recommend disabling a new binding. This flag does not prevent discovery from removing capture bindings whose resource paths are absent from the discovered bindings.

The table describes how this API derives policy. PostgREST clients continue to supply update_only directly. Missing or null autoDiscover also disables periodic automatic discovery. The API does not insert the object, change its flags, or accept overrides for an individual job.

Data plane

The optional dataPlane argument selects the data plane for this discovery operation. Omission and null have the same meaning:

• An existing live capture uses its current data plane, even when the draft contains an edited definition. This also applies when the caller lacks CatalogRead on the live capture. That permission controls reading the live definition, not selecting the data plane. A supplied dataPlane must name the current data plane.
• A new capture uses a supplied dataPlane if publication's rules for new specifications permit it under the applicable storage mapping.
• Without an explicit selection, a new capture uses the mapping's primary data plane: the first data plane in its list.

The earlier discussion allowed an explicit selection from the storage mapping for either a new or an existing capture. This API uses a narrower rule for existing captures. Named secrets permit decryption only from the capture's current data plane. An endpoint may also restrict connections to that data plane's addresses. The rule rejects a different data plane at submission instead of accepting a job that can fail for these reasons.

Publication ignores an explicit data plane for an existing capture. Discovery instead rejects a different data plane, so it does not silently substitute one the client did not request.

For a new capture, submission fails if no storage mapping applies or the mapping does not permit the supplied data plane. If selection needs a primary data plane, the mapping must list one. Discovery reads the most specific matching storage mapping from the request’s authorization Snapshot, which contains its prefix and ordered data plane list. Stores, recovery mappings, and unrelated malformed mappings do not affect selection. Discovery and publication share the data plane selection rules, including the exception that an explicit data plane is permitted by the ops/ mapping even when its list is empty. Publication's placement behavior remains unchanged.

At submission, every selected data plane must pass authorization and be present in the API's Snapshot with a usable connector route and first HMAC signing key. This includes the current data plane of an existing capture. Discovery permission denials and data-plane failures use the existing request error handling: when the Snapshot predates the operation's start time, the API requests a Snapshot refresh and returns HTTP 307; otherwise it returns a terminal GraphQL error that includes the status code and message. Storage-mapping selection failures continue to request a background Snapshot refresh and return an ordinary error. Storage-mapping edits become visible to discovery after the Snapshot refreshes.

The mutation stores the selected data plane in discovers.data_plane_name and returns it as Discover.dataPlaneName. This does not set the data plane for publication. A later publication independently selects the data plane for a new capture. Clients that require the same data plane must also select it when publishing.

discover(id)

For an authenticated caller, discover(id) returns the row if the caller owns its draft, and null otherwise. It reads rows created through either GraphQL or PostgREST.

status reports progress or the outcome. QUEUED includes both waiting and running. SUCCESS means that discovery merged its results into the draft, not that the draft passed publication validation. errors returns the draft's current errors, which can change independently of this discover's status.

Staging definitions leaves existing errors in place. Each discover or publication replaces them when it applies its outcome. Some discover failures clear the errors without inserting new ones: NO_DATA_PLANE, TAG_FAILED, WRONG_PROTOCOL, and IMAGE_FORBIDDEN report only the status.

The enum retains all existing status values, including MERGE_FAILED, DEPRECATED_BACKGROUND, and PULL_FAILED, so historical rows remain readable. GraphQL uses these enum values while stored job_status.type strings retain their camelCase representation. Discover.status exposes only that discriminator. Historical success records can contain publication_id and specs_unchanged; they remain readable, but those fields are not exposed or written by the current executor. This requires no rewrite of stored JSON or change for PostgREST readers.

Logs

Ordering

The agent's log writer currently gives all lines in a batch the same timestamp. The table has no other ordering field. A cursor containing only a timestamp cannot resume from the middle of a batch.

Change this writer to assign timestamps at PostgreSQL's microsecond precision. Each timestamp must be at least one microsecond later than the previous timestamp from that writer, including across batches. It must also be no earlier than the writer's current clock reading at that precision. The writer must retain the previous timestamp between batches, even if its clock moves backward.

The data-plane controller's writer already increments timestamps by one microsecond within each batch. The proposed change also orders separate batches from the same writer. It requires no table migration.

This changes logs for every operation that uses the agent's writer, including publications, connector tag jobs, and validation. It provides order within one writer's lifetime. A restarted writer or another agent can produce duplicate or earlier timestamps for the same discover, because writers do not coordinate clocks or commits.

Timestamps will come from the agent's clock instead of the database's. Differences between those clocks can affect both pagination across writers and retention. Cleanup compares logged_at with the database clock and deletes lines older than two days. Small clock differences only shift that retention window slightly, but the API cannot assume all clock differences are small.

Existing lines retain their shared timestamps until cleanup deletes them. Agents that still run the old writer during deployment can also produce such lines. A page boundary within one of these groups can skip lines.

Reading

Discover.logs returns lines in ascending loggedAt order. It uses the existing TimestampCursor, with the last returned line's timestamp as the next cursor. Subsequent pages select timestamps strictly greater than that cursor. The default page size is 100, and first must be between 0 and 1000 inclusive. Invalid cursors, negative sizes, and sizes above 1000 return errors. The cap applies to one page; clients can paginate through more than 1000 lines. With first: 0, the connection returns no edges or end cursor and reports hasNextPage: true if a matching line is visible.

A delayed write from an earlier attempt can have a timestamp at or before a cursor the client already received. Subsequent pages will miss those lines. Duplicate timestamps from separate writers have the same problem. Pagination therefore remains best effort, even after the writer change.

The existing index on token can locate a discover's logs. The query must then sort them by timestamp. This plan adds no index or migration.

Lines reach the table asynchronously, and some can arrive after the status leaves QUEUED. There is no signal that all logs are available. hasNextPage: false means the query found no additional visible lines beyond the returned page. A client can continue polling after discovery finishes, but an empty page does not prove that no more lines will arrive.

A completion guarantee would require the log writer to acknowledge committed lines before the executor records the final status. No such coordination exists today.

LogLine and its connection can later support publication logs as well. Resolving logs through their parent discover keeps logs_token out of the GraphQL API. Each log query must enforce draft ownership, as the draft's nested resolvers do.

Deferred work

The API leaves these changes to separate work:

• Rename or alias autoDiscover to discovery, preserving the distinction between an absent field and an empty object.
• Revisit the permission required on a data plane, including the effect on the Editor bundle. This API keeps the executor's current authorization rule.
• Coordinate concurrent edits and jobs on the same draft.
• Enforce token restrictions on definitions that executors copy into drafts.

[jshearer]

The inner fields that Success used to have were vestigial. I found no reads of either field in the UI, and no consumers remain in the backend. #1764 moved auto-discovers into capture controllers and stopped producing meaningful values for these fields. Since then, the discover handler/executor has written discovers.job_status from Success { publication_id: None, specs_unchanged: false }. Serde's skip_serializing_if = "Option::is_none" and skip_serializing_if = "std::ops::Not::not" already omitted both fields, so successful jobs were already stored as {"type":"success"}.

[strix-security]

Strix Security Review

No security issues found.

Review summary

Reviewed the full PR diff for the new createDiscover and discover(id) GraphQL APIs, the discovers::create admission path, the shared data-plane selection and storage-mapping snapshot helpers, the placement refactor, the draft-error resolver extraction, the agent-side discover executor change, and the JobStatus model migration.

Every authorization boundary was traced to its enforcement point. Draft ownership is enforced by SQL on d.user_id in the create path and in both the query and the errors/logs resolvers. createDiscover gates on SpecEdit at the GraphQL boundary; create then checks CatalogRead on the live capture (when no staged entry is present), CatalogRead on each binding target with the token's capability mask and prefix scope applied (closing the executor-side exfiltration gap), connector-tag readiness, and read plus connector-route validity on the selected data plane. Staged/live precedence, update_only derivation from autoDiscover, and storage-mapping data-plane selection all match the documented policy and existing publication semantics. All new SQL uses bound parameters, and the first pagination bound is enforced with negative and oversized values rejected at the connection layer.

No security issues were identified; authorization, input handling, and resource bounds are correctly implemented and covered by the accompanying tests.

_{Updated for a9e0d4e.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}

[GregorShear]

i don't have an opinion on this yet, and don't love the idea of introducing much more complexity here, but i wonder if we want to authorize against the user_id AND a capability bit just to give a human user the opportunity to withhold that bit from a token mask (does that make sense)?

[jshearer]

So far capability bits have always existed in the context of a catalog name/prefix. There's no fundamental reason why there can't be a "universal" capability bit (ManageDrafts, ManageDiscovers) that's by-default granted to everyone/in the most widely distributed bundle(s). I had a similar thought but didn't want to expand scope here. Easy enough to do later if we want to? Or do you think it's necessary?

[GregorShear]

no need to it now, especially if we're not sure what the right thing is. just wanted to raise for discussion

[bbartman]

I had claude, and codex review the tests, I would like you to consider adding tests for the following cases. I understand that some of these may not be possible, but as I'm still learning the system please just call out which ones are extremely difficult to test.

1. A deserializable capture with "delete": true.
2. A non-connector endpoint.
3. An image without a tag or digest.
4. A successful digest-form image.
5. No applicable storage mapping.
6. A mapping with no primary plane.
7. The special ops/ explicit-plane exception.
8. Verifying that we are cleaning the errors from previous drafts. I want to make sure that I understand this part correctly, if we have previous error we can attempt another discover and that should clear out the previous errors from the draft, right?

Are you planning on adding the secrets changes that Johnny requested later? My assumption here is that they are out of scope or included as part of something else.

[jshearer]

Alright @GregorShear @bbartman, thanks for the reviews!

You both called out function length. Feedback taken. I originally had this feedback myself when self-reviewing the PR but didn't want to just split it up into a bunch of single-use helpers. So instead I cut down what it has to do:

• I moved submission out of the resolver into discovers::create. It reads top to bottom as a sequence of steps, with the pure checks as named helpers (select_capture_model, extract_discovery_endpoint). The resolver went from ~270 lines to ~40.
• No more transaction or row locks. Once submission stopped copying the live capture into the draft (the executor already does that when it runs), a single INSERT was the only write left, and its trigger schedules the executor within the same statement. The locks only covered the gap between reading the draft and inserting, but the executor re-reads the draft when it runs, and we don't serialize draft edits between submission and execution anyway. If the draft gets deleted mid-submission, the insert just fails its foreign key check. So the "lock inversion" comment is gone too.
• One query checks draft ownership and reads the capture's draft entry.

WRT auth:

• Discovery and publication placement now share select_data_plane. Discovery reads storage-mapping data planes from the authorization Snapshot (storage_mapping_for) instead of querying them per request.
• I want createDiscover to do all the auth up front, but the executor can only treat discovers as already authorized once every client goes through GraphQL. Until then it still runs its own checks on everything it picks up. Separately, you can still change the draft between submission and the executor picking it up. The plan is to content-hash the draft so the executor knows it's working over the same state that was authorized, but that's out of scope for this PR.
• A capture's ability to write to its binding targets is checked against its own role at publication, so the token mask doesn't apply there. But the executor also copies the targets' live definitions into the draft, filtered by the user's CatalogRead, and runs without the token's mask or scope. So createDiscover now checks CatalogRead on every binding target using the request's masked subject. I added a regression test where the owner can read a target but a prefix-scoped token can't. The executor's own check can go away once all clients go through the API.
• I left out a capability bit for reading a discover for now. It's just draft ownership, same as drafts themselves. Happy to revisit if we want masks to apply to draft reads.

I also consolidated the GraphQL tests (~1,300 lines down to ~720). They now cover a few rejections that weren't tested before.

[bbartman]

This is easier to read. I do have one question about how we handle the 307 retry logic, do we want to enforce the same thing for other is_authorized checks?

[jshearer]

I do have one question about how we handle the 307 retry logic, do we want to enforce the same thing for other is_authorized checks?

I read this as asking why some authorization denials get different response handling from others. With that in mind, I updated the other discovery authorization checks to return typed tonic::Status errors so they go through the same response handling.

The resolver now passes those statuses through the existing authorization_outcome handling. When the Snapshot predates the operation’s start time, the server requests a Snapshot refresh and returns 307.

[bbartman]

LGTM!