Approval request ID changes after process crash between emission and durable checkpoint

[xuelongmu]

Summary

On eve@0.52.1, killing the runtime after it publishes input.requested but before the enclosing turnStep checkpoints can regenerate the model step and approval request ID on recovery. A response to the approval already shown to the user then refers to the old ID and cannot approve the pending action. The session asks for approval again instead of resuming the original request.

The tested path refuses the stale approval; no unauthorized tool execution was observed. The failure is loss of approval continuity after process interruption, with an additional model attempt.

Link to a minimal reproduction

No public standalone reproduction repository is available yet. The completed reproduction is an isolated integration fixture in a private application repository, using the built Eve service, a synthetic model, an approval-gated tool and a loopback application callback. The relevant crash barrier, sequence and synthetic evidence are included below so this report does not depend on access to that repository. This is a limitation relative to the issue template's requested public reproduction link.

Steps to reproduce

1. Build and start the Eve service on eve@0.52.1, using local persistent Workflow storage. Keep the same build and storage directory when replacing the child process.
2. Drive a synthetic model into a tool call requiring human approval. Capture the request ID from input.requested on the client stream. No tool execution should have occurred yet.
3. Make the first turn.completed hook await a loopback HTTP callback. Have the callback handler perform its normal application write, but withhold its successful HTTP response. This leaves the hook pending and prevents the computing turnStep from returning/checkpointing. Apply this barrier only once, before any tool execution.
4. Consume the client response only through the turn.completed event. Do not consume MessageResponse through the later session.waiting, and do not wait for a parked Workflow checkpoint: those move past the target crash window.
5. Once the callback is held, forcibly terminate the fixture's Eve child process and wait for its exit. Only then release the held callback. Start a replacement process against the same runtime data.
6. Attach to the same session and answer the original captured request ID using session.respond(...) with the original approval option.
7. Observe a newly emitted approval request with a different ID instead of execution of the original approved tool call.

The relevant test barrier, expressed as pseudocode rather than a standalone runnable example:

// In the application callback server; normal handler already ran.
const response = await handleApplicationCallback(request);
if (firstCompletionBeforeToolExecution && response.ok) {
  signalBarrierReached();
  await barrierReleasedAfterOriginalProcessExit;
}
sendHttpResponse(response);

// Client/harness.
for await (const event of initialResponse) {
  if (event.type === "input.requested") {
    originalRequest = event.data.requests[0];
  }
  if (event.type === "turn.completed") break;
}
await barrierReached;
await forciblyTerminateAndWaitForExit(originalChild);
releaseBarrier();
await startReplacementWithSameBuildAndStorage();
// Attach to the original session and respond to originalRequest.requestId.
// Assert that recovery does not replace it or recompute its model/tool call.

For the fast diagnostic, the child receives WORKFLOW_INLINE_OWNERSHIP_LEASE_SECONDS=5. This only shortens the observation time and is not proposed as a production setting. An earlier timing-based probe with the installed default lease also reproduced approval-ID replacement after approximately 860 seconds. The callback barrier makes the interruption point deterministic; a timing-only kill can miss it and pass.

Current vs. expected behavior

Current: the deterministic diagnostic failed after 6,367 ms. The stream exposed request A before termination, then request B after replacement. The response submitted by the client still named A. There were three synthetic model calls, 330 reported tokens and zero gated-tool submissions. The extra attempt was retained in usage accounting.

Expected: after a request is externally visible, recovery retains that request's identity, exact tool call/input and pending execution state. Responding to the original request resumes that exact action without asking again or repeating model computation for the committed approval. Retried publication may deliver identical events again, provided consumers can deduplicate them.

Control tests that kill only after the durable parked approval checkpoint pass, as do persisted generation-wait replacement and completed-history replay. Those controls do not cover the earlier emission/checkpoint window.

eve version

• Runtime reproduction: exact npm package eve@0.52.1.
• Source inspection: upstream commit 4b5fad41beeecae1286611974ea970d565d13dde, whose package version is 0.52.2, retains the publication-before-return ordering described below. This is source evidence, not a completed runtime reproduction on 0.52.2.
• No known-good version established.

Environment

OS: Windows
Node.js: v24.19.0
Package manager: npm
Runtime: built Eve service launched as a local Node child process
Workflow storage: local .eve/.workflow-data, retained across replacement
External model/tool dependencies: synthetic; no paid provider calls
Application-state fixture: in-memory PGlite with loopback HTTP callbacks

Where does the bug occur?

Other: local process-loss test of the built service. This is not an eve dev hot-reload test and has not been reproduced on hosted Vercel.

Deployment

No hosted deployment involved.

Build and runtime logs

Synthetic diagnostic excerpt (application paths omitted):

AssertionError: Recovery replaced the pending approval; the submitted response no longer applies

Expected: ["aitxt-4trl2wb0LNIaM1beb6PgvvEf"]
Received: ["aitxt-wLv7Gd5h1hs2MuAJnLKCegfA"]

Test assertion failed after 6367 ms.

Retained probe data:

{
  "restartAt": "approval-emitted",
  "ownershipLeaseSeconds": 5,
  "approvalEmissionBarrier": {
    "heldAt": "2026-09-06T22:38:22.180Z",
    "releasedAt": "2026-09-06T22:38:22.211Z"
  },
  "emittedApprovalRequestIds": [
    "aitxt-4trl2wb0LNIaM1beb6PgvvEf",
    "aitxt-wLv7Gd5h1hs2MuAJnLKCegfA"
  ],
  "providerCalls": 3,
  "syntheticGenerationSubmissions": 0,
  "waitPolls": 0,
  "usage": [{ "status": "pending", "total_tokens": 330 }],
  "captures": []
}

Source analysis

The following is the suspected cause, based on the installed package and the pinned upstream source:

1. turnStep runs the harness inside one Workflow step, then returns serialized context and session state.
2. The tool loop constructs the pending input batch and awaits approval emission and the turn epilogue before returning that state.
3. Event emission calls the channel adapter/writes the stream, and lifecycle hooks run, before the Workflow step result is checkpointed.
4. After interruption, the previous supplied session snapshot is used to retry the step. Public events do not reconstruct the lost pending harness state. The repeated model step can generate a fresh approval ID; stale-response handling then prevents the earlier answer from approving it.

Suggested implementation prompt

Ensure that externally published blocking input requests always correspond to committed pending execution state. A possible approach is to checkpoint the complete pending session plus an ordered publication batch, then publish through a separate durable step with stable event/request identities and retry-safe hook effects. Preserve the exact tool call and input; changing IDs deterministically alone does not prevent model repetition or changed tool arguments.

Cover inline, turn-owned and legacy execution paths, cancellation and buffered responses, post-hook context, stream closure, and compatibility with retained histories/pinned drivers. Verify crashes before publication, after the stream write, and after callback success but before the publication checkpoint. The original request must remain answerable, the tool must execute once, and every actual provider attempt must remain accounted for. This does not promise exactly-once provider execution in the earlier window before any model result is durable.

Additional context / related work

• refactor(eve): interpret HITL requests as one ledger transition #2863, stacked on refactor(eve): centralize durable HITL request state #2822, appears closely related: its description persists group completion before delivering owner effects. Does that work also protect initial input.requested publication before the computing step commits, or should this window be covered separately?
• world-local 30s hook timeouts redeliver and re-bill a long model call; process death mid-call does not always replay #2425 reports repeated model calls and inconsistent process-death replay; this report identifies the narrower approval publication boundary.
• Graceful shutdown exits before the workflow queue releases in-flight jobs — async turns wedged ~14 min after restart #1981 reports approximately 14-minute restart recovery on world-postgres. Similar delay, different backend and shutdown trigger.
• Multi-request approval batch answered via separate respond() calls settles every request (approval.settled ×N) but the turn never resumes #2182 strands approvals after separate responses to a multi-request batch; this reproduction does not require a multi-request batch.

This report requests confirmation of the runtime guarantee and the appropriate regression/fix boundary. No upstream implementation is being claimed.