feat(e11): a persistent-connection Go Exec client (#294)

[pdettori]

Closes #294.

Stacked on #293 (fix/291-e11-driver-artifact), not on main — it builds on that PR's
driver-control arm and its repaired sampler. Review or merge #293 first.

Why

#291 deferred its item 3 — replacing grpcurl with a persistent-connection client — on the
grounds that committing before measuring would be a guess. #293's control arm supplied the
measurement, on bare metal, with no relay, no Redis, no worker and no VMM behind it:

c	tput/s	p95	coresBusy /72
8	241.6	44 ms	14.52
64	231.6	753 ms	64.13

The driver alone peaks at c=8 and then declines, burning 64 of 72 cores at c=64 with its
own p95 of 753 ms. That is the same knee position and curve shape §E11 published for both real
arms. The published density knee is reproduced by an arm with no backend at all.

What this adds

remote-worker/cmd/exec-driver — one process per rung, one grpc.ClientConn brought to READY
before any slot starts, and c goroutines in place of c bash subshells. Four small files:
cause classification (a deliberate mirror of grpc_exec_record's ordered grep chain), the rung
plan and its refusals, the issuer, and the binary.

It writes the same per-slot <ms> <status> <cause> lines the bash path wrote, so warmup
trimming, percentile, execErrorsByCause, the sampler bracket and the record writer are
untouched — not by claim, but because the file format is the interface.

Opt-in. SH_E11_EXEC_CLIENT defaults to grpcurl. It stays the reference until both clients
have run on one host, which is #294's own acceptance criterion, and every rung record carries
execClient so two ladders can never be silently mixed.

One substantive edit to the reference path, stated plainly. The bash path's driving mechanism
is unchanged — one grpcurl process per Exec, the subshell loop moved into an else branch and
nothing more. But grpc_exec_record's latency arithmetic was changed: it recorded whole
milliseconds, and the Go client's per-Exec latency is sub-millisecond, so at the Go client's measured
rates every recorded value truncated to 0 and p95Ms collapsed with it. Both paths now record
three decimal places, computed from integer microseconds with no floating point. They had to move
together: one path fractional and the other integer would make the comparison a rounding artifact.

The consequence worth knowing: grpcurl ladders published before this change carry integer p95Ms,
and new ones carry fractional. The spec and the runbook say so too.

Converge stays on grpcurl and drivingModel stays closed-loop-per-slot — both stated
non-goals, and both asserted by tests rather than left to trust.

What this does NOT include

The authoritative comparison. That belongs on the host that produced the table above, and is
not run here. docs/notes/e11-go-exec-client-comparison-runbook.md has the reference numbers, the
two invocations, and the checks to make before quoting anything — including that the Go arm needs
its sampler cadence lowered rather than its Exec count raised, because at matched Exec counts its
rungs are ~50x shorter and the driver refuses a rung that produced zero sampler ticks.

§E11's conclusions therefore stay marked under repair. A sweep run today would measure the
driver's knee again.

Indicative local numbers only

Both clients against the null-responder on a 10-core laptop, identical Exec counts (2000/slot):

c	grpcurl Exec/s	go Exec/s	gap
1	38.4	2133.6	55.6x
4	183.9	11289.9	61.4x
8	286.5	13975.7	48.8x

Not the 72-core comparison the acceptance criteria name: no /proc sampling, so no
hostCpuFraction, coresBusy or p95 at all; mix reduced to one command; n=1; and
e11-density.sh itself never ran. It shows both clients work and the per-Exec gap is large. It
says nothing about where any knee sits.

Corroboration worth noting for whoever books the rig: grpcurl's rates here land close to the
published metal driver-control figures (56.8 / 171.6 / 241.6) despite 10 cores versus 72 — which
is what you expect if the bound is per-call process spawn rather than the host.

One defect found and not fixed here

packages/sandbox-relay/src/relay.ts's routeExec yields an in-stream ExecEvent.error and then
returns a gRPC OK status, so grpcurl exits 0 and the bash path counts a failed Exec toward
throughput, into the p95 distribution, and never into execErrorsByCause. Filed as #295.

Deliberately left split: the Go client classifies it correctly, the reference path is unchanged,
and on driver-control — the only arm #294 measures — the divergence is exactly zero, because
the null-responder never emits an ExecError. Fixing relay.ts here would move the reference path
mid-comparison. Every rung record discloses which behaviour produced it, on both paths, and #295
records that the fix must land before any cross-client comparison on the container or microvm arms.

Verification

gofmt, go vet, 16 Go tests, shellcheck -x -S warning on both shell files, all four
deploy/microvm suites, tsc --noEmit and vitest for experiments — all clean.

The seam is proven end to end by a test that drives the real write_rung_plan into the real
exec-driver against the real null-responder and reads the times files back: e11-density.sh
needs /proc, cgroups and Linux, so that test is the only off-rig proof the boundary works, and it
needs none of them.

Several checks were mutation-tested rather than trusted — shortening runSlot's loop, swapping the
two disclosure arms, removing the Go branch's pids+= — each producing exactly the attributable
failures it should. Six vacuous guards were found and fixed along the way, most of them in the
plan's own text.

🤖 Generated with Claude Code

[pdettori]

Review: Approved, one fix requested before quoting metal numbers

I stacked-verified this against #293 (83fd348 is a real ancestor), ran go vet/gofmt -l/go test ./... on remote-worker (clean), ran the full deploy/microvm/tests/e11-density.test.sh including the real seam test that builds and runs the actual exec-driver/null-responder binaries (449 checks, 0 failures), ran shellcheck -x -S warning and shellcheck -x -o check-unassigned-uppercase -S warning on both touched files (0 findings each), and diffed EXPERIMENTS.md / predictions.json / e10-lifecycle.sh / packages/sandbox-relay against the stated mandate (additions-only, byte-identical, untouched, untouched). All 21 commits are DCO-signed with Assisted-By: Claude, no Co-Authored-By.

Please fix before the metal run: sub-millisecond truncation collapses p95/knee on the Go arm

remote-worker/cmd/exec-driver/drive.go (oneExec's fail closure and the success path) computes:

ms: time.Since(t0).Milliseconds()

an int64 truncation to whole milliseconds. This PR's own indicative numbers put the Go client at ~0.47ms/Exec at c=1 and ~0.07ms/Exec at c=8 against the null-responder — so on the driver-control arm almost every recorded ms will be 0. That's not just a cosmetic loss of precision, it degrades two downstream metrics to the point of being uninformative:

• percentile() in deploy/microvm/e11-density.sh does sort -n over these integers with no floor — a non-empty, near-all-zero file legitimately yields p95Ms=0 (it's not refused).
• detectKnee in experiments/src/sharing.ts sets bound = baseline.p95Ms * degradeX. With a 0 baseline, bound is 0, so healthy = cur.p95Ms <= 0 && ... — the detector becomes a step function on the exact tick where per-Exec latency first rounds up to 1ms, not a real health signal.
• coldAcquireRate (threshold 50ms by default) will be pinned at 0 for the whole Go-arm ladder — not wrong, just uninformative.

coresBusy/hostCpuFraction (the sampler's independent host-CPU signal) is not affected by this, since it doesn't come from Exec latency. So this doesn't need to block running the driver-control comparison — the coresBusy delta at high c is still meaningful without this fix. But please land a fix before anyone quotes a p95 or knee number from the Go arm, since right now those numbers would be measuring the millisecond clock's resolution, not the driver.

Suggested fix, and it looks cheap: record fractional milliseconds (e.g. %.3f) on both paths so the two remain comparable. require_numeric's regex (\-?[0-9]+(\.[0-9]+)?) already accepts decimals, and percentile's sort -n handles them fine — I didn't find anything else that assumes integer ms.

Worth a runbook note, not a blocker: sampler cadence vs. PSS-walk frequency

docs/notes/e11-go-exec-client-comparison-runbook.md is right that matching Exec count and lowering SAMPLE_INTERVAL_MS/SAMPLE_SLICE_MS is the way to keep both arms' Exec counts comparable, and it correctly notes SAMPLE_SLICE_MS (not just SAMPLE_INTERVAL_MS) is the real cadence floor. One thing it doesn't call out explicitly: SAMPLE_LOW_EVERY throttles the pgrep+smaps_rollup PSS walk relative to tick rate, and tick 1 always samples PSS regardless — so pushing tick rate up to the ~70Hz needed for 10 ticks in a ~140ms Go-arm window also pushes PSS/pgrep sampling to an absolute rate well past the ~1Hz-ish envelope the #291 fix was sized to avoid perturbing. Since none of p95/throughput/coresBusy actually require matched Exec counts to be individually valid, sizing to a common wall-clock window instead (letting Exec count differ) would sidestep that risk entirely. Not asking for a code change — just suggest adding a line to the runbook naming this tradeoff before someone cranks cadence past the point where PSS sampling itself becomes the next artifact.

Everything else I went looking for came back clean

• cause.go's classification order mirrors grpc_exec_record's grep chain exactly (confirmed by diff and by TestCauseForPrefersTheEarlierRuleWhenTwoMatch's overlapping-input case).
• req_id disjointness is enforced (plan.go:validateReqIDRanges) and tested both at the boundary and server-side.
• The sandbox-relay: an in-stream ExecError returns a gRPC OK status, so a failed Exec counts as a success #295 relay defect is genuinely zero-difference on driver-control: null-responder only ever sends ExecEvent_End, never .Error, and packages/sandbox-relay is untouched by this PR or fix(e11): repair the density instrument's measurement artifacts (#291) #293.
• SH_E11_EXEC_CLIENT defaults to grpcurl and is validated before any build step.
• No live extract_fn/shellcheck traps in the touched files; the seam test that builds and runs the real Go binaries against the real null-responder passes.
• Constraints held: set -uo pipefail as the driver's first set line, drivingModel unchanged, converge_slot never touches EXEC_CLIENT, EXPERIMENTS.md additions-only, e10-lifecycle.sh/its test/predictions.json byte-identical to #293.

Bottom line: land the fractional-ms fix (small, both paths) before quoting any p95/knee number off the metal run. Everything else here is solid enough to run the driver-control comparison as-is for the coresBusy signal.

Assisted-By: Claude Code

[pdettori]

Both points addressed in 57152f3. Thank you — the truncation finding is a real defect, and the
evidence for it was sitting in this PR's own indicative table. I had the 0.469 ms/Exec figure in
front of me and did not follow it through to p95Ms.

Fractional milliseconds, both paths

Recorded latency is now %d.%03d milliseconds, computed from integer microseconds with no
floating point, on both clients — so the two stay exactly comparable:

• drive.go — execOutcome's field is renamed ms → us so the unit is unambiguous, filled from
  time.Since(t0).Microseconds(), and formatted in line().
• e11-density.sh — grpc_exec_record does the same.

One constraint shaped the bash side: that function is asserted to contain no command substitution
at all — the #291 item-2 no-fork-in-the-timed-window guarantee — so $(printf '%.3f' …) was not
available. The construction is arithmetic and parameter expansion only:

frac=$((1000 + us % 1000))
ms="$((us / 1000)).${frac#1}"

1000 + us % 1000 lands in 1000..1999, so ${frac#1} is the remainder zero-padded to three
digits. Both no-fork assertions still pass unchanged:

ok:   ...and its body runs no command substitution -- not even one avoided fork
ok:   ...while still using $((...)) arithmetic expansion for the ms computation

You were right that nothing downstream needed changing — I re-verified each: require_numeric's
\-?[0-9]+(\.[0-9]+)?, percentile's sort -n, and cold_count's awk '$1>=t'.

Before/after, driving the real binaries against the real null-responder (8 Execs, c=1). Every
one of these lines read 0 ok - before the fix:

1.836 ok -
1.689 ok -
0.431 ok -
0.324 ok -
0.271 ok -
0.209 ok -
0.205 ok -
0.204 ok -

percentile()'s own arithmetic over them gives p95Ms = 1.689 where it previously gave 0. (The
first two calls are connection warmup settling to ~0.2 ms — the low-c startup inclusion the
runbook already discloses.)

Tests: the seam test now pins the three-decimal shape rather than tolerating it; the
"no command substitution" fixture was updated to the new arithmetic so it stays representative; and
TestDriveRecordsSubMillisecondLatencyRatherThanZero is new — it would have failed before this
commit and is what stops the regression coming back. 17 Go tests pass, Total failures: 0 across
449 checks, shellcheck clean on both files.

Records written before this change carry integer ms; that is noted in the spec and the runbook.

Cadence versus the PSS walk

Added to the runbook rather than changed in code, as you suggested. It now names SAMPLE_LOW_EVERY
and the always-sample-on-tick-1 behaviour, states the tradeoff plainly — cranking cadence to reach
10 ticks inside a ~140 ms window can make the pgrep + smaps_rollup walk the next artifact,
well past the ~1 Hz envelope #291 was sized for — and gives your alternative: size both arms to a
common wall-clock window and let the Exec counts differ, since p95, throughput and coresBusy
are each individually valid without matched counts. Matched counts were a cleanliness choice of
mine, not a correctness requirement, and the runbook now says so and leaves the operator to choose.

🤖 Generated with Claude Code

[pdettori]

Automated review notes

Adds remote-worker/cmd/exec-driver, a persistent-connection Go alternative to the bash+grpcurl Exec client, gated behind opt-in SH_E11_EXEC_CLIENT=go (default remains grpcurl). The Go code is well-tested, defensively validates its plan input, and a genuine bash/Go integration test exercises the real boundary. No secrets, no .claude/.vscode changes, no CI workflow changes.

⚠️ Blocking: PR is currently unmergeable

GitHub reports mergeable_state: "dirty". The recorded base (f20b0225) predates #293's merge into main (now e68ab80):

• The diff shown on the Files tab (~11,200 lines) is inflated by ~6,500 lines of already-merged fix(e11): repair the density instrument's measurement artifacts (#291) #293 content, reappearing only because the base is stale. The true incremental diff (main...PR-head, verified locally with git merge-base/git diff) is 15 files, +4666/-42.
• There's a real, if trivial, append/append conflict in deploy/microvm/tests/e11-density.test.sh (both main via fix(e11): repair the density instrument's measurement artifacts (#291) #293 and this branch appended a new test section at the same spot, around line 2536 — the SH_E11_EXEC_CLIENT section vs. main's "between-arm relay teardown kills by PORT" section). git merge-tree confirms deploy/microvm/e11-density.sh and deploy/microvm/EXPERIMENTS.md (also touched on both sides) auto-merge cleanly — only the test file needs manual resolution.

Suggested fix: git fetch origin main && git rebase origin/main, keeping both appended test blocks. This resolves the conflict and shrinks the PR's displayed diff to its true size.

Suggestions (non-blocking)

• remote-worker/cmd/exec-driver/cause.go:17 — causeFor is a hand-maintained mirror of grpc_exec_record's bash grep chain, kept in sync only by a DRIFT GUARD test, not a single source of truth. Worth a follow-up (e.g. a shared data file) so a future edit to one side can't silently desync the other — which would defeat the PR's own goal of comparable ladders.
• remote-worker/cmd/exec-driver/drive.go:199 — On a rung with multiple failing slots, only the first (by index) error is returned; the rest are silently dropped even though errs holds all of them. Not a correctness bug (the rung is refused either way), but loses diagnostics. Consider errors.Join over all non-nil entries.

Nit

• remote-worker/cmd/exec-driver/drive.go:136,141 — times.Close()/errFile.Close() errors are discarded. Low risk since Flush() is checked just before, but a close-time write failure would go unnoticed.

---

Author: pdettori (MEMBER — maintainer)
Commits: 40, all signed off (DCO passes)
CI status: passing (DCO is the only check configured)

Note: this is posted as a plain comment, not a formal GitHub review — GitHub blocks self-review (REQUEST_CHANGES/APPROVE) on your own PR via the API.

🤖 Generated with Claude Code

[pdettori]

Rebased onto main (now e68ab80, with #293 merged) and both suggestions applied. Head is
af1c366.

Blocking item resolved

mergeable is now true, and the displayed diff has collapsed to its true size:

	before	after
base	f20b0225 (pre-#293)	e68ab80
changed files	19	15
additions/deletions	~11,200 shown	+4682 / −42

That matches the +4666/−42 you computed locally, plus the 16 lines of the two fixes below.

On the conflict: your diagnosis was exactly right — append/append in
deploy/microvm/tests/e11-density.test.sh, main's "between-arm relay teardown kills by PORT"
section against this branch's SH_E11_EXEC_CLIENT section, both appended at the same spot. Resolved
by keeping both, main's first. e11-density.sh and EXPERIMENTS.md auto-merged cleanly, as
git merge-tree predicted.

I checked one thing before rebasing that would have changed the approach: whether 83fd348 was
still an ancestor of main. #293 landed as a merge commit rather than a squash, so it is — meaning
the rebase replayed only this branch's 22 commits rather than trying to re-apply 18 already-merged
ones under new SHAs. A backup ref (57152f3) is kept locally in case anything needs recovering.

Post-rebase verification on the combined tree, since main's newer driver code (kill_relay_by_port,
the changed stop_*_stack functions) now sits alongside this branch's: gofmt and go vet clean,
17 Go tests pass, shellcheck -x -S warning clean on both touched shell files, and all seven
deploy/microvm suites green — including the e12/e13 suites that arrived with #293 and which
this branch had never been tested against before.

Both suggestions applied

drive.go — every failing slot, not just the first. errors.Join over all non-nil entries.
It returns nil on an empty slice, so the success path is unchanged. You were right that the rung is
refused either way; the cost was making someone re-run a rung once per slot to discover that three
failed for three different reasons.

drive.go — close-time write failures no longer discarded. Both Close() calls now report
through a named return. Flush() pushes bufio's buffer into the file, but the write-back can still
fail at Close, and a silently truncated times file would then be aggregated as though it were
complete. The err file's close — diagnostics only — is reported only when nothing worse happened,
so it cannot mask a real rung error.

On the cause.go drift suggestion

Agreed, and deliberately not done here. causeFor is a hand-maintained mirror of
grpc_exec_record's grep chain held together only by the drift-guard test, and you are right that a
future one-sided edit could silently desync it — which would defeat this PR's own goal. A shared
source of truth is the correct fix and it is a bigger change than this PR should carry: it touches
the reference path's classification, which everything here has been careful not to disturb while the
two-client comparison is outstanding. Happy to file it as a follow-up alongside #295 if you want it
tracked.

One correction to the notes

CI here is more than DCO — check, lint, deploy-scripts, proto, microvm-gates, codeql and
Trivy all run. They were mid-flight when I posted this; Trivy/trivy-scan had already reported
success. mergeable_state reads blocked because branch protection on main requires one
approving review, which GitHub will not accept from the PR author via the API — the limitation you
already noted, not a remaining conflict.

🤖 Generated with Claude Code

[pdettori]

CI on af1c366: 13 checks, 13 success, 0 otherwise — check, lint, shellcheck, hadolint, proto, deploy-scripts, microvm-gates, CodeQL, codeql (javascript-typescript), dependency-review, Trivy, trivy-scan, DCO.

microvm-gates and deploy-scripts are the two worth naming: they exercise this branch against the post-rebase tree, where main's newer kill_relay_by_port and stop_*_stack changes now sit alongside these. That combination had never been tested before the rebase.

mergeable: true, and mergeable_state stays blocked only on the one required approving review that GitHub will not accept from the PR author. Nothing outstanding from my side.

[huang195]

Mamma mia, Paolo — this is a beautiful piece of instrument work. Finding that the published c=8 knee belonged to the driver, and then proving it with an arm that has no backend at all, is the kind of result most people would have shipped as a sweep re-run instead. And the discipline around it is what makes it trustworthy: grpcurl stays the default, the times-file format is the interface rather than a claim, execClient is on every rung so two ladders can't silently mix, and the relay ExecError divergence is disclosed on both paths instead of only on the one that has it. Perfetto.

The seam test earns its keep — real write_rung_plan → real exec-driver → real null-responder, and it needs neither /proc nor Linux. Verified independently while reviewing:

• remote-worker/go.mod is go 1.26.0, so drive's per-iteration i/s capture is sound (that comment is load-bearing — on a pre-1.22 directive it would be a race).
• cause.go's switch mirrors grpc_exec_record's grep chain key-for-key and order-for-order.
• validateReqIDRanges' off-by-one is right: next == prev+calls really does collide the next slot's converge with the previous slot's last Exec, and TestLoadPlanReqIDBoundaryIsExact pins it in both directions.
• The new fractional-ms field survives everything downstream — percentile sorts with sort -n, cold_count and throughput are awk, require_numeric already accepted %.4f. Nothing breaks.

Four non-blocking items inline. Two more for the description rather than the code:

• The base-branch note is stale. #293 merged at 13:17Z today, so GitHub now shows 23 commits / 15 files — all yours. The "read this before reviewing the file list" block can go.
• "the bash path is byte-identical in substance … additions only" no longer holds. 77aaec8 replaced grpc_exec_record's ms=$(( … / 1000 )) with the three-decimal form. That is the right call — both paths must record the same precision or the comparison is against a rounding artifact — but it is a substantive edit to the reference path, and it means published grpcurl ladders carry integer p95Ms while new ones carry fractional. Worth stating outright rather than leaving the old claim standing.

And one minor, outside the diff so I can't pin it inline: e11-density.sh:540's require_tool grpcurl reason still says "every arm drives its Exec RPCs through grpcurl". On the Go path grpcurl is genuinely still required — converge stays on it — but the justification now misleads whoever debugs a SH_E11_EXEC_CLIENT=go run.

Bellissimo work, and §E11's conclusions staying marked under repair is the honest call. Approving.

---

Assisted-By: Claude Code

[huang195]

suggestion — the comment above says writes are "flushed once at the end", but bufio.NewWriter takes the default 4096-byte buffer, so that is true only while a slot's whole output fits in it.

At the documented ITERS_PER_SLOT=200 a slot writes ~203 lines × ~11 B ≈ 2.2 KB — fine. At the 2000/slot this PR's own local-numbers table used it is ~22 KB, so bufio flushes ~5 times inside the timed window: the write syscall the comment says was removed is back, just at 1/370th the rate.

Sizing it from the plan makes the comment true at any count:

w := bufio.NewWriterSize(times, 32*calls+4096)

(calls is already computed just below — you'd want to hoist it one line.) Small thing, but the whole binary exists to keep driver cost out of that window.

[huang195]

suggestion — ^0\.[0-9]{3}$ pins the whole-millisecond part to 0, not just the fractional precision. One Exec against the in-process fake crossing 1 ms — a loaded CI runner, a GC pause, a scheduler hiccup — fails this test with "does not have the sub-millisecond shape 0.XXX", which is the opposite of the bug it exists to catch.

The truncation defect is fully caught by the two properties that don't depend on wall-clock luck:

if !msFieldShape.MatchString(msField) { ... }   // three decimal digits present
if msField == "0.000" { ... }                    // and not truncated to zero

msFieldShape is already in this file. Against the pre-fix code the field was the bare integer 0, so it fails the first check — the mutation-test property you were after holds without the flake.

[huang195]

nit — gating the err-file write on o.errMsg != "" drops exactly the case the code just went out of its way to classify correctly.

An in-stream ExecEvent.error with an empty message (the one TestDriveClassifiesAnEmptyMessageInStreamErrorAsFailure covers) records err / unknown in the times file and writes nothing to the err file — which is verbatim the state plan.validate's own errFile refusal warns about: "a failing Exec's message would be lost and execErrorsByCause would say unknown with nothing to look at".

Gating on the status instead makes the two agree:

if o.status == "err" {
    _, _ = fmt.Fprintf(errFile, "req %d: %s\n", s.ReqBase+1+uint64(i), o.errMsg)
}

An empty line under a req id at least tells the operator the relay sent an error with no message, rather than leaving them to infer it from a times file.

[huang195]

nit — the fixed port is a small flake surface. Two concurrent runs of this suite on one host, or a listener left behind by a SIGKILL (the new INT/TERM trap covers cancellation, but not -9 or an OOM kill), both surface as the null-responder came up: no — with the stale responder happily answering on 18447 and no hint in the output that the port was the problem.

If null-responder accepts --listen 127.0.0.1:0 and logs the bound address, taking the port out of responder.log alongside the existing serving sandbox.v1.SandboxExec wait would remove the class entirely. If it doesn't, no action needed — worth a look since the wait loop is already parsing that file.

[pdettori]

Grazie — and thank you for going after the parts that undercut themselves. All seven items
addressed in 0054a58.

Two of the four inline catches were this branch's own additions contradicting their own purpose, which
is the most useful kind of review to get:

The buffer wasn't sized for the runs this PR documents. You're right that bufio.NewWriter's
default 4096 makes "flushed once at the end" true at ITERS_PER_SLOT=200 and false at the 2000/slot
my own local-numbers table used — about five flushes inside the timed window, in a binary whose
entire reason for existing is keeping syscalls out of it. Now bufio.NewWriterSize(times, 32*calls+4096)
with calls hoisted, and the comment says it is sized from the plan so the claim holds at any count
rather than only at the documented default.

The sub-millisecond test was a flake pointing the wrong way. ^0\.[0-9]{3}$ pinned the
whole-millisecond part to 0, so one Exec crossing 1 ms on a loaded runner would have failed the very
test meant to catch truncation. Now the two luck-independent properties: msFieldShape (already in
that file, so no second regex) for three decimal digits, and != "0.000" for not-truncated. I proved
it still catches the defect rather than assuming — reverted line() to the old %d whole-ms form
behind a file copy, confirmed the test fails with "0" does not have three decimal digits, restored,
and confirmed git diff showed no revert residue.

The err-file gate dropped exactly the case the code had just fixed. Gating on o.errMsg != ""
meant an empty-message ExecEvent.error recorded err/unknown in the times file and wrote nothing
to the err file — verbatim the state plan.validate's own errFile refusal warns about. Now gated on
o.status == "err", with a test asserting the line appears.

Ephemeral port. Verified before changing anything that null-responder --listen 127.0.0.1:0 binds
and logs on 127.0.0.1:54560, so this wasn't conditional after all. The seam test now binds :0,
parses the bound address out of the same serving sandbox.v1.SandboxExec line the wait already polls,
and refuses an empty parse rather than handing the driver a blank target. The stale-listener and
concurrent-run classes are gone, including the -9/OOM case the trap can't cover.

require_tool grpcurl's reason now says converge always uses grpcurl regardless of
SH_E11_EXEC_CLIENT and that the Go path only replaces the timed Exec loop. A block comment two lines
above carried the same misleading claim, so that was reworded too.

Both description items

The stale base-branch note is removed — #293 merged, and the PR now shows its true 15 files.

And you were right to push on the other one: "byte-identical in substance … additions only" no longer
held, and I'd left it standing. The description now states plainly that the driving mechanism is
unchanged but grpc_exec_record's latency arithmetic was substantively edited, why both paths had
to move together, and the consequence — grpcurl ladders published before this change carry integer
p95Ms, new ones carry fractional. That belonged in the description, not just the spec.

Gates after all of it: gofmt/go vet clean, 17 Go tests pass, shellcheck -x -S warning clean on
both shell files, Total failures: 0.

One housekeeping note: the fix commit initially picked up a stray Co-Authored-By trailer alongside
Assisted-By, which this repo's CLAUDE.md forbids. Caught before pushing and corrected via a soft
reset — same tree, message only. All 24 commits now carry Assisted-By and Signed-off-by alone.

🤖 Generated with Claude Code

[pdettori]

CI proto failure: not from this branch, and re-run

proto failed on 0054a58. I chased it rather than re-running blind, and it is a pre-existing
timing flake in internal/vmpool, not something this branch caused.

The failure:

--- FAIL: TestAnExecWaitingOnTheRunGateCanStillBeCancelled/timeout (2.20s)
    serialize_test.go:208: condition not met within 2s

Why it cannot be this branch's: the diff changes zero Go files outside
remote-worker/cmd/exec-driver/ — no go.mod, no go.work, nothing under internal/. There is no
causal path from a new package under cmd/ to a test in internal/vmpool. proto also passed on the
previous head af1c366, and 0054a58 only touched drive.go, drive_test.go, e11-density.sh and
its test file.

What it actually is. waitFor (replenish_test.go:184) is a fixed 2-second wall-clock
deadline polling every 1 ms, and it fails with a bare t.Fatal("condition not met within 2s") —
no mention of which condition or what value it observed, which is why the failure reads more alarming
than it is. The failing assertion is one of two such waits on pool state right after a fake-clock
advance.

Measured, not assumed. 20 runs on unmodified rossoctl/main: all pass. 60 runs on this branch
(3 × 20): all pass. I did reproduce it locally once, while the machine was busy — and that single
observation briefly led me to the wrong conclusion, that a local repro ruled out load. It doesn't;
one sample can't distinguish the two. The 60 subsequent passes are what settle it.

I have re-requested the job. If it goes green, nothing here needs changing.

Worth fixing in main, separately. The helper is load-sensitive by construction: a hard 2 s budget
on CI runners of unknown contention, and a message that tells you nothing about what didn't happen.
Two cheap improvements, neither belonging in this PR:

• make the deadline overridable (or scale it), so a slow runner doesn't read as a logic failure;
• have waitFor take a description and report the observed value, so condition not met within 2s
  becomes liveCount never reached 1 (last saw 0).

Happy to file that as a follow-up if you want it tracked — say the word and I will, alongside #295.

🤖 Generated with Claude Code

[pdettori]

proto re-run: success. All 13 checks green on 0054a58, and mergeable_state is now clean.

As expected from the 80 local runs (20 on main, 60 on this branch), the failure was the waitFor flake and not this diff. The two helper improvements I suggested above still stand as a follow-up for main whenever someone wants it tracked.

Nothing outstanding from my side — the branch is ready to merge. The authoritative 72-core driver-control comparison remains the one thing this PR deliberately does not contain; docs/notes/e11-go-exec-client-comparison-runbook.md is the procedure, and §E11 stays marked under repair until it runs.

[pdettori]

Both follow-ups filed:

• vmpool tests: waitFor has a hard 2s budget and an undiagnosable failure message, so load flakes read as regressions #297 — vmpool tests: waitFor's hard 2 s budget and undiagnosable message. Includes the measurements (20/20 on main, 60/60 on this branch), and two details that make it worth more than a re-run: the helper has 19 call sites across five test files, not one, and it surfaces in the proto job — which runs go test ./..., so the failure looks like a codegen problem and is not.
• #298 — the bash/Go cause-vocabulary mirror, from @huang195's review. On checking, the gap is slightly wider than the review assumed: cause_test.go references e11-density.sh only in a comment and hardcodes its expectations, and the bash suite has no cause assertion at all — the check labelled DRIFT GUARD covers execTimeoutS, not the causes. So the mirror rests on a comment and reviewer diligence today. The issue proposes a cross-language drift test as the first move, since unlike a shared data file it closes the risk without touching the reference path and can land before the metal comparison rather than after.

Neither belongs in this PR. Nothing outstanding here — all 13 checks green on 0054a58, mergeable_state: clean.