Skip to content

feat(core): signal-managed graceful shutdown for pipeTo - #211

Merged
mo4islona merged 1 commit into
mainfrom
bb/fix-the-issue-with-pipes-sdk-210-thr_jzswe86xxx
Nov 6, 2026
Merged

mo4islona merged 1 commit into
mainfrom
bb/fix-the-issue-with-pipes-sdk-210-thr_jzswe86xxx

Conversation

@mo4islona

@mo4islona mo4islona commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Signal-managed graceful shutdown for pipeTo

Closes #210.

Summary

  • pipeTo intercepts SIGTERM and SIGINT by default for every target (ADR-24): it cancels only the source side, so an in-flight commit or fork rollback runs on; the batch in flight is committed, the cursor is saved, resources close, and the promise rejects with PipelineStoppedError (E3001) — a stop must not look like a finished range
  • shutdown: fn | { timeout, onShutdown } | false — a cleanup callback, a '30s'-style budget (E3002 PipelineShutdownTimeoutError on expiry) and an explicit opt-out; invalid settings reject with ShutdownConfigurationError (E0005) before the pipe starts
  • Portal reads, streams and retry pauses end on the stop signal, including calls made through the cache and transformer hooks; the shared listener pair exists only while a managed run does, and the library never ends the process
  • Targets stop through their normal flush-and-close path: BigQuery closes its streams but leaves a caller's writer open, the Pub/Sub publisher no longer closes a client it does not own, ClickHouse drops its extra store.close()
  • Docs: docs/graceful-shutdown.md, ADR-24, specs 04/05/08/09/13/14/15, error registry (92 thrown = documented = registered)

Coverage

Full-suite v8 coverage on the remote box, base (origin/main) vs head:

Module Stmts Δ Branch Δ
All files 93.1% +0.5 90.3% +0.4
src/core 94.0% +2.5 94.6% -0.0
src/http-client 74.1% +3.9 78.3% +2.9
src/internal 95.9% +0.8 88.6% +3.0
src/portal-client 98.6% +0.0 93.8% +0.7
src/targets/bigquery 93.4% +0.0 91.3% +0.3
src/targets/clickhouse 95.5% -0.3 92.7% +0.4
src/targets/parquet 96.2% +0.0 91.6% +0.1
src/targets/pubsub 97.5% +0.0 90.3% +0.2

125 new tests (1543 → 1668).

Test plan

  • Full suite with services (Postgres, ClickHouse) on the remote box: 1668 passed, 37 skipped, 0 failed — including the previously flaky listener-count assertion (a legacy test dropped a managed pipeTo promise, which kept the shared pair registered past the test; it now awaits the run, and the opt-out assertion reads the count in a synchronous window around the call)
  • New unit suites: shutdown option parsing and drain/timeout behavior, portal-source shutdown integration (stop during startup, mid-drain, on shutdown callback ordering), HTTP and portal client abort semantics, target ownership (BigQuery streams, Pub/Sub client, ClickHouse, Parquet)
  • New conformance suites with real child processes: conformance/shutdown/signals.test.ts (both signals, budgets, double signal), conformance/pubsub/backfill-stop.test.ts (SIGTERM mid-backfill leaves a resumable cursor)
  • node spec/check-spec.mjs — error registry in sync
  • Biome, typecheck, build

🤖 Generated with Claude Code

@mo4islona
mo4islona force-pushed the bb/fix-the-issue-with-pipes-sdk-210-thr_jzswe86xxx branch 10 times, most recently from af3e52f to 7c0e3e6 Compare November 6, 2026 00:58
Handle SIGTERM and SIGINT by default, stop source reads, and finish pending
writes before cleanup. Support a timeout, cleanup callback and opt-out,
and distinguish interrupted runs from completed ranges.

Share one abort signal among active pipes, forward finality and cancellation
explicitly, and keep caller-owned clients open. A deadline limits waiting;
the callback runs after draining and resource cleanup settle.

Closes #210.

Co-Authored-By: Claude Code <noreply@anthropic.com>

Co-Authored-By: Codex <codex@openai.com>
@mo4islona
mo4islona force-pushed the bb/fix-the-issue-with-pipes-sdk-210-thr_jzswe86xxx branch from 7c0e3e6 to cca4f82 Compare November 6, 2026 14:15
@mo4islona
mo4islona merged commit 7c596db into main Nov 6, 2026
2 checks passed
@mo4islona
mo4islona deleted the bb/fix-the-issue-with-pipes-sdk-210-thr_jzswe86xxx branch November 6, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default graceful shutdown in pipeTo with callback, readable timeout, and opt-out

1 participant