Torrent v2 1/8: Contracts, verification infrastructure and resource limits

.github/workflows/tests.yml

@@ -4,7 +4,7 @@ on:
   push:
     branches: [ main ]
   pull_request:
-    branches: [ main, "codex/torrent-*" ]
+    branches: [ main, "codex/torrent-*", "torrent-v2-*" ]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -18,7 +18,7 @@ permissions:
 jobs:
   jvm-tests:
     name: JVM Tests
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     steps:
       - uses: actions/checkout@v6
 
@@ -29,13 +29,41 @@ jobs:
 
       - uses: gradle/actions/setup-gradle@v5
 
-      - name: Install independent torrent client
-        run: sudo apt-get update && sudo apt-get install -y transmission-daemon
+      - name: Install fixture build prerequisites
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y cmake build-essential libcurl4-openssl-dev libssl-dev
+
+      - name: Cache pinned Transmission fixture
+        uses: actions/cache@v4
+        with:
+          path: ${{ runner.temp }}/ketch-transmission
+          key: transmission-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('test-fixtures/torrent/clients.properties', 'tools/torrent/build_transmission.py') }}
+
+      - name: Build authenticated independent torrent client
+        run: python3 tools/torrent/build_transmission.py --output "$RUNNER_TEMP/ketch-transmission"
+
+      - name: Verify conformance gate behavior
+        run: python3 -m unittest discover -s tools/torrent -p 'test_*.py'
 
       - name: Run JVM tests
         env:
-          TRANSMISSION_DAEMON: /usr/bin/transmission-daemon
-        run: ./gradlew jvmTest :library:torrent:verifyNoNativeTorrentRuntime
+          TRANSMISSION_DAEMON: ${{ runner.temp }}/ketch-transmission/build/daemon/transmission-daemon
+        run: ./gradlew jvmTest :library:torrent:verifyNoNativeTorrentRuntime -PtorrentConformance=true
+
+      - name: Require executed interoperability scenarios
+        run: |
+          python3 tools/torrent/verify_conformance.py \
+            --reports library/torrent/build/test-results/jvmTest \
+            --revision "$GITHUB_SHA" \
+            --output library/torrent/build/reports/conformance/executed.json
+
+      - name: Upload conformance evidence
+        uses: actions/upload-artifact@v7
+        with:
+          name: torrent-conformance
+          path: library/torrent/build/reports/conformance/executed.json
+          if-no-files-found: error
 
       - name: Upload test results
         if: always()
@@ -173,6 +201,24 @@ jobs:
           name: test-results-torrent-device
           path: library/torrent/build/outputs/androidTest-results/**/TEST-*.xml
 
+  required-checks:
+    name: Required Checks
+    runs-on: ubuntu-24.04
+    needs: [ jvm-tests, android-tests, ios-tests, js-tests, torrent-desktop-tests, torrent-device-tests, publish-results ]
+    if: always()
+    steps:
+      - name: Require every test job to succeed on this revision
+        env:
+          JOB_RESULTS: ${{ toJSON(needs) }}
+        run: |
+          python3 - <<'PY'
+          import json, os
+          results = json.loads(os.environ['JOB_RESULTS'])
+          failed = {name: value['result'] for name, value in results.items()
+                    if value['result'] != 'success'}
+          assert results and not failed, f'Required checks did not pass: {failed}'
+          PY
+
   publish-results:
     name: Publish Test Results
     runs-on: ubuntu-latest
@@ -183,9 +229,12 @@ jobs:
         uses: actions/download-artifact@v8
         with:
           pattern: test-results-*
-          merge-multiple: true
+          # Host jobs use identical report paths. Keep each artifact in its own directory.
+          merge-multiple: false
 
       - name: Publish test results
         uses: EnricoMi/publish-unit-test-result-action@v2
         with:
           files: '**/TEST-*.xml'
+          action_fail: true
+          action_fail_on_inconclusive: true

docs/design/torrent-control-contract.md

@@ -0,0 +1,102 @@
+# Torrent control protocol, version 1
+
+This contract implements the API boundary decision in [roadmap #162][roadmap]. The types live in
+`library:api`, so WebAssembly and remote clients can use them without linking a torrent engine.
+This PR defines inspection, capability negotiation, ordering, and command preconditions. Runtime
+wiring and the typed mutation methods ship with their capabilities, with SDK/HTTP/SSE parity checked
+in roadmap step 27. A contract declaration is not an implemented runtime capability.
+
+## Availability and negotiation
+
+`KetchApi.torrents` is nullable and defaults to null for existing backend implementations. Null means
+no typed control protocol, not no torrent downloader. Local and remote adapters expose a controller
+only when they implement inspection. They advertise only executable features. The backend's
+platform determines capabilities: a browser connected to a daemon can have capabilities unavailable
+in a local browser. Protocol major versions other than 1 disable all known features in this SDK.
+Unknown capability strings survive decoding. New mandatory semantics require a new major version;
+optional fields may be added without changing the interpretation of existing fields.
+
+Page sizes and subscription limits are explicit bounded ceilings. Oversized requests fail rather
+than being silently truncated. Cursors bind to task, authenticated principal, filters, sort, and
+revision. Mutations that invalidate a page return a stale-cursor error; no mixed-revision file list.
+Peer addresses and other sensitive details are requested separately, not broadcast in every event.
+
+## State and completion
+
+`TorrentSnapshot` is a bounded aggregate. Metadata-unavailable counters are null, not invented zeroes.
+Selected verified bytes, wanted bytes, payload received, uploaded payload, discarded payload, and
+protocol overhead are separate. Virtual padding never counts toward user payload. A recheck can
+reduce verified progress. Counters cannot exceed their content bounds or become negative.
+
+A selection has a monotonically increasing generation. Completion belongs to that generation and
+can coexist with seeding. Expanding a completed selection requires an explicit restart; existing
+`awaitCompletion` callers retain their original generation. Empty selections can complete once
+metadata is known. They do not imply possession of the torrent's payload. Download queue slots and
+seed slots remain separate. Legacy upload remains disabled unless explicitly enabled; the named
+production profile can enable uploads while downloading. Post-completion seeding remains opt-in.
+
+## Revision and reconnect ordering
+
+Every published task state has an opaque catalog epoch and a nonnegative sequence. Sequences never
+wrap. A backend changes the epoch if it loses monotonic state, including restoration of an older
+catalog. Epoch strings are compared for equality, never sorted. Full snapshots can jump forward;
+deltas require their exact predecessor. Older or duplicate same-epoch messages are ignored. Gaps
+in delta history and epoch changes require an authoritative resync.
+
+`compareIncoming` implements this merge decision. It assumes messages belong to the active
+connection generation. Adapters separately reject late responses from old connections. Reconnect
+cancels the old subscription, fetches authoritative state, and installs the new epoch under a new
+connection generation. An old HTTP response cannot overwrite a newer event in the same epoch.
+An HTTP response from an obsolete connection cannot reset the new epoch.
+
+Inspection streams emit full snapshots and conflate slow consumers to the latest snapshot. Null
+means a tombstone and completes the task stream. A tombstone is terminal for that task ID, which is
+never reused. The client rejects all subsequent responses for that removed task in the current
+connection generation. Transport failures terminate observation; reconnect is explicit. Persisted
+idempotency outcomes for removed tasks remain available during the supported retry window.
+
+## Mutations and operations
+
+Each existing-task mutation carries `TorrentCommandContext`: an idempotency key and the expected
+revision. Scope the key to principal and task. Check the persisted retry ledger before revision
+validation: an exact retry returns its original result, even if state has since advanced. Reusing
+a key for a different canonical request fails. New requests with stale revisions return a conflict
+and current revision, without partial mutation. Concurrent duplicates execute only once.
+
+Long-running operations return an operation ID with queued/running/succeeded/failed/canceled state,
+progress, and cancellation support. Acceptance never implies completion. A mutation's committed
+outcome and idempotency record are persisted atomically. Ledger capacity is bounded; reject new
+admissions rather than evicting entries still inside the advertised retry window. After expiration,
+a retry fails explicitly instead of unexpectedly executing an old destructive command again.
+
+The typed command families and required semantics are:
+
+| Command family | Contract |
+| --- | --- |
+| Selection | Stable file IDs; priorities; sequential mode; verified-range deadlines; explicit restart |
+| Transfer | Download/upload limits; pause/resume; runtime admission applies to both directions |
+| Seeding | Ratio and duration goals; start/stop; separate seed queue; no implicit upload opt-in |
+| Integrity | Recheck/import as cancelable operations; publish only verified state |
+| Trackers | Edit ordered tiers and reannounce; retain private/proxy policy and credential context |
+| Storage | Move/rename as recoverable operations; reject unsafe paths and ownership conflicts |
+| Removal | Explicit keep-data or remove-owned-data policy; never delete unrelated files |
+| Export | Export metainfo/magnets without publication, tracker requests, or starting seeding |
+| Creation | Stable source snapshot, format and piece policy; cancelable; no implicit publication |
+| Streaming | Authenticated task/file/range reads; only verified ranges; bounded lifetime and buffers |
+
+Creation has no existing task revision; its request is scoped to the principal's creation ledger.
+Operation cancellation has its own idempotency key and operation revision. Errors distinguish
+unsupported capability, invalid input, conflict, policy denial, resource exhaustion, storage failure,
+integrity failure, not found, and cancellation. They must not expose tracker credentials or raw
+untrusted paths. Unsupported requests fail before any side effect or network access.
+
+## Migration gates
+
+Existing `KetchApi` implementations compile with the default null controller. New clients connected
+to old daemons keep legacy downloads working and hide unsupported controls. Adding this API does not
+change the existing resume format. Checkpoint v2 migration retains old state until verified commit,
+rehashes legacy v1 data, and preserves unsupported native blobs for explicit recovery. No optimistic
+conversion may claim unverified bytes. Format identity and checkpoint implementation remain in
+roadmap steps 04–09. These gates are pending until exercised by their implementation tests.
+
+[roadmap]: https://github.com/linroid/Ketch/issues/162

docs/design/torrent-resource-accounting.md

@@ -0,0 +1,38 @@
+# Torrent resource accounting
+
+The v2 roadmap requires simultaneous limits for memory, peers, active tasks, payload handles,
+metadata, files, pieces, and hash layers. The current foundation provides some of these limits;
+it does not yet implement or certify the complete mobile and desktop production profiles.
+
+## Payload handles
+
+`TorrentConfig.maxOpenPayloadFiles` bounds aggregate open payload handles across an engine's
+sessions. It defaults to 32 and accepts values from 1 through 128. The engine gives every piece
+store the same coroutine semaphore. Waiting is cancellable and happens before dispatch to the
+blocking I/O executor, so a waiting session does not occupy an I/O thread or open a payload file.
+
+A piece-store operation opens at most one payload handle at a time, including the boundary-piece
+sidecars used for partial selection. A semaphore permit spans the whole storage operation, rather
+than an individual open/close pair. Initialization, verification, reads, commits, final truncation,
+and cleanup consequently share admission. Long scans can hold a permit across multiple sequential
+file opens; the implementation favors a strict bound over maximum concurrency.
+
+Cancellation cannot release a permit while a blocking provider call is still executing. The
+permit returns only after the I/O block returns or throws and its scoped handles close. Canceling
+a waiter removes that wait without taking a permit. A provider failure returns the permit so
+another torrent can proceed. This does not make a blocked filesystem provider cancellable or prove
+the production pause/stop latency gate.
+
+Ownership journals and checkpoints can open an additional non-payload handle while a payload
+handle is open. They are outside this payload ceiling, as are sockets and platform/runtime file
+descriptors. Process descriptor counts must be measured separately. Recovery-only stores used by
+source cleanup read ownership records and remove files; they do not open payload handles.
+
+## Remaining production work
+
+The current buffer, metadata-cache, and session admission estimates are not a complete accounting
+of engine working memory and cannot establish a process RSS ceiling. Raw metainfo parsing still
+has limits below the proposed desktop profile. Separate seed admission, external hash-layer
+storage, complete parser/index accounting, and the named production profiles remain roadmap work.
+Physical-device, lifecycle, memory, descriptor, and soak evidence must cover the completed runtime
+before either profile is described as production-ready.