Skip to content

feat(spec-review): collate-evidence --comment writes spec-review-verdict.json itself (FAFF-1467) - #1183

Merged
alechill merged 6 commits into
mainfrom
FAFF-1467-spec-review-collate-evidence-writes-verdict
Nov 5, 2026
Merged

alechill merged 6 commits into
mainfrom
FAFF-1467-spec-review-collate-evidence-writes-verdict

Conversation

@alechill

@alechill alechill commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Summary

faff spec-review collate-evidence --comment now writes the anchored spec-review-verdict.json itself, from the exact comment text it prints. A build that skips the prose step that pipes the comment to the writer can no longer drop the evidence.

  • Placement. A new placeSpecReviewEvidence check in evidence.js places the file from the scratch dir alone:

    • in run mode, into <run-dir>/<ISSUE>, through the unchanged guardWriterPaths;
    • in interactive mode, into .faff/spec-review/<ISSUE>.

    It refuses a symlinked .faff, .faff/spec-review or scratch dir.

  • One writer. writeSpecReviewVerdictTo is split out of writeSpecReviewVerdict, so both paths go through the same parse, redact, anchor and validate steps. The written file is byte-identical to what faff evidence write spec-review-verdict produces from the posted comment.

  • No lost comment. A failed write prints a stderr warning and never changes stdout or the exit code. After placement, a refusal removes any earlier file. A fault moves it to spec-review-verdict.stale.json.

  • Carry path. A new faff spec-review write-evidence --dir --file verb covers prep's paths-only carry.

  • Prose and docs. faff-prep/SKILL.md gains the two edits the spec specified. docs/guide/cli.md documents the side effect and the new verb.

  • Tracker access. No faff command reads or writes the tracker.

Deviations from the spec

  • Extra prose trims. faff-prep/SKILL.md also trims three unrelated restatements, without changing their meaning:

    • the confidence-gate composition parenthetical;
    • the plot-refuse analogy;
    • the self-review pointer.

    The spec says "no other prose changes". On the operator's instruction the trims went in anyway, so the spec's own edits fit under the zero-headroom prose-bytes baseline instead of raising it. The baseline is lowered from 104391 to 104335.

  • Subdirectory gap. collate-evidence run from a subdirectory with a relative --dir still exits 2: round reading is cwd-relative, which the spec leaves unchanged. Placement itself resolves against the repo root, and that is tested through write-evidence. This is recorded as possible follow-up work.

Verification

  • Gates: faff gates run passes all 9 rungs.
  • Tests: test/spec-review-evidence-write.test.mjs has 27 tests, all passing. The collate-evidence cases moved there out of the spec-review.js selftest. test/evidence-write.test.mjs is unchanged and passes.
  • Stdout unchanged: main and this branch print a byte-identical collate-evidence --comment for the same rounds.
  • Code review: pass. 7 findings across 2 passes: 2 fixed in the second commit, 5 accepted with a reason. The adversarial second opinion came from a different model family.
  • Holdout: meets-spec on all 15 judged criteria, re-run after the review fix. The evaluator saw only the spec and the branch CLI, never the code.

Acceptance Criteria

  • faff-1467#done:1 collate-evidence --comment, run against a run-mode scratch, leaves <runDir>/<ISSUE>/spec-review-verdict.json on disk with no further command.
    Verified: test passes: run-mode --comment leaves //spec-review-verdict.json
    Method: test test/spec-review-evidence-write.test.mjs::run mode: --comment prints the comment, exits 0 and leaves the file, byte-equal to the evidence-write replay
  • faff-1467#done:2 The diff adds no require of node:http, node:https, node:net or any tracker adapter module to spec-review.js or evidence.js.
    Verified: test passes: no node:http/https/net or tracker require in either module; git diff adds only require("./evidence")
    Method: test test/spec-review-evidence-write.test.mjs::the diff adds no network or tracker require to the two modules
    Method: command git diff origin/main -- plugin/skills/faff/bin/lib/spec-review.js plugin/skills/faff/bin/lib/evidence.js | grep -E '^\+.*require\('
  • faff-1467#done:3 guardWriterPaths is byte-unchanged, and faff evidence write spec-review-verdict with a --run-dir outside .faff/runs still exits 2.
    Verified: guardWriterPaths body identical to origin/main; outside-runs replay exits 2 (test passes)
    Method: command diff <(git show origin/main:plugin/skills/faff/bin/lib/evidence.js | awk '/^function guardWriterPaths/,/^}/') <(awk '/^function guardWriterPaths/,/^}/' plugin/skills/faff/bin/lib/evidence.js)
    Method: test test/spec-review-evidence-write.test.mjs::faff evidence write spec-review-verdict still refuses a --run-dir outside .faff/runs with exit 2
  • faff-1467#done:4 placeSpecReviewEvidence returns mode: interactive for a direct child of a real .faff/spec-review/ and mode: run for <runDir>/<ISSUE>/spec-review with runDir under .faff/runs/. It throws WriterFault for any other directory, for a symlinked child that resolves outside, for a .faff or .faff/spec-review that is itself a symlink, and for an invalid issue id.
    Verified: tests pass: both modes placed; other dirs, symlinked child, symlinked .faff and .faff/spec-review, bad issue id throw WriterFault
    Method: test test/spec-review-evidence-write.test.mjs::placeSpecReviewEvidence throws WriterFault for every other directory
    Method: test test/spec-review-evidence-write.test.mjs::placeSpecReviewEvidence refuses a symlinked child, a symlinked .faff/spec-review and a symlinked .faff
  • faff-1467#done:5 A test in test/ asserts that, for both modes, placeSpecReviewEvidence(specReviewDir(issue, runDir)) returns the expected target_dir and mode, pinning the duplicated scratch layout.
    Verified: test passes: placeSpecReviewEvidence(specReviewDir(issue, runDir)) pinned for both modes
    Method: test test/spec-review-evidence-write.test.mjs::placeSpecReviewEvidence pins specReviewDir for both modes
  • faff-1467#done:6 writeSpecReviewVerdict(runDir, issue, …) delegates to writeSpecReviewVerdictTo, and every existing case in test/evidence-write.test.mjs passes unchanged.
    Verified: writeSpecReviewVerdict now returns writeSpecReviewVerdictTo(guardWriterPaths(...)); test/evidence-write.test.mjs unchanged, 14/14 pass
    Method: command node --test test/evidence-write.test.mjs
  • faff-1467#done:7 placeSpecReviewEvidence and writeSpecReviewVerdictTo are exported from evidence.js.
    Verified: both exported: typeof is function for each
    Method: command node -e 'const e=require("./plugin/skills/faff/bin/lib/evidence.js");console.log(typeof e.placeSpecReviewEvidence, typeof e.writeSpecReviewVerdictTo)'
  • faff-1467#done:8 SPEC_REVIEW_USAGE lists write-evidence --dir <scratch> --file <file|->.
    Verified: usage error output contains write-evidence --dir --file <file|->
    Method: command node plugin/skills/faff/bin/faff spec-review --bogus
  • faff-1467#done:9 collate-evidence --comment stdout is byte-identical before and after this change for the same inputs, and on success stderr includes exactly one line faff spec-review collate-evidence: wrote <path> (existing skipped … lines may also appear).
    Verified: origin/main and branch binaries print byte-identical stdout for the FAFF‑1467 rounds (cmp clean); exactly one wrote line asserted by test
    Method: command node plugin/skills/faff/bin/faff spec-review collate-evidence --dir <copy of .faff/spec-review/FAFF‑1467 rounds> --dispositions dispositions.json --comment, run from main and from the branch, then cmp
    Method: test test/spec-review-evidence-write.test.mjs::stdout of --comment is the rendered comment alone, and the file digest matches its block
  • faff-1467#done:10 On a placement fault or a writer refusal, collate-evidence --comment exits 0, prints the comment, and stderr includes exactly one line beginning faff spec-review collate-evidence: warning: spec-review-verdict.json not written:.
    Verified: tests pass: exit 0, comment printed, exactly one warning line on placement fault and refusal
    Method: test test/spec-review-evidence-write.test.mjs::a symlinked .faff/spec-review warns, exits 0, prints the comment and writes nothing in the link target
    Method: test test/spec-review-evidence-write.test.mjs::a run dir outside .faff/runs warns and still prints the comment
  • faff-1467#done:11 After a placed-but-refused write from either command, no spec-review-verdict.json remains in the target. After a placed write that faults, an earlier file is at spec-review-verdict.stale.json, not the canonical path.
    Verified: tests pass: refusal removes the earlier file; a write fault moves it to spec-review-verdict.stale.json
    Method: test test/spec-review-evidence-write.test.mjs::write-evidence: a digest mismatch exits 1 with refused: and removes an earlier file
    Method: test test/spec-review-evidence-write.test.mjs::write-evidence: a refusal into an interactive scratch removes the earlier file
    Method: test test/spec-review-evidence-write.test.mjs::a write fault moves an earlier file to spec-review-verdict.stale.json and the warning is printed
  • faff-1467#done:12 A relative --dir of .faff/spec-review/<ISSUE> places correctly when the command runs from a subdirectory of the repository.
    Verified: test passes: relative .faff/spec-review/ places from a subdirectory (exercised via write-evidence; collate's round read stays cwd-relative, unchanged)
    Method: test test/spec-review-evidence-write.test.mjs::a relative --dir places correctly when the command runs from a subdirectory
  • faff-1467#done:13 collate-evidence without --comment writes no file.
    Verified: test passes: plain mode writes no file in either layout
    Method: test test/spec-review-evidence-write.test.mjs::plain mode (no --comment) writes no file in either layout
  • faff-1467#done:14 The --from-judge --spec … --comment path writes the same file as the --dispositions path when both resolve to the same dispositions.
    Verified: test passes: --from-judge and --dispositions files are byte-equal
    Method: test test/spec-review-evidence-write.test.mjs::the --from-judge path writes the same file as the --dispositions path
  • faff-1467#done:15 write-evidence exits 0 and writes the file for a carry comment. It exits 1 with refused: for a digest mismatch or a missing section, and exits 2 for missing flags, an unreadable --file, or a placement fault.
    Verified: tests pass: exit 0 carry (file and stdin), exit 1 refused (digest, missing section), exit 2 (flags, unreadable file, placement fault)
    Method: test test/spec-review-evidence-write.test.mjs::write-evidence: exit 0 and the path on stdout for a carry comment, from a file and from stdin
    Method: test test/spec-review-evidence-write.test.mjs::write-evidence: a refusal with no earlier file writes nothing; a missing section exits 1
    Method: test test/spec-review-evidence-write.test.mjs::write-evidence: missing flags, an unreadable --file and a placement fault exit 2
  • faff-1467#done:16 The file collate-evidence writes is byte-identical to the file faff evidence write spec-review-verdict writes from its stdout.
    Verified: tests pass: written file byte-equal to the evidence-write replay of stdout
    Method: test test/spec-review-evidence-write.test.mjs::run mode: --comment prints the comment, exits 0 and leaves the file, byte-equal to the evidence-write replay
    Method: test test/spec-review-evidence-write.test.mjs::the integration smoke: a replay of the printed comment leaves the written file unchanged
  • faff-1467#done:17 faff-prep/SKILL.md :263 carries the two in-place replacements, and :553 carries the write-evidence call. No other prose changes.
    Verified: git diff shows both :263 replacements and the :553 write-evidence call; three restatements elsewhere were also trimmed at the operator's direction to hold the prose-bytes baseline
    Method: command git diff origin/main -- plugin/skills/faff-prep/SKILL.md
  • faff-1467#done:18 faff-prep/SKILL.md stays at or under 611 lines, its :263 line stays at or under 200 words, and faff validate-adapters passes with SKILL_LINE_BASELINE unchanged.
    Verified: 611 lines; :263 is 198 words; validate-adapters PASS; SKILL_LINE_BASELINE unchanged; prose-bytes baseline lowered 104391 -> 104335
    Method: command node plugin/skills/faff/bin/faff validate-adapters
  • faff-1467#done:19 docs/guide/cli.md documents the collate-evidence side effect and the write-evidence row.
    Verified: collate-evidence row extended and write-evidence row added; lint-cli-doc rung passed in the gate ladder
    Method: command git diff origin/main -- docs/guide/cli.md
  • faff-1467#done:20 Running the terminal seam twice leaves one file whose bytes match the latest comment.
    Verified: test passes: second seam run leaves one file matching the latest comment
    Method: test test/spec-review-evidence-write.test.mjs::the terminal seam run twice leaves one file matching the latest comment
  • faff-1467#done:21 Round readers ignore spec-review-verdict.json in an interactive scratch. A test runs collateSpecReviewRounds on a dir that holds the file and reports no skip for it.
    Verified: test passes: collateSpecReviewRounds over a scratch holding the file and its temp sibling reports no skip
    Method: test test/spec-review-evidence-write.test.mjs::round readers ignore spec-review-verdict.json and its temp sibling in an interactive scratch
  • faff-1467#done:22 New tests live in test/, for example test/spec-review-evidence-write.test.mjs, run under node --test, and cover every item above using a temp git root with .faff/runs and .faff/spec-review, as test/evidence-write.test.mjs does.
    Verified: 26 tests in test/spec-review-evidence-write.test.mjs pass under node --test, using a temp git root with .faff/runs and .faff/spec-review
    Method: command node --test test/spec-review-evidence-write.test.mjs
  • faff-1467#done:23 The collate-evidence cases in the spec-review.js selftest (:240-259) move into test/ and are deleted from the selftest block. No new selftest cases are added to either module.
    Verified: collate cases deleted from specReviewSelftest (0 collate references remain in it) and moved into test/
    Method: test test/spec-review-evidence-write.test.mjs::collation: an objection-bearing round before a clean approve is kept, a malformed round is skipped, the plain CLI exits 0 (moved from the selftest)
  • faff-1467#done:24 node --test over test/ and faff regions check pass.
    Verified: faff gates run UNIT rungs pass; faff regions check exits 0
    Method: command faff gates run --json --resume
    Method: command node plugin/skills/faff/bin/faff regions check

Closes FAFF-1467

alechill and others added 4 commits November 4, 2026 21:09
…c-review-verdict.json itself, so a skipped build step cannot drop the evidence

Signed-off-by: Alec Hill <alec@shftwst.dev>
…ict.json itself (FAFF-1467)

The command that renders the spec-review evidence comment now also writes
the anchored spec-review-verdict.json from that exact text, through the
existing writer. A new placeSpecReviewEvidence check places the file from
the scratch dir: <run-dir>/<ISSUE> in run mode (through the unchanged
guardWriterPaths), or .faff/spec-review/<ISSUE> for interactive prep.
A failed write warns on stderr and never changes stdout or the exit code.
A new spec-review write-evidence verb covers prep's paths-only carry.

faff-prep/SKILL.md carries the two spec-mandated edits; three unrelated
restatements were trimmed so the file stays under its prose-bytes
baseline, which is lowered to the new size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alec Hill <alec@shftwst.dev>
…ent names an unresolvable root (FAFF-1467)

From the adversarial review of FAFF-1467: `write-evidence --file -` waited
forever on a terminal stdin, and placement reported an unresolvable
repository root as ".faff/spec-review resolves outside the repository".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alec Hill <alec@shftwst.dev>
Signed-off-by: Alec Hill <alec@shftwst.dev>
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Ready to land. Run this locally in a real terminal:

faff effects declare --run-dir .faff/anchors/run-20261104-210928-graft-FAFF-1467 --issue FAFF-1467 --step merge <<'EOF'
[{"kind":"merge","target":"pr:1183","reversible":true}]
EOF
faff merge-gate --pr 1183 --issue FAFF-1467 --run-dir .faff/anchors/run-20261104-210928-graft-FAFF-1467 --level L2 \
  --execute --merge-args "--squash --delete-branch"

alechill and others added 2 commits November 5, 2026 05:02
…rose edits

The rule-inventory check reached main while this branch was open. It
needs the spec to map each deleted faff-prep/SKILL.md line. The amended
spec is re-posted on the issue, and this commits its builder view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alec Hill <alec@shftwst.dev>
@alechill
alechill merged commit 1a918e4 into main Nov 5, 2026
14 checks passed
@alechill
alechill deleted the FAFF-1467-spec-review-collate-evidence-writes-verdict branch November 5, 2026 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant