Pin eval model to DeepSeek V4.1 Flash

[gontzess]

Summary

Pin the eval scenarios to the current model generation.

• All five evals/scenarios/*.json now pin together/deepseek-ai/DeepSeek-V4.1-Flash (was together/deepseek-ai/DeepSeek-V4-Flash-0731); reasoningEffort stays high.
• evals/runner/scenario.test.ts real-scenario model assertions updated to match.
• evals/README.md states that the scenarios are pinned to V4.1-Flash while the committed reference (evals/results/baseline.json) was recorded on V4-Flash-0731.

Baseline

Re-record the baseline at the pinned model before comparing new runs against it. baseline.ts validates model_version consistency across the records it reads. This PR leaves the committed reference unchanged.

Validation

• npm run eval:test: 139 passed, 0 failed.
• npm run typecheck, npm run typecheck:decl, npm run typecheck:parity: passed.

[github-actions]

Superseded — see the current review report for commit b52e0f0e363b

General PR Review: Pin eval model to DeepSeek V4.1 Flash

Blocking Issues: 0 | Suggestions: 2 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base 8953c5efd271.
Review mode: full
View review run

Review Summary

The PR switches the model field in all five scenario JSONs (tier1-directory, tier1-directory-guide-only, tier1-directory-full, pre1-directory-proceed, pre1-noiam-park) from DeepSeek-V4-Flash-0731 to DeepSeek-V4.1-Flash. reasoningEffort stays high. It adds the same "Do not delegate or spawn sub-agents" line to both the pre-1 and funnel prompts in buildPrompt, updates the two real-scenario model checks in scenario.test.ts, and adds a README note about re-recording the baseline.

I scanned the full PR diff for security and correctness and checked it against baseline.ts. That file's cross-record model_version check (baseline.ts:198-209) works as the PR describes. The committed baseline.json is left on the old model on purpose. The remaining V4-0731 strings in unit-test fixtures are self-contained and don't depend on the scenario files.

No repo-local criteria loaded, so this review used only the base criteria.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

• New (confidence: medium) — evals/runner/agent.ts:23,43: no test covers the new no-delegation instruction. agent.test.ts checks other prompt guardrails with includes (lines 49-105) but not this one, so if the line is removed later, no test fails.
• New (confidence: medium) — evals/README.md:202: this line says the CXF-217 baseline is pinned to V4.1-Flash, but evals/results/baseline.json:4 and the CXF-233 "Reference status" section (README.md:244) still say V4-Flash-0731. Under the regression-gate contract, until the baseline is re-recorded, V4.1 runs would be compared against a reference recorded on a different model.

Prompt for AI agents

Verify each finding against the current code and only fix it if needed.

## Suggestions

In `evals/runner/agent.test.ts`:
- Around lines 49-59 (funnel prompt test) and 97-105 (pre-1 prompt test): Add `assert.ok(prompt.includes("Do not delegate or spawn sub-agents; do the work yourself."))` to both tests. This covers the new instruction in `buildPrompt` (`evals/runner/agent.ts` lines 23 and 43) so it can't be dropped without a test failing.

In `evals/README.md`:
- Around line 200-203: Rewrite the sentence so it doesn't say the CXF-217 baseline was recorded on V4.1-Flash. Say that the scenario files are pinned to `together/deepseek-ai/DeepSeek-V4.1-Flash`, while the committed `evals/results/baseline.json` was recorded on `DeepSeek-V4-Flash-0731`, matching the CXF-233 section near line 244. Optionally, state that the regression gate must refuse to compare when `baseline.json.model` differs from the scenario model.

[github-actions]

Superseded — see the current review report for commit 0f5248624b71

General PR Review: Pin eval model to DeepSeek V4.1 Flash

Blocking Issues: 0 | Suggestions: 2 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base 8953c5efd271.
Review mode: full
View review run

Review Summary

The PR switches the model field in all five scenario JSONs (tier1-directory, tier1-directory-guide-only, tier1-directory-full, pre1-directory-proceed, pre1-noiam-park) from DeepSeek-V4-Flash-0731 to DeepSeek-V4.1-Flash. reasoningEffort stays high. It adds the same "Do not delegate or spawn sub-agents" line to both the pre-1 and funnel prompts in buildPrompt, updates the two real-scenario model checks in scenario.test.ts, and adds a README note about re-recording the baseline.

I scanned the full PR diff for security and correctness and checked it against baseline.ts. That file's cross-record model_version check (baseline.ts:198-209) works as the PR describes. The committed baseline.json is left on the old model on purpose. The remaining V4-0731 strings in unit-test fixtures are self-contained and don't depend on the scenario files.

No repo-local criteria loaded, so this review used only the base criteria.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

• New (confidence: medium) — evals/runner/agent.ts:23,43: no test covers the new no-delegation instruction. agent.test.ts checks other prompt guardrails with includes (lines 49-105) but not this one, so if the line is removed later, no test fails.
• New (confidence: medium) — evals/README.md:202: this line says the CXF-217 baseline is pinned to V4.1-Flash, but evals/results/baseline.json:4 and the CXF-233 "Reference status" section (README.md:244) still say V4-Flash-0731. Under the regression-gate contract, until the baseline is re-recorded, V4.1 runs would be compared against a reference recorded on a different model.

Prompt for AI agents

Verify each finding against the current code and only fix it if needed.

## Suggestions

In `evals/runner/agent.test.ts`:
- Around lines 49-59 (funnel prompt test) and 97-105 (pre-1 prompt test): Add `assert.ok(prompt.includes("Do not delegate or spawn sub-agents; do the work yourself."))` to both tests. This covers the new instruction in `buildPrompt` (`evals/runner/agent.ts` lines 23 and 43) so it can't be dropped without a test failing.

In `evals/README.md`:
- Around line 200-203: Rewrite the sentence so it doesn't say the CXF-217 baseline was recorded on V4.1-Flash. Say that the scenario files are pinned to `together/deepseek-ai/DeepSeek-V4.1-Flash`, while the committed `evals/results/baseline.json` was recorded on `DeepSeek-V4-Flash-0731`, matching the CXF-233 section near line 244. Optionally, state that the regression gate must refuse to compare when `baseline.json.model` differs from the scenario model.

---

Reviewed commit: b52e0f0e363b

[github-actions]

No blocking issues found — see the full review report

[github-actions]

Superseded — see the current review report for commit 0f5248624b71

General PR Review: Pin eval model to DeepSeek V4.1 Flash

Blocking Issues: 0 | Suggestions: 0 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base 8953c5efd271.
Review mode: full
View review run

Review Summary

This PR changes the model field from together/deepseek-ai/DeepSeek-V4-Flash-0731 to together/deepseek-ai/DeepSeek-V4.1-Flash in all five scenario files under evals/scenarios/, and no other scenario file exists there. It updates the two real-scenario assertions in evals/runner/scenario.test.ts to match. It also rewrites the README baseline paragraph: the scenarios are pinned to V4.1-Flash, while the committed evals/results/baseline.json (line 4) is still on V4-Flash-0731. That matches the unchanged CXF-233 "Reference status" section.

I scanned the full PR diff for security and correctness issues and found none. The README's claim that baseline.ts checks model_version consistency across records is accurate (evals/runner/baseline.ts:198-203). The README says the reference model must match the scenario model, but nothing in code enforces that. The README presents this as a requirement for the regression gate, which is a later PR, so it isn't a defect here. Synthetic fixtures in other tests (run.test.ts, record.test.ts, agent.test.ts, baseline.test.ts, tier0/driver.test.ts) still use the V4-Flash-0731 string. That's fine because they don't load the scenario files. No repo-local criteria loaded, so I applied only the base criteria.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

None.

Resolved prior findings

• Fixed: evals/README.md said the CXF-217 baseline was recorded on V4.1-Flash. evals/README.md:205-210 now correctly says the scenarios are pinned to V4.1-Flash while baseline.json was recorded on V4-Flash-0731, and tells readers to re-record the baseline before comparing runs against it.
• Obsolete: the earlier note about missing test coverage for the no-delegation instruction in evals/runner/agent.ts. That instruction is no longer in agent.ts, and this PR doesn't touch that file.

[github-actions]

General PR Review: Pin eval model to DeepSeek V4.1 Flash

Blocking Issues: 0 | Suggestions: 0 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base 8953c5efd271.
Review mode: full
View review run

Review Summary

This PR changes the model field from together/deepseek-ai/DeepSeek-V4-Flash-0731 to together/deepseek-ai/DeepSeek-V4.1-Flash in all five scenario files under evals/scenarios/, and no other scenario file exists there. It updates the two real-scenario assertions in evals/runner/scenario.test.ts to match. It also rewrites the README baseline paragraph: the scenarios are pinned to V4.1-Flash, while the committed evals/results/baseline.json (line 4) is still on V4-Flash-0731. That matches the unchanged CXF-233 "Reference status" section.

I scanned the full PR diff for security and correctness issues and found none. The README's claim that baseline.ts checks model_version consistency across records is accurate (evals/runner/baseline.ts:198-203). The README says the reference model must match the scenario model, but nothing in code enforces that. The README presents this as a requirement for the regression gate, which is a later PR, so it isn't a defect here. Synthetic fixtures in other tests (run.test.ts, record.test.ts, agent.test.ts, baseline.test.ts, tier0/driver.test.ts) still use the V4-Flash-0731 string. That's fine because they don't load the scenario files. No repo-local criteria loaded, so I applied only the base criteria.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

None.

Resolved prior findings

• Fixed: evals/README.md said the CXF-217 baseline was recorded on V4.1-Flash. evals/README.md:205-210 now correctly says the scenarios are pinned to V4.1-Flash while baseline.json was recorded on V4-Flash-0731, and tells readers to re-record the baseline before comparing runs against it.
• Obsolete: the earlier note about missing test coverage for the no-delegation instruction in evals/runner/agent.ts. That instruction is no longer in agent.ts, and this PR doesn't touch that file.

---

Reviewed commit: 0f5248624b71

[github-actions]

No blocking issues found — see the full review report