Eval system: next-iteration roadmap

[doodlewind]

Roadmap for iterating on the parsing-accuracy eval (the hard codex eval + the fixed rubric benchmark). Ordered by priority — from "do immediately" to "nice-to-have".

Context (where we are)

• The hard eval runs the real codex exec annotate pipeline on eval/eval-cases.json (43 cases, medium → very-hard) and scores byte-exact resolution.
• The instrument now reports two signals — reliability (codex accuracy) vs capability (can the library express it) — plus auto-located drift, cross-round per-case reliability, and a clickable parse modal (Eval tab).
• eval/gap-backlog.md tracks discovered ts-def gaps (fixed / won't-fix / open).
• Round 4 hit 100%, but that was N=1 — a single lucky-consistent sample. The cross-round view shows the truer per-case picture.

---

1. Immediate: multi-sample to turn the headline into trustworthy numbers

Round 4's 100% is one sample. Run a round at --samples 3–5 so every case gets a reliability band: solid (5/5) / flaky (2/5) / real gap (0/5). Until then every trend number still carries single-shot noise and shouldn't drive conclusions.

• Run eval-codex --samples 3 (or 5) on the full set; publish per-case reliability bands.

2. Core: couple the eval to the real course corpus (both directions)

Today the eval runs synthetic hard sentences, decoupled from product content.

• Forward: sample sentences from the 384 course snippets, run codex on them → measures "can codex reproduce what the course actually teaches" and catches drift between codex's analysis and the curated corpus.
• Reverse: feed gap-backlog library gaps back into the course/library — close .d.ts gaps so spots that today must be raw literals become properly modeled.
• Add a corpus-sampling mode to the eval (draw jp from chapter examples).
• Wire the gap-backlog → library fixes loop (gated against the full corpus).

3. Core: make the loop actually closed + add a CI regression gate

Each round is currently driven manually.

• Closed loop: one command → run → record → auto-classify failures (ts-def / prompt / unmodelable) → auto-apply only safe ts-def fixes (full-corpus gated) → open a PR for review; prompt changes stay as proposals, never auto-applied.
• CI gate: wire the medium set (the "should always pass" zone) into CI / a scheduled job so any regression in annotate.ts or the library is caught automatically (this was the point of building the medium tier).
• Single closed-loop entry script with gated auto-apply + PR.
• CI / cron regression gate on the medium tier.

4. Growth: rotate the case set + map coverage to grammar points

A fixed 43-case set will get overfit.

• Keep a frozen regression core + a periodically regenerated, rotating fresh batch.
• Map each eval case to the 47-chapter grammar points; auto-fill cases for any grammar point with no coverage, so capability measures generalization, not memorization.
• Frozen core + rotating fresh batch.
• Grammar-point ↔ eval-case coverage map + gap-filling.

5. Deepen: structured failure analysis + partial credit

Failures are currently binary + one drift span.

• Auto-classify drift (wrong conjugation / missing particle / wrong POS / dropped punctuation) and aggregate codex's top error classes, so fixes are prioritized by data instead of hand-debugging individual long sentences (diminishing returns).
• Partial credit (per-morpheme alignment) so small improvements register in the metric.
• Drift taxonomy + aggregated error-class report.
• Partial-credit scoring.

---

Open question (sets priority)

Is this eval primarily a quality gate for the course/library (then prioritize 3 + 4) or a research engine pushing codex+library capability (then prioritize 2 + 5)? Recommendation regardless: do 1 first (cheap, makes the numbers trustworthy), then 2 (lands value on the actual product).