feat(eval): compare models on one table, and stop pricing cloud runs at zero

[kovtcharov]

Comparing two models meant opening two scorecards side by side and doing the arithmetic by hand, and the one column you would reach for first was wrong: MODEL_PRICING listed only Claude, so every Fireworks run was costed at $0.00. A million input tokens on GLM 5.3 reported as free; it is $1.62. That is the worst shape a number can have — it looks measured, it sits in a column called cost, and nobody re-checks it.

gaia.eval.model_comparison renders one row per model: pass rate, quality, time, TTFT, throughput, steps, tool calls, tokens, cache share, cost, and cost per passing scenario — the column that actually decides a model, since one that is cheap per token and fails half the work is not cheap. It is a library building block for now (compare() / render_markdown()); no gaia subcommand calls it yet, and wiring one is a follow-up.

The table below was produced from a real 14-task sweep run by an external Fireworks benchmark harness (bench/run_task.py), which sums the agent conversation's per-step cached_tokens stats. The eval pipeline on this branch did not receive that field before review — it now reads every provider's spelling of it (below), and Lemonade forwarding it for Fireworks lands in #3739.

| Model                                             | Pass | Pass rate | Quality | Time    | Steps | Tools | Input | Cached | Output | Cost    | $/pass  |
|---------------------------------------------------|------|-----------|---------|---------|-------|-------|-------|--------|--------|---------|---------|
| fireworks.accounts/fireworks/routers/glm-5p2-fast | 8/14 | 57%       | —       | 17m 41s | 121   | 170   | 1.3M  | 77%    | 94.1k  | $1.4575 | $0.1822 |

Three things that column set is careful about:

• Caching is most of an agent run's bill. That sweep served 77% of its input tokens from cache; pricing them at the full input rate overstates the run more than twofold. Cached tokens are read from Lemonade's cached_tokens, Claude's cache_read_input_tokens, or the raw OpenAI prompt_tokens_details.cached_tokens, and bill at their own rate — an absent rate means "same as input", not "free".
• A metric nobody measured renders as —, never as 0. A run whose provider never reported a cache count shows — under Cached; a reported zero shows 0%.
• Local is free, unpriced cloud is unknown. A locally served model is a defined $0.00 and is named as local under the table; only a cloud-routed id (fireworks., amd., claude-) with no rate row shows no dollars, and it sorts after priced models rather than as the cheapest.

Models are priced from their own token counts rather than from whatever rates each scorecard happened to be written under — two models costed under two different tables is not a comparison.

Test plan

• pytest tests/unit/test_model_comparison.py tests/unit/eval/test_performance_extractor.py tests/unit/eval/test_quality_metrics.py — 112 pass, including cache counts fed from ClaudeProvider._capture_usage's real output, feat(tui): show what a session costs — time, steps, tools, tokens, dollars #3739's Lemonade usage shape, and a raw Fireworks usage body; — vs 0%; local $0.00 vs unpriced cloud; unpriced sorting last; explicit rates never borrowing the table's cached rate.
• pytest tests/ -k "eval or performance or scorecard or quality_metric or model_comparison or benchmark" — 833 pass, 14 skipped.
• Rates checked against the provider's published card (Fireworks read 2026-09-13); local models still cost 0.0 rather than falling through to the "default" row.

🤖 Generated with Claude Code

[github-actions]

Request changes

This adds a one-table model comparison and teaches the cost calculation about cached tokens — a real gap, and the pricing arithmetic in the table is right. But the cache half of it never actually runs: nothing in GAIA produces the token count the new code reads, so every run is priced as if nothing was cached.

The headline problem. The new "cached tokens" number is looked up under a name no provider in this repo actually uses. The cloud path doesn't report cache usage at all, and Claude reports it under a different name. The result is that the cache discount never applies and the comparison table prints a confident 0% cached for every run — a measured-looking zero for something that was never measured, which is the exact failure this PR sets out to fix, just pointed the other way (costs now read too high instead of free). Fix the plumbing that feeds the number before this ships, and add a test that exercises a real provider's usage shape rather than a hand-written one.

A locally-served model now reads as "we don't know what this costs." Local inference is a defined $0.00, not an unknown price, but the table drops it into the "no published rate" footnote alongside genuinely unpriced cloud models — and sorts it as though it were the cheapest option. For a project whose whole pitch is running locally, the table can't currently say that the local option is free.

No way to actually run it. The new comparison module isn't reachable from any gaia command and has no docs page, so the table in the PR description can't be reproduced by a reader. Either wire it to a subcommand (plus docs/reference/cli.mdx and docs/docs.json) or say plainly that it's a library-only building block for now.

Real-world evidence

No evidence-bundle.md was produced for this PR, and the change adds no CLI/API/UI surface to exercise — so the verdict rests on static review plus a local reproduction I ran against this checkout.

I fed the two provider usage dictionaries this repo actually builds through the new code path:

lemonade/fireworks cloud   -> StepResult.cached_tokens = 0
claude                     -> StepResult.cached_tokens = 0
scorecard total_cached_tokens = 0   (provider reported 77000 cache reads)

| Model               | Pass | Pass rate | Quality | Time  | Steps | Tools | Input | Cached | Output | Cost    | $/pass  |
|---------------------|------|-----------|---------|-------|-------|-------|-------|--------|--------|---------|---------|
| Gemma-4-E4B-it-GGUF | 1/1  | 100%      | 8.00    | 30.0s | 3     | 2     | 1.0M  | 0%     | 50.0k  | —       | —       |
| fireworks.glm-5p3   | 1/1  | 100%      | 8.00    | 30.0s | 3     | 2     | 1.0M  | 0%     | 50.0k  | $1.6200 | $1.6200 |

No published rate for Gemma-4-E4B-it-GGUF — tokens are measured, dollars are not guessed.

That's the first three findings in one run: the cache count is always zero, Cached renders 0% rather than —, and the local model is described as unpriced and sorted first as if free.

The PR description shows a sweep reporting 77% cached. I could not reproduce a non-zero cache share through this diff's code path — the token counts it reads are never populated. I'm not claiming the sweep didn't happen; please say which harness produced it, because it doesn't appear to be this one.

🔍 Technical details

🔴 Critical — cached_tokens is never populated, so cache-aware pricing is inert

src/gaia/eval/performance.py:275 reads stats.get("cached_tokens", 0), but no producer in the repo emits that key into performance_stats:

• LemonadeProvider._last_usage (src/gaia/llm/providers/lemonade.py:440) builds exactly {prompt_tokens, completion_tokens, total_tokens, tokens_per_second} — it never forwards the OpenAI-shape usage.prompt_tokens_details.cached_tokens that Fireworks returns. This is the path the new MODEL_PRICING rows target.
• ClaudeProvider._capture_usage (src/gaia/llm/providers/claude.py:635) emits cache_read_input_tokens / cache_creation_input_tokens.
• turn_metrics.py:198 uses input_tokens_cached, and it's a prefix-overlap estimate in a different dict.

Consequence: total_cached_tokens is always 0, compute_cost always bills the full input rate, and ModelRun.cached_share returns 0.0 (not None) so _fmt_pct prints 0% — contradicting the module docstring's own rule at model_comparison.py:81.

Note the existing convention at agent.py:483: step stats carry either input_tokens or prompt_tokens depending on backend, and readers accept both. The cached field needs the same treatment:

                        cached_tokens=(
                            stats.get("cached_tokens")
                            or stats.get("cache_read_input_tokens")
                            or (stats.get("prompt_tokens_details") or {}).get(
                                "cached_tokens"
                            )
                            or 0
                        ),

That alone fixes Claude. Fireworks-via-Lemonade additionally needs _last_usage to carry prompt_tokens_details through at lemonade.py:440.

Per CLAUDE.md's boundary-contract rule: the test suite asserts against a hand-built performance_summary fixture (tests/unit/test_model_comparison.py:51), which proves the flattening works but never that any provider supplies the field. A test built from a real _last_usage shape would have caught this.

🟡 Important — a local model is reported as unpriced, not as free

_price() (model_comparison.py:243) ends return usd or None. compute_cost deliberately returns 0.0 for models absent from MODEL_PRICING — its docstring at quality_metrics.py:340 calls this "defined behavior, not a fallback". Collapsing that 0.0 to None erases the distinction and routes every Lemonade-served model into the "No published rate … dollars are not guessed" footnote (model_comparison.py:353).

tests/unit/test_model_comparison.py:596 (test_an_unpriced_model_gets_no_dollars) uses "some-local-gguf" as its fixture, so the test locks in the conflation rather than catching it.

Suggest distinguishing the two: a model in the table costs what the table says (including a real $0.00); a model absent from the table is $0.00 local and only a cloud-prefixed id with no row is genuinely unpriced.

🟡 Important — the module has no entry point and no docs

grep -rn "model_comparison" matches only tests/unit/test_model_comparison.py. No gaia subcommand constructs the table, so the output in the PR description isn't reproducible by a reader. CLAUDE.md requires new CLI commands in both src/gaia/cli.py and docs/reference/cli.mdx, and new pages in docs/docs.json. If it's intentionally library-only for now, say so in the description.

🟢 Nits

1. Unpriced models sort as cheapest — model_comparison.py:328 uses r.usd if r.usd is not None else 0.0, so a model with no price wins the cost tiebreak. Same "absent must not read as free" principle the module argues for.

        key=lambda r: (-(r.pass_rate or 0.0), r.usd if r.usd is not None else float("inf")),
   

2. Dead code — _mean (model_comparison.py:162) is never called, and ModelRun.judged / _JUDGED are computed at line 189 but appear in no column and no consumer.
3. Mixed rate cards in compute_cost — quality_metrics.py:367: with explicit cost_per_1m_input/_output overrides, the cached rate still falls back to MODEL_PRICING, producing one number from two rate cards. Defaulting cached_rate to the explicit in_rate when overrides are supplied would match "explicit overrides win".
4. Comment density — CLAUDE.md's "Code Comments — Short or Skip" asks for one-line why comments; several docstrings here run to multi-sentence rationale (e.g. model_comparison.py:134-141, 226-234, and the 12-line block at config.py:11-25). The reasoning is good — most of it belongs in the commit body.
5. PR description footer — the 🤖 Generated with [Claude Code] line is explicitly prohibited by CLAUDE.md's "No Claude Attribution of Any Kind".

Strengths

• The pricing arithmetic checks out. Reproducing the PR's own example (1.3M input, 77% cached, 94.1k output at the glm-5p2-fast rates) gives $1.459 against the reported $1.4575 — the cached/uncached split is implemented correctly, it just never receives non-zero input.
• _price() deliberately re-costs from raw token counts rather than trusting each scorecard's own cost block, and the docstring explains why. That's the right call and easy to get wrong.
• cost_per_pass is a genuinely better decision column than raw cost, and returning None on zero passes avoids an infinity in the table.
• The config.py comment about absent-vs-zero cached_per_mtok being different offers is a real distinction most pricing tables get wrong.

[kovtcharov]

Addressed in 6e9dbc8. The table now shows real cache shares, keeps "never reported" apart from zero, and prices local models at $0.00 instead of calling them unpriced.

1. Cache counts were never read. Fixed. The eval now reads Claude's, Lemonade's and Fireworks' raw usage spellings, and the tests use Claude's real usage output instead of hand-built fixtures. For Fireworks through Lemonade, the other half is in feat(tui): show what a session costs — time, steps, tools, tokens, dollars #3739, which makes Lemonade pass the count along. Once both merge, the chain is complete.
2. 0% for unmeasured cache. A run whose provider never reported a cache count now shows —. A reported zero still shows 0%.
3. Local models looked unpriced. A local model now costs a defined $0.00 and is named as local under the table. Only a cloud model (fireworks., amd., claude-) with no rate row is unpriced.
4. Unpriced sorted as cheapest. Unpriced rows now sort after priced ones.
5. Dead code. Removed _mean and judged, which grep confirmed nothing used.
6. Mixed rate cards. When explicit input/output rates are given, cached tokens bill at the explicit input rate unless a cached rate is also passed.
7. No entry point. Left out on purpose to keep this PR small. The description now says it is a library building block for now. It also names the harness behind the 77% figure: an external Fireworks benchmark harness (bench/run_task.py) that sums the per-step cached_tokens stats. That field never reached the eval pipeline on this branch, which is why you couldn't reproduce the figure (point 1).
8. Comment density. Cut the flagged rationale blocks to one-line why comments.

On the footer: this session's instructions require the PR-description attribution footer, which conflicts with CLAUDE.md, so I left it for a maintainer to decide.

🔍 Technical details

• 1: _cached_tokens(stats) in performance.py checks cached_tokens, cache_read_input_tokens, and prompt_tokens_details.cached_tokens in that order. It uses is not None rather than an or chain, so a reported 0 survives. extract_step_stats also accepts prompt_tokens/completion_tokens, following the convention in agent.py. Without that, Claude and cloud-Lemonade steps read as 0 input and the cache share was meaningless. feat(tui): show what a session costs — time, steps, tools, tokens, dollars #3739's LemonadeProvider._capture_usage emits the flattened cached_tokens key. New TestCachedTokensFromRealProviderShapes runs ClaudeProvider._capture_usage itself, plus feat(tui): show what a session costs — time, steps, tools, tokens, dollars #3739's shape and a raw Fireworks usage body, through extract_from_agent_result → to_performance_summary.
• 2: StepResult.cached_tokens and RunResult.total_cached_tokens are now Optional[int] and None when no step reported a count. from_scorecard's _add already skips None, so cached_share stays None. No other consumer sums total_cached_tokens (checked benchmark.py, scorecard.py, runner.py).
• 3: New is_unpriced_cloud_model() is true for an id absent from MODEL_PRICING that starts with claude- or satisfies gaia.llm.lemonade_client.cloud_model_provider. _price() returns None only for those ids or for a run with no token counts, and no longer collapses 0.0 to None. test_an_unpriced_model_gets_no_dollars now uses fireworks.some-unlisted-model. New tests cover Gemma-4-E4B-it-GGUF = $0.00 and the local footnote.
• 4: The sort key uses float("inf") for usd is None.
• 6: The override branch of compute_cost sets pricing = None, so cached_rate falls back to in_rate unless cost_per_1m_cached is given.
• Claude cache writes: Claude's cache_creation_input_tokens still bill at the input rate. Claude rows carry no cached_per_mtok, so Claude cost is unchanged by this PR.
• Validation: the three affected test files pass (112 tests). The eval/performance/scorecard/benchmark sweep passes (833, 14 skipped). flake8 and pylint with util/lint.py's flags are clean on the changed files. The repo-wide gates fail with the same single error before and after this commit.