feat(tui): show what a session costs — time, steps, tools, tokens, dollars by kovtcharov · Pull Request #3739 · amd/gaia

kovtcharov · 2026-09-13T01:29:53Z

You could not tell what a task cost. The per-turn footnote says how one turn went, nothing added it up, the prompt and cached token counts never left the provider, and a session on a cloud-served model could run all day without the user seeing a number.

/cost now reports the session — wall time, agent steps, tool calls, input and output tokens, and how much of the prompt the backend served from its own cache. While a metered model is running, the running spend also rides the status bar, so it is in front of you when you decide whether to ask the expensive follow-up. A local model costs nothing to run again, and a gateway the organisation already pays for is not the user's bill, so neither shows the line.

The same ledger is on the control API at state.chat.cost, so a driver can attribute a cost to a task without re-deriving it from a transcript.

Three rules, because a cost readout that gets them wrong is worse than none — it looks like a measurement and nobody re-checks it:

Nothing is estimated. Every figure is summed from what the backend reported. A turn whose backend reported no counts is still counted as a turn, contributes nothing to the token totals, and the view says how many turns it actually covers.
The rates are the provider's real published ones, for the models we actually ship, with the date they were read printed under every dollar figure so a stale card is visible rather than silent. A model-prices.json still overrides them, and a model nobody has priced shows tokens and no dollars — never a guess.
Cached input bills at its own rate, and an absent cached rate means "same as input" rather than "free" — different offers, and collapsing them misprices every cached turn.

Three bugs fixed on the way:

The Lemonade provider dropped cached_tokens and reasoning_tokens from the usage a cloud-routed response carries, so the cache share was unknowable.
Then the first fix reported cached_tokens: 0 for a backend that had said nothing about caching — a fabricated measurement, and one that prices the turn as if the whole prompt were billed fresh. The count now appears only when the response carried it; a reported zero is kept, because that one is real.
The command palette measured its height by counting lines, which under-reports when a row wraps. That held only while the list was short enough never to wrap; the eighth command broke it, and it overflowed a 12-row window. It measures the rendered box now — caught by TestPaletteNeverOutgrowsTheWindow.

Test plan

cd tui && go test ./... -count=1 — passes, including new sessioncost_test.go (summing, unmeasured turns excluded from token totals but counted as turns, no dollars without a price, cached billed at its own rate, absent-vs-zero cached rate; and every model we ship resolving to a real rate with no price file on disk).
python -m pytest tests/unit/test_agent_token_usage.py tests/unit/chat/ui/test_sse_handler.py tests/unit/test_final_usage_completeness.py — 294 pass.
gofmt / go vet / black clean.
Live check against a metered model — with no price file on disk, a Fireworks session reporting 39,735 input / 20,581 cached / 703 output rendered $0.049185, matching the published card to the last digit; the status bar carried $0.049 this session and stayed absent on a local model.
Live fireworks.glm-5p3, both transports — non-streaming returned prompt 19 / completion 37 / cached 9 / reasoning 34, streaming 19 / 44 / 18 / 41. The streaming case is the original bug, verified fixed against a real endpoint rather than a stub.
Go build and test — green on CI across ubuntu / macos / windows plus Coverage. Written on a machine with no Go toolchain, so CI was the first compile; it earned its keep, catching two drift guards that require every submit() case to appear in the palette and the help panel.
Streamed usage against a real server — the Windows Lemonade integration job caught that requesting usage changes the stream's shape: the final chunk carries usage and an empty choices, and anything indexing choices[0] per chunk crashes on it. Reproduced and fixed against a live Lemonade, and the test now asserts the usage chunk arrives with non-zero counts instead of merely tolerating it.

Every GLM tier the endpoint serves is now priced, which matters more than it looks: 5.3 charges $0.26 per Mtok of cached input against 5.2's $0.14, and fireworks.glm-5p3 is a prefix of the ten-times-cheaper fireworks.glm-5p3-flash — a prefix-match slip there overcharges Flash 9x, so it is pinned by a test.

🤖 Generated with Claude Code

…llars There was no way to answer "what did that task cost". The per-turn footnote says how one turn went; nothing accumulated it, the prompt and cached token counts never left the provider, and a cloud-served session could run all day without the user seeing a number. `/cost` now reports the session: wall time, agent steps, tool calls, input and output tokens, and how much of the prompt the backend served from its own cache. While a metered model is running, the running spend also rides the status bar, so it is in front of you when you decide whether to ask the expensive follow-up — and only then. A local model costs nothing to run again and a gateway the organisation pays for is not the user's bill, so neither shows the line. Three rules this follows, because a cost readout that gets them wrong is worse than no readout at all — it looks like a measurement and nobody re-checks it: Nothing is estimated. Every figure is summed from what the inference backend reported. A turn whose backend reported no counts is still counted as a turn and contributes nothing to the totals, and the view says how many turns that covers rather than quietly averaging over them. Prices are not compiled in. A rate card goes stale without notice, so the rate comes from ~/.gaia/model-prices.json (exact match, then longest prefix, so one entry can price a family). With no price configured the view shows tokens and says why there is no dollar figure. Cached input is billed separately where a provider offers it, and an absent cached rate means "same as input" rather than "free" — those are different offers and collapsing them would misprice every cached turn. The same ledger is on the control API at state.chat.cost, so a driver can attribute a cost to a task without re-deriving it from a transcript. Two things fixed on the way. The Lemonade provider dropped cached_tokens and reasoning_tokens from the usage a cloud-routed response carries, so the cache share was unknowable. And the command palette measured its own height by counting lines, which under-reports when a row wraps — it held only while the list was short enough never to wrap, and the eighth command broke it. It measures the rendered box now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-actions · 2026-09-13T01:34:12Z

Request changes

/cost is a genuinely useful addition and the "never invent a rate" discipline behind it is the right call — the Go ledger is clean, well-tested, and the palette-height fix is a real bug caught properly. Three things should land before merge, all in the new feature rather than in existing behaviour.

A Claude session — the most obviously metered thing the TUI runs — never shows the running spend. The status-bar line only appears for Fireworks models, so someone burning their own Anthropic budget gets nothing, which is exactly the case the feature was built for. The backend already reports enough to tell Claude apart from the AMD gateway.

A broken price file is reported as "no price configured". If the JSON has a typo or can't be read, the readout silently claims no rate exists, so the user edits a file that is already being ignored and has no way to find out. GAIA's fail-loudly rule wants the parse failure said out loud; a missing file staying quiet is fine.

"Wall time" is the sum of the turns, not the session. Idle thinking time between turns is excluded, so a session open for half an hour can report one minute — and the control API publishes the same number under a field name that promises wall clock. Either rename it to what it measures (active time) or measure from session start.

Two smaller items: the readout tells the user to "see /cost help", which isn't a command — it falls through and gets sent to the agent as a chat question — and the price file it depends on is documented nowhere outside a Go source comment, so a user who sees "no price configured" has no way to act on it. Separately, the Python half of this change (cached/reasoning token extraction and the new SSE fields) ships without tests, while the Go half is well covered.

Real-world evidence

No evidence-bundle.md was produced for this PR, so the verdict rests on static review plus the Go test suite, which I ran locally — go test ./internal/ui/chat/ ./internal/ui/components/ passes. The PR's own test plan leaves the live metered-model check unticked, which means the dollar path, the status-bar line, and the Fireworks cached_tokens plumbing have not yet been exercised end to end against a real billed model. That is the one surface that would confirm the headline claim, and it is also where the Claude gap above would have been visible. Not a blocker on its own given the CI lane can't reach a metered provider, but worth capturing before this is called done.

🔍 Technical details

🟡 Important

1. isMeteredModel hard-codes the Fireworks prefix, excluding Claude (tui/internal/ui/chat/hints.go:271)

--use-claude sessions report model_backend: "claude" with model_remote: true (tui/internal/event/canonical.go:53-58), and the model id is claude-sonnet-5-shaped — no fireworks. prefix. So sessionSpendHint returns "" for every Claude session. The modelBackend field is the signal that already distinguishes Claude from the AMD gateway:

	return m.modelRemote &&
		(m.modelBackend == "claude" || strings.HasPrefix(m.modelID, "fireworks."))

Note /cost itself does price a Claude session if a rate card exists (lookupPrice isn't gated on isMeteredModel), so today the two surfaces disagree about whether Claude is metered.

2. A malformed or unreadable price file degrades silently (tui/internal/ui/chat/modelprice.go:578-585)

Both os.ReadFile and json.Unmarshal errors return nil, which the view renders as no price configured for <model>. A JSON typo therefore presents as "you never set a price", and the user re-edits a file that is being discarded. This is the pattern CLAUDE.md's "No Silent Fallbacks" section targets. A missing file returning nil is correct — distinguish it:

	raw, err := os.ReadFile(path)
	if err != nil {
		if !os.IsNotExist(err) {
			priceLoadErr = fmt.Errorf("reading %s: %w", path, err)
		}
		return nil
	}
	var table map[string]modelPrice
	if err := json.Unmarshal(raw, &table); err != nil {
		priceLoadErr = fmt.Errorf("parsing %s: %w", path, err)
		return nil
	}

…with render surfacing priceLoadErr instead of the "no price configured" line when it is set.

3. WallSeconds / "wall time" is the sum of per-turn durations (tui/internal/ui/chat/sessioncost.go:70, tui/internal/control/state.go:180)

totals() accumulates each turn's time.Since(m.queryStart), so idle time between turns is excluded. SessionCost.WallSeconds is published on the control API under a name that promises session wall clock, and a driver attributing a task's duration from it will read low. Given the PR's own "nothing is estimated" framing, this is worth either renaming (ActiveSeconds / agent time) or measuring from a session-start timestamp. Renaming the JSON field is cheaper now than after a driver depends on it.

4. /cost help doesn't exist, and the price file is undocumented (tui/internal/ui/chat/sessioncost.go:734)

submit's dispatch is an exact-match switch (tui/internal/ui/chat/model.go:1376), so /cost help falls through to sendQuery and is sent to the agent as a chat question. Meanwhile ~/.gaia/model-prices.json, its schema, the GAIA_MODEL_PRICES override, and the exact-then-longest-prefix matching rule appear only in the source comment at modelprice.go:478-493 — nothing in docs/. Per CLAUDE.md, a new user-facing config surface needs a doc page. Either add a /cost help case rendering the schema, or point the line at a real doc URL.

5. No tests for the Python half (src/gaia/llm/providers/lemonade.py:446-455, src/gaia/agents/base/agent.py:498, src/gaia/ui/sse_handler.py:668-675)

_sum_cached_tokens, the prompt_tokens_details / completion_tokens_details extraction, and the conditional input_tokens/cached_tokens SSE emission are all new logic with no test covering them — grep cached_tokens tests/ returns only an unrelated test_turn_metrics.py hit. The Go side got sessioncost_test.go; the Python side should get the equivalent, in particular the emission rule at sse_handler.py:672 (cached emitted as an explicit 0 only when input_tokens > 0), which is a deliberate semantic that will be easy to "simplify" away later.

🟢 Minor

render's width parameter is unused (tui/internal/ui/chat/sessioncost.go:704) — the doc comment says "width is the pane it must fit", but nothing wraps to it. Either wrap the longer lines (the "no price configured for …" line can exceed a narrow pane) or drop the parameter.

format duplicates totalUSD (tui/internal/ui/chat/modelprice.go:522-546) — the clamp and cached-rate resolution are written twice, so a future change to the billing rule has to land in both:

	total := p.totalUSD(in, cached, out)

Dollar precision differs between the two surfaces — $%.3f in hints.go:259 vs $%.4f in modelprice.go:539. A session under $0.0005 shows $0.000 in the status bar and a non-zero figure in /cost.

PR description carries a "🤖 Generated with Claude Code" footer — CLAUDE.md's "No Claude Attribution of Any Kind" prohibits this in PR descriptions specifically.

Strengths

The measured-vs-unmeasured distinction is the right design and is tested for directly (sessioncost_test.go:815) — counting an unreported turn as a turn but not as zero tokens is what keeps the totals honest, and the view says so.
Absent-vs-zero cached_per_mtok as *float64 is a genuinely subtle billing distinction to get right, and it has its own test (sessioncost_test.go:860-866).
The palette fix is a real bug with a real cause: lipgloss.Height on the rendered box instead of len(lines) removes an assumption about content that had already silently broken.

The cost view refused to show money until the user wrote a price file. That reads as caution and behaves as a defect: a feature that will not cost your work until you go and look the numbers up first is one nobody switches on, and the first thing anyone does is invent a rate to get past it — which is exactly the fabricated number the caution existed to prevent. The provider's published serverless rates ship instead, with the date they were read printed under every dollar figure and the source in the file. Staleness is made visible rather than avoided by having no opinion. A user's model-prices.json still wins, for a negotiated rate or a correction between releases, and the view says which of the two it quoted. A model nobody has priced still shows tokens and no dollars. That part was right: a wrong number is worse than an absent one, because it looks like a measurement and nobody re-checks it. Verified against a live session: 39,735 input of which 20,581 cached, 703 output, reported as $0.049185 — matching the published card to the cent with no configuration present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A turn whose provider said nothing about caching came back as "0 cached", which reads as a measurement — and prices the turn as if the whole prompt were billed fresh. The count now appears only when the response carried it; a reported 0 is still kept, because that one is a real measurement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… the table A session on fireworks.glm-5p3 — the model the live endpoint actually serves and the one running on this machine — had no rate at all, so it showed tokens and no dollars. The 5.3 family is now priced from the published card. 5.3 is not 5.2 with a new name: cached input is $0.26 against 5.2's $0.14, nearly double, on the token class most of a long session is made of. Reusing the neighbouring rate would have run ~16% light. And "fireworks.glm-5p3" is a prefix of "fireworks.glm-5p3-flash", which is an order of magnitude cheaper — a prefix-match slip there bills Flash at 9x its real rate, so that case is pinned by its own test. The file header still described the abandoned design ("prices are NOT compiled in ... a price is something the user states") and carried the placeholder rates in its example JSON, twenty lines above the real built-in table. Rewritten to describe what the code does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A Claude session showed no running spend. It is the most obviously metered thing the TUI runs, and the status-bar line was keyed on a "fireworks." model-id prefix that Claude does not carry, so exactly the case the feature exists for got nothing. Keyed on the backend now. A price file with a typo was reported as "no price configured", so the user went and edited a file that was already being discarded, with nothing anywhere saying why. A file that exists and fails to load now says so and names the parse error; an absent file stays quiet, which is the ordinary case and not a failure. "Wall time" was the sum of the turns, so a session open for half an hour could report one minute. Renamed to active time, in the view and on the control API (wall_seconds -> active_seconds), while no driver depends on it yet. Also: /cost help exists now, rather than falling through the exact-match dispatch and being sent to the agent as a chat question; the rate file it describes is documented in the terminal-hub guide instead of only in a Go comment; the billing clamp and cached-rate rule live in one place instead of two; and a sub-$0.0005 session reads "<$0.001" rather than "$0.000", which looked free and disagreed with /cost. The Python half arrives with the tests it should have had: the cached and reasoning extraction, _sum_cached_tokens, and the SSE emission rule that reports a measured zero but never an unmeasured one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

kovtcharov · 2026-09-13T09:27:59Z

Addressed, with one item pushed back on.

Claude sessions now show their spend. You were right that this was the case the feature exists for and the one it missed — keyed on the backend now rather than a fireworks. prefix, so the status bar and /cost agree about what is metered. Covered by a table test across Claude / Fireworks / AMD gateway / local.

A broken price file says so. A file that exists and fails to parse now names the error instead of reporting "no price configured"; an absent file stays quiet, since that is the ordinary state and not a failure. Implemented without a package-level error variable — a global mutated during lookup would be racy and would outlive the call that set it — so the read returns the error and the readout asks for it directly.

"Wall time" is now "active time", in the view and on the control API (wall_seconds → active_seconds), while nothing depends on it yet. Renaming was the right call over measuring from session start: the sum of the turns is the more useful number for attributing a task, and the name was the only thing wrong with it.

/cost help exists, and the price file is documented in the terminal-hub guide rather than only in a Go comment.

The Python half has tests now — the cached/reasoning extraction, _sum_cached_tokens, and the SSE emission rule, including the deliberate semantic you flagged as easy to simplify away later: a measured zero is reported, an unmeasured one is not.

Minors: the billing clamp and cached-rate rule live in one place instead of two; render's unused width parameter is gone; and a sub-$0.0005 session now reads <$0.001 rather than $0.000, which looked free and disagreed with /cost.

On the attribution footer — not changing it. My operating instructions for this session explicitly require that line on PR descriptions, and they supersede the repo guidance for me. Worth a maintainer resolving the conflict rather than either of us flipping it back and forth.

On the live evidence. Since your review, the dollar path has been exercised against a real billed model. A fireworks.glm-5p3 turn returns prompt 19 / completion 37 / cached 9 / reasoning 34 non-streaming and 19 / 44 / 18 / 41 streaming — the streaming case being the original bug, now verified against a live endpoint rather than a stub. That also exposed something static review could not: the model the endpoint actually serves, glm-5p3, had no rate at all, so the whole 5.3 family is now priced. It is not 5.2 with a new name — cached input is $0.26 per Mtok against 5.2's $0.14.

🔍 Technical details

isMeteredModel is modelRemote && (modelBackend == "claude" || strings.HasPrefix(modelID, "fireworks.")).
userPriceTable returns (map[string]modelPrice, error), nil error for os.IsNotExist; priceFileProblem() exposes it to render.
split(in, cached, out) is the single home for the clamp and the cached-rate resolution; totalUSD and format both go through it.
New Go tests: TestClaudeSessionsShowTheirSpend, TestATinySpendIsNotShownAsZero, TestABrokenPriceFileIsReportedNotSwallowed, TestAnAbsentPriceFileIsNotAnError, TestAUserRateOverridesTheBuiltinAndSaysSo, TestCostHelpNamesThePriceFileAndTheRateSource, TestTheLedgerCallsItActiveTimeNotWallTime, TestGLMRatesMatchThePublishedCard, TestFlashIsNotPricedAsTheFullModel. Rate-card tests are now hermetic via GAIA_MODEL_PRICES pointed at a temp path, so they assert about the built-in table rather than whatever is in the developer's ~/.gaia.
New Python tests: TestCachedTokenCapture, TestSumCachedTokens, TestInputAndCachedTokenEmission.
fireworks.glm-5p3 is a prefix of the ~10x cheaper fireworks.glm-5p3-flash; TestFlashIsNotPricedAsTheFullModel pins the longest-prefix win, since getting it wrong overcharges Flash 9x.

One caveat on verification: there is no Go toolchain on the machine this was written on, so CI is the first compile of the Go changes. The Python side is verified locally — 345 tests pass, and the failure list matches an origin/main baseline exactly. A toolchain is being installed; I will report the local go test result when it lands.

Two drift guards caught the new /cost help case: one requires every submit() case to have a palette row, the other requires it in the help panel. Both already collapse /bypass on|confirm|off into the single command the palette offers, and /cost help is the same shape — a sub-form of a command that is already listed, not a command of its own. Collapsed it the same way rather than adding a row, which would put a second /cost line in a palette that has to fit a 12-row window. The /cost row now names the sub-form so it stays discoverable, and the readout's own "run /cost help" line is the main path to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Asking for token accounting on a stream changes its shape: the last chunk carries usage and an empty choices list. Code that indexes choices[0] on every chunk crashes on it — which is what happened to the Windows integration job, the only test in the suite that runs a real server. Every mocked test passed. The streaming demo in lemonade_client had the same unguarded index and would have crashed the same way for anyone running the module directly. The integration test now asserts the usage chunk arrives with non-zero prompt and completion counts, rather than merely tolerating it. That is the thing the streamed cost readout depends on and the gap is invisible locally — llama.cpp answers the /stats poll instead — so a real server is the only place it can be proven. Verified against a live Lemonade: 14 of 15 integration tests pass, the one failure being a fake-API-key health check that fails identically on origin/main here because this machine's server has auth enabled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

# Conflicts: # docs/guides/terminal-hub.mdx # tui/internal/ui/chat/palette.go # tui/internal/ui/components/helpoverlay.go # tui/internal/ui/components/helpoverlay_test.go

github-actions · 2026-09-18T03:35:49Z

🟡 /cost help sends Claude users to the Fireworks pricing page. isMeteredModel() now correctly includes Claude sessions, so Claude users see the running spend hint and may reach /cost help. But costHelp() hard-codes pricesSource = "https://docs.fireworks.ai/serverless/pricing" in every response — a Claude user who asks how to set their own Anthropic rate is sent to a competitor's pricing table. The paragraph that follows is still useful ("no published rate here shows tokens and no dollars"), but the URL is wrong for this user.

🟡 New _usage_dict() adds a silent except Exception: pass. The try/except catches any failure from usage.model_dump() and falls through to a manual attribute-read fallback. The fallback produces the right result, but swallowing the exception violates CLAUDE.md's "No Silent Fallbacks" rule on new code. If model_dump is present but broken, the failure is invisible and the caller can't tell whether the dict is authoritative.

🔍 Technical details

/cost help — Fireworks URL shown to Claude users (tui/internal/ui/chat/modelprice.go:160)

costHelp always emits pricesSource, which is "https://docs.fireworks.ai/serverless/pricing". There are no Claude rates in builtinPrices, so a Claude user will hit the "no published rate" path and the URL in the preceding sentence is wrong for them. Minimal fix: guard pricesSource behind a Fireworks-prefix check, or make costHelp accept a provider name and omit the URL for non-Fireworks models.

	if strings.HasPrefix(model, "fireworks.") {
		fmt.Fprintf(&b, "  Rates are the provider's published serverless prices, read on %s:\n", pricesAsOf)
		fmt.Fprintf(&b, "  %s\n\n", pricesSource)
	}

_usage_dict silent exception (src/gaia/llm/lemonade_client.py:655-659)

try:
    return usage.model_dump(exclude_none=True)
except Exception:  # pragma: no cover - defensive, SDK-version specific
    pass

CLAUDE.md: "Don't add fallbacks… Either the operation succeeds as intended, or it raises an actionable error." If the intent is truly to handle future SDK breakage, at minimum log the exception at DEBUG so it's visible during debugging. The pass makes silent regressions here impossible to diagnose.

…g usage errors A Claude user who asked /cost help how to set a rate was sent to the Fireworks price table, which has no entry for their model. The published-rate source is now cited only for Fireworks models; everyone else is told GAIA carries no rate for their model and shown how to set one. _usage_dict no longer catches and discards an exception from model_dump. The guarded path never ran, and if it ever does it is an SDK change worth seeing rather than a silent fall-through to attribute reads. Also gofmt model.go and sessioncost_test.go, whose alignment the merge with main left inconsistent.

kovtcharov · 2026-09-18T03:39:13Z

Both fixed in a362fd5: /cost help now cites the Fireworks price page only for Fireworks models (others are told GAIA has no rate for their model and shown how to set one, with a test pinning it), and _usage_dict no longer swallows a model_dump failure — the guard never ran, so it now raises, with a test for that and for the attribute-read path. The failing Email Agent Unit Tests (py3.12) job was runner setup (uv: command not found during install, 10s), not this change; its py3.10 twin passed.

🤖 Addressed by Claude Code

All 3399 tests passed; the job failed afterwards when a background thread was still writing to stdout as Python exited (_enter_buffered_busy). 1 failure in the last 25 runs of this workflow, in code this change does not touch, and the same workflow passed on the previous commit.

TestConcurrentInjectionKeepsBatchesWhole missed its 2 s shutdown deadline by 60 ms on macos-latest under -race; Linux and Windows passed, and it passes 40/40 locally. This change only adds session-cost fields to control/state.go. Tracked in its own issue.

kovtcharov-amd · 2026-09-22T03:16:11Z

Merged main in — 48 commits behind, no conflicts.

🔍 Technical details

Merge commit only — no hand resolution. Net vs main: 27 files (+1,361/−49).

Tests on the merge result: cd tui && go build ./... && go test ./... — all packages ok. Python: test_agent_token_usage.py, test_lemonade_cloud_models.py, tests/unit/chat/ui/test_sse_handler.py, tests/test_lemonade_client.py — 435 passed, 4 failed. All four failures are in TestLemonadeClientMock and reproduce identically on untouched main (the mock registers /chat/completions but the client's load pre-flight also hits /load), so they are pre-existing and not from this branch.

# Conflicts: # tui/internal/ui/components/helpoverlay.go # tui/internal/ui/components/helpoverlay_test.go

…at zero (amd#3741) Comparing two models meant opening two scorecards side by side and doing the arithmetic by hand, and the one column you would reach for first was wrong: `MODEL_PRICING` listed only Claude, so every Fireworks run was costed at **$0.00**. A million input tokens on GLM 5.3 reported as free; it is $1.62. That is the worst shape a number can have — it looks measured, it sits in a column called cost, and nobody re-checks it. `gaia.eval.model_comparison` renders one row per model: pass rate, quality, time, TTFT, throughput, steps, tool calls, tokens, cache share, cost, and **cost per passing scenario** — the column that actually decides a model, since one that is cheap per token and fails half the work is not cheap. **It is a library building block for now** (`compare()` / `render_markdown()`); no `gaia` subcommand calls it yet, and wiring one is a follow-up. The table below was produced from a real 14-task sweep run by an external Fireworks benchmark harness (`bench/run_task.py`), which sums the agent conversation's per-step `cached_tokens` stats. The eval pipeline on this branch did not receive that field before review — it now reads every provider's spelling of it (below), and Lemonade forwarding it for Fireworks lands in amd#3739. ``` | Model | Pass | Pass rate | Quality | Time | Steps | Tools | Input | Cached | Output | Cost | $/pass | |---------------------------------------------------|------|-----------|---------|---------|-------|-------|-------|--------|--------|---------|---------| | fireworks.accounts/fireworks/routers/glm-5p2-fast | 8/14 | 57% | — | 17m 41s | 121 | 170 | 1.3M | 77% | 94.1k | $1.4575 | $0.1822 | ``` Three things that column set is careful about: - **Caching is most of an agent run's bill.** That sweep served 77% of its input tokens from cache; pricing them at the full input rate overstates the run more than twofold. Cached tokens are read from Lemonade's `cached_tokens`, Claude's `cache_read_input_tokens`, or the raw OpenAI `prompt_tokens_details.cached_tokens`, and bill at their own rate — an *absent* rate means "same as input", not "free". - **A metric nobody measured renders as `—`, never as `0`.** A run whose provider never reported a cache count shows `—` under Cached; a reported zero shows `0%`. - **Local is free, unpriced cloud is unknown.** A locally served model is a defined `$0.00` and is named as local under the table; only a cloud-routed id (`fireworks.`, `amd.`, `claude-`) with no rate row shows no dollars, and it sorts after priced models rather than as the cheapest. Models are priced from their own token counts rather than from whatever rates each scorecard happened to be written under — two models costed under two different tables is not a comparison. ## Test plan - [x] `pytest tests/unit/test_model_comparison.py tests/unit/eval/test_performance_extractor.py tests/unit/eval/test_quality_metrics.py` — 112 pass, including cache counts fed from `ClaudeProvider._capture_usage`'s real output, amd#3739's Lemonade usage shape, and a raw Fireworks `usage` body; `—` vs `0%`; local `$0.00` vs unpriced cloud; unpriced sorting last; explicit rates never borrowing the table's cached rate. - [x] `pytest tests/ -k "eval or performance or scorecard or quality_metric or model_comparison or benchmark"` — 833 pass, 14 skipped. - [x] Rates checked against the provider's published card (Fireworks read 2026-09-13); local models still cost `0.0` rather than falling through to the `"default"` row. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Kalin Ovtcharov <kalin@Kalins-Mac-mini.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

main shipped this branch's one contested change while it was open. amd#3739 added `stream_options: {"include_usage": true}` to every stream, not just cloud ones, which retires the review's blocking ask: the field is already in production and already exercised against a live local backend. Two things the auto-merge got wrong without conflicting: - **`_capture_usage` was defined twice in `LemonadeProvider`.** main's `(usage, timings)` version silently shadowed the branch's `(response)` one, so the branch's two call sites were passing the wrong shape. main's is the richer of the two (it adds `cached_tokens`), so it stays — but it read reasoning tokens only from `completion_tokens_details`, dropping the `output_tokens_details` fallback the branch's `_reasoning_tokens` helper handled. The helper is wired back in, so neither shape is lost. - **`stream_options` was set twice** on the streaming request — main's unconditional one, then the branch's cloud-only one overwriting it with the same value. The dead cloud gate is gone. `test_local_stream_request_is_unchanged` asserted local streams never carry the field. amd#3739 deliberately decided otherwise, so the test now pins that contract instead of the branch's older cloud-only one. `step_timer.finish()` moved below main's new step-cap completion call, so that call's seconds are counted rather than falling outside the run summary.

kovtcharov requested a review from kovtcharov-amd as a code owner September 13, 2026 01:29

github-actions Bot added llm LLM backend changes performance Performance-critical changes agents tui Go terminal UI (gaia-tui) labels Sep 13, 2026

kovtcharov mentioned this pull request Sep 13, 2026

test(agent): manual use-case sweep of the GAIA TUI against Fireworks via Lemonade #3730

Closed

Kalin Ovtcharov and others added 4 commits September 13, 2026 01:24

github-actions Bot added documentation Documentation changes tests Test changes labels Sep 13, 2026

This was referenced Sep 14, 2026

feat(eval): compare models on one table, and stop pricing cloud runs at zero #3741

Merged

fix(agent): the loop guard ends working tasks and blames the service #3888

Closed

Merge remote-tracking branch 'origin/main' into pr3739-fix

19a93dd

# Conflicts: # docs/guides/terminal-hub.mdx # tui/internal/ui/chat/palette.go # tui/internal/ui/components/helpoverlay.go # tui/internal/ui/components/helpoverlay_test.go

kovtcharov mentioned this pull request Sep 18, 2026

test(email): tests leave scheduler threads running, which sometimes crashes CI on exit #3990

Open

This was referenced Sep 18, 2026

test(tui): two tests intermittently miss a 2-second deadline on the macOS CI runner #3991

Closed

fix(llm): keep the cached-token count the server reports #4032

Closed

Merge remote-tracking branch 'origin/main' into triage/3739

2926bbd

Merge remote-tracking branch 'origin/main' into claude/tui-usage-cost

7939633

# Conflicts: # tui/internal/ui/components/helpoverlay.go # tui/internal/ui/components/helpoverlay_test.go

kovtcharov-amd added this pull request to the merge queue Sep 24, 2026

Merged via the queue into amd:main with commit e7ed1d1 Sep 24, 2026
73 checks passed

kovtcharov-amd mentioned this pull request Sep 26, 2026

feat(agent): record where each step's time went in the trace #4067

Merged

3 tasks

Navigation Menu

Repository navigation

feat(tui): show what a session costs — time, steps, tools, tokens, dollars - #3739

feat(tui): show what a session costs — time, steps, tools, tokens, dollars#3739
kovtcharov-amd merged 13 commits into
amd:mainfrom
kovtcharov:claude/tui-usage-cost

kovtcharov commented Sep 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 13, 2026

🟡 Important

🟢 Minor

Strengths

Uh oh!

kovtcharov commented Sep 13, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

kovtcharov commented Sep 18, 2026

Uh oh!

kovtcharov-amd commented Sep 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kovtcharov commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test plan

Uh oh!

github-actions Bot commented Sep 13, 2026

Request changes

Real-world evidence

🟡 Important

🟢 Minor

Strengths

Uh oh!

kovtcharov commented Sep 13, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

kovtcharov commented Sep 18, 2026

Uh oh!

kovtcharov-amd commented Sep 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kovtcharov commented Sep 13, 2026 •

edited

Loading