fix(dash): price removed tokens as the provider counts them — closes #240

[OsherElhadad]

Closes #240.

request_components.saved_usd, baseline_cost_usd and everything derived from them took our own tokenizer's count of the content a component removed and priced it at the provider's rate. A quantity in one unit, a price in another.

cost_usd was never affected — it is built from the four token tiers the provider itself reports — so spend does not move. Only the counterfactual does.

The numbers

Deployed service, snapshotted read-only 2026-09-15 11:58 UTC: 281,421 requests, 29,346 sessions, 19 tenants, 2026-08-17 → 09-15. The service was never written to or restarted.

	old	corrected	change
provider spend SUM(cost_usd)	$32,203.3367	unchanged	—
context-guru's own model spend	$16.4819	unchanged	—
gross saved	$103.0428	$173.6241	+68.5%
net saved	$86.5609	$157.1422	+81.5%
net saved as a share of spend	0.2688%	0.4880%	+0.219 pp

Month-to-date, the window /metrics and Grafana publish: cg_saved_usd $53.6004 → $90.3410, cg_net_saved_usd $44.2255 → $80.9661.

Per component (deduplicated — the table has no primary key, see below):

component	old $	corrected $
searchfold	46.8334	78.9319
extract_llm	21.4457	36.1510
textclean	12.0469	20.3075
dedup	4.6769	7.8821
extract	3.7383	6.3017
format	3.1941	5.3392
cmdfilter	2.0020	3.3623
linecap	1.2844	2.1651
extract_llm_sweep	1.0902	1.8377
toon	0.0472	0.0791
cachesplit · failed_run · toolfilter	0.0000	0.0000
total	96.3591	162.3574

Shares barely move — 98.4% of the saving is on one tokenizer. This makes no component profitable that was not; extract_llm still spends real money on its own calls and a larger gross does not settle its verdict.

Why the correction is what it is

schema.MessagesTokens counts message text with o200k_base — an OpenAI BPE — applied to every model. No system prompt, no tool declarations, no envelope. baselineDeltaUSD then priced that count at the provider's CacheWrite/repeatRate.

The factor is on a difference:

f = (billed_before − billed_after) / (ours_before − ours_after)

Everything our counter cannot see is in both terms of the numerator and cancels. That is why the 3.4× EstimatorDivergence is not a substitute — that is a ratio of levels, and using it would over-report by roughly 2×. Issue #240 reached for it, and so did its review.

Measured by apply.TestBilledPairColdReplay: run the production pipeline, then send both bodies to the gateway with every cache_control stripped so each bills its whole prompt as fresh input, and read usage off each reply.

model	pooled f	median	range	under-report
claude-haiku-4-5	1.3686	1.5030	1.168–1.517	26.9%
claude-sonnet-5	1.6857	1.8197	1.519–1.897	40.7%
claude-opus-5	1.6857	1.8197	1.519–1.897	40.7%

Sonnet 5 and Opus 5 gave byte-identical deltas on every pair, so they share a tokenizer and haiku-4-5 does not — hence a per-family factor, not one constant.

Those 11 pairs are successive turns of two lineages, so a second, independent instrument was needed. tokens.TestBilledFactorByContentClass measures f per content class on generated text: it spans 1.04 to 1.76, and the worst class is grep output — exactly what searchfold removes, and searchfold is 52% of all savings. Weighted by this corpus's own component mix it gives 1.3150 / 1.5774, agreeing with the end-to-end figures to 4–7% with no shared machinery.

Pre-registered kill criterion (if f's interval spans 1.0, nothing moves): all 33 pairs and all 12 class measurements came back above 1.0. The direction is settled; the magnitude is stated as "roughly this" in the helper's own doc comment, because the effective n is a handful of lineages.

An instrument that had to be thrown away

/v1/messages/count_tokens first gave f = 0.9961 — which would have refuted the issue. It is unreliable in a size-dependent way, which is worse than plainly broken because a small-body check certifies it:

• small body: exactly additive — messages 1,024 + tools 4,677 + system 902 = 6,603 ✓
• real 340 KB body: full = no-tools = no-system = messages-only = 51,569, while the same tools+system with a one-character message = 35,221

So 35,221 tokens vanish at exactly the sizes compaction happens at, and what survives is a tiktoken count over messages — our own tokenizer, agreeing with itself. The same body billed through /v1/messages returned 93,375. No number from that endpoint is used anywhere here; TestCountTokensIsNotABillingOracle carries the finding.

Where #240 was wrong

• Magnitude understated. 12.4% came from one turn, and a cross-turn one (turn 15's cache-write as turn 17's counterfactual). Real: 26.9%/40.7%.
• "Divergence narrows to 1.1–1.5× on large transcripts" is false. On compacted rows it holds at 3.26–3.71× at every size above 10k; untouched rows do converge (1.74× past 200k), compacted rows do not.
• The review's FilterUSD precedent does not exist. filtered_decl_tokens is tokens.Count(t.Raw) — the same estimator. What is measured there is the tier, never the quantity. There is no billed-count precedent in this repo.

Also fixed

• A fabricated cache-write premium on every non-Anthropic model. Both price paths filled a missing rate with Input × 1.25, Anthropic's published premium. No other family charges to create an entry. cost_usd was unaffected (those providers report cache_write = 0) but baselineDeltaUSD applies the rate to our saved-token count regardless — so it reached the savings on traffic that could never pay it. Worth at most $0.15 here, and reported that way; its significance is that it is an over-report, which is why "the error is always conservative" was never safe. Both directions are pinned by a test.
• An over-specified test that required a creation premium on every model, encoding the defect as an invariant.
• The comment that kept this alive — "Dollars are unaffected: they come from the provider's own reported usage". True of cost_usd, false of baseline_cost_usd beside it. A reader auditing exactly this was told not to. Deleted.
• EstimatorDivergence measured on the wrong population — only rows where nothing was compacted, the strict complement of where savings come from. Both are now published.

Deliberately not done

• No backfill. History keeps its figures. requests.billed_token_factor defaults to 0 to mark a pre-fix row — a different fact from "measured, correction 1.0" — and the Overview shows how much of a window is still on the old arithmetic. Additive column, no schema bump, no history discarded (verified against an existing DB).
• No per-component billed counterfactual, because there is no such observable: the provider bills per request, saved_usd is per component, and request_components has no billed columns and cannot have any.
• Token counts unchanged. Component gates are evaluated against them; moving those changes what gets compacted, not what gets reported.
• extract_econ.go:132 still has the same * 1.25, in a gate that decides whether a call is made. Flagged, not touched here.
• cachehistory.go / toolsuggest.go have the same root cause on different content classes (cache-boundary offsets, raw tool JSON). Applying a factor measured on removed message text to those would be the extrapolation this exercise is about avoiding. $0.0914 and $0.00 on this corpus.
• 27,564 duplicate request_components rows inflate the raw sum by $0.9515 (0.99%). All from one day, 2026-08-22, none since. A UNIQUE index cannot be created while they exist and removing them is a migration this change avoids. Recorded with its exact cost.
• The 1h-tier asymmetry (event.go:642 adds the premium to cost, never to the baseline) is real and inert: cache_write_1h = 0 on all 281,421 rows.

Verification

• go test ./... — 29/29 test packages pass; the other 4 have no tests. go vet and gofmt clean.
• New tests confirmed to have executed (go test -v naming them, IgnoredGoFiles empty for the touched packages) rather than trusting an exit code.
• Both live tests skip without their env vars, so CI needs no fixture, no network and no money.
• End-to-end on a local single-tenant instance: real Claude Code session → factor stored per row (1.3686 haiku, 1.6857 sonnet) → 73 tokens × $2.85/MTok × 1.6857 = $0.000351, exactly the stored delta.
• Additive migration verified against an existing 50-row DB: column added, every historical row correctly marked pre-fix.
• Dashboard rendered in headless Chrome at 1280px and 390px in both themes: new tiles present and correct, no console errors, no horizontal overflow.

Full report, with every calculation and the reproduction commands: docs/results/saved-usd-correction-2026-09.md.

After merge

cg_saved_usd is month-to-date, so it will not jump to the corrected figure at restart — it drifts up as corrected rows accumulate and pre-fix rows age out. The Overview's "Savings on the OLD arithmetic" tile reports how much of the window is still on the old definition; when it disappears, the window is fully corrected.

[amiddavid]

Review

I checked the arithmetic by hand, read the accounting paths, ran the suite on the eval box, and replicated one of the two live instruments. Splitting findings by what each rests on, because that matters here.

What I ran: go vet ./..., gofmt -l ., go test ./dash/... ./internal/modelinfo/... ./internal/tokens/... ./apply/... — all green. Plus one probe test of my own (finding 1) and a replication of TestBilledFactorByContentClass on haiku-4-5. I did not run TestBilledPairColdReplay — it needs a capture of the same two session lineages, so a fresh capture would be a different experiment wearing the same name — and I have no access to the 281k-row snapshot.

---

What holds up

Every figure in §1 and §3 is internally consistent: baseline deltas, net/gross, share of spend, the 1 − 1/f column, the §2 worked example at $1.90/MTok, and the end-to-end 73 × $2.85/MTok × 1.6857 = $0.000351. No arithmetic errors.

The per-component "corrected" column was genuinely recomputed per row rather than scalar-multiplied — the ratios differ by component (1.6716 format, 1.6795 cmdfilter, 1.6858 textclean) and track each one's model mix. That was the first thing I tried to break.

The central argument is right, and right for the reason given. I checked the quantity alignment specifically, since that is where this class of correction usually goes wrong: the test's denominator is res.Run.TokensBefore − res.Run.TokensAfter, and Event.Saved() is literally e.TokensBefore − e.TokensAfter over the same schema.MessagesTokens. Same quantity, so f multiplies what it was measured on. Markers and offload placeholders sit inside both arms. cost_usd is genuinely untouched and the test pins it.

TestBilledFactorByContentClass replicated exactly on haiku-4-5, all six classes to four decimals (1.0412 / 1.0506 / 1.1768 / 1.2600 / 1.3339 / 1.4728). Seeded generators, so that is a real replication rather than agreement in aggregate. Worth saying out loud: a paid measurement that reproduces byte-for-byte from a doc table is not the norm.

---

1. The read paths retroactively correct history, so "no backfill" is not true where the per-component numbers come from

DecomposeComponentSavedUSD, EstimateComponentSavedUSD and creditTurn all take the factor from tokens.BilledDeltaFactor(model) — the model name. None reads r.billed_token_factor. A row written by the old code is corrected on read anyway. Measured, not inferred (probe test, pre-fix row reconstructed exactly as history has it):

STORED saved_usd (pre-fix arithmetic)   = 0.0019000000
read-time decomposition of that row     = 0.0032028300
cross-check ratio = 1.6857          (1.0 = still a check)

blank-column historical row valued on read = 0.0032028300
same row under the OLD arithmetic          = 0.0019000000

Two consequences.

(a) The write-path/read-path cross-check is now broken by exactly the factor. SavedUSDDecomposedStored vs SavedUSD exists to catch a fix landing in one implementation and not the other — readvalue.go:229 says so. On every historical row carrying a stored figure it now reads 1.69, permanently, for any window straddling the fix. TestCrossCheckCoversOnlyStoredRows cannot see it: it uses model "m1", an unmeasured family, factor 1.0. The one test guarding the reconciliation is blind to the change that breaks it.

(b) The Components tab is backfilled; the Overview is not. Per honesty_test.go:609, 100,573 of 100,579 production request_components rows have a blank saved_usd and are valued by EstimateComponentSavedUSD on read — which now returns the corrected level. So the components view is effectively corrected for all 281k requests while SUM(baseline_cost_usd) is not; the two disagree by ~69% on the same window, and only the Overview has a boundary tile. "Savings on the OLD arithmetic" measures a boundary the tab beside it does not have.

The column added here is the right instrument and is not used for this. COALESCE(NULLIF(r.billed_token_factor,0), 1.0) in those three queries makes the read path reproduce what was stored, keeps the cross-check a check, and makes "no backfill" true. Either resolution is defensible — reproduce history, or correct it everywhere and say so — but right now it is neither and the disagreement is silent.

2. billed_token_factor is write-only

It is in the INSERT and in Overview's aggregates. It is not in requestCols, so every Event scanned back reports billed_token_factor: 0 — which by this schema's own semantics means "predates the correction" — on every corrected row, in the requests list, the single-request view and the JSON API.

The field's stated justification is that "a reader who sees a savings figure move needs to be able to see, and divide out, the thing that moved it." Outside one aggregate tile, they cannot.

BilledTokenFactorMeasured has no column at all, so "measured, correction 1.0" versus "unmeasured, no correction" — the distinction the schema comment says is the whole reason the default is 0 rather than 1 — is discarded at the storage boundary. Inert while all traffic is Claude; wrong the first time another family appears, and wrong in the flattering direction, since CorrectedSavingRows will count it as corrected.

3. cachesplit_saved_usd has the identical unit mismatch and is not in the "not done" list

cachesplitSavedUSD prices e.SplitStableTokens — its own comment says "this counts BPE tokens over the text" — at the provider's CacheWrite − CacheRead spread. Same quantity in one unit, price in another. It is not corrected, and §6 does not cover it: that list names cachehistory.go, which values only the split_stable_tokens = 0 pre-instrumentation window ($0.0914). The stored column is a different and larger figure.

It matters because query.go:1186 is g.TotalSavedUSD = g.SavedUSD + g.CachesplitSavedUSD, and cg_tenant_cachesplit_saved_usd is exported. After this PR a headline total is one corrected term plus one uncorrected term. Leaving it uncorrected is defensible; summing it into a corrected total without a word is not.

---

On the measurement

The pooled-vs-median asymmetry says f falls as the removal gets bigger, and that is the axis the money is on. Pooled sits below median on both families (1.3686 vs 1.5030; 1.6857 vs 1.8197). Pooled is token-weighted, so that can only mean the largest deltas carry the lowest f — for haiku, near the 1.168 floor. Money concentrates in the largest removals and searchfold is 52% of the total, so this is the most important thing the data says and §3 does not engage with it.

The class instrument predicts the opposite for that same traffic: grep_output is the worst class (1.4728 / 1.7565) and grep output is what searchfold removes in the biggest chunks. Two instruments disagreeing in direction about the rows carrying half the dollars is a different situation from "they agree to within 4–7%".

A fourth instrument already in the repo reads 1.50. cachesplitSavedUSD's doc comment records a real provider-reported delta against a real o200k count on Claude traffic: 8,499 tokens moved per the provider's usage numbers, 5,654 per ours. That is f = 1.503 — below the shipped 1.6857, and applying 1.6857 to those 5,654 gives 9,555, more than the provider said moved. Confounded by the added block boundary, granted. But it is an in-repo billed-vs-ours delta on the relevant tokenizer, it points the same way as the class-weighted 1.5774, and it is not cited.

The sonnet column of §3.3 has an artifact, and I can now be specific. My haiku replication confirms english_prose at 1.0506 with ours = 1800 — the smallest denominator in that table by 5–10× (every other class carries 9.5k–18k). The reported sonnet value for the same row, 1.5428, implies billed = 2,777: 886 tokens more than haiku on the identical text pair, against an o200k delta of 1,800. Two tokenizers in one model family cannot differ by 49% on words drawn from a 24-word English vocabulary. Per-class haiku→sonnet gaps are 10%, 47%, 13%, 7%, 27%, 19% — the 47% outlier is exactly the row with the least signal, and it is 1/6 of the pool. Raising CG_CLASS_REPS on english_prose, or generating it at the same token scale as the others, should collapse it. Until it does, the 6.9% agreement between the two sonnet instruments is partly this artifact, and §3.3 is the sole reason §3 claims two independent instruments rather than one at n≈2 lineages.

"98.4% of the saving is on the sonnet-5/opus-5 tokenizer" cannot be a dollar share. The observed overall correction is 1.68497. A 98.4/1.6 dollar mix of 1.6857 and 1.3686 tops out at 1.6806; landing on 1.68497 needs 99.77% at 1.6857. It reconciles cleanly if 98.4% is a token share — haiku is far cheaper per token, so its dollar share is ~0.2%. As written it contradicts the headline it is explaining.

§7 is titled "Reproducing every number here" and has no command for any corrected column. Only the old figures have SQL. For a document whose purpose is that the next reader re-derives this rather than trusts it, that is the half needing the commands.

The end-to-end instrument does not assert its own cancellation premise. f is valid only if the two arms differ solely in messages, because the denominator is messages-only. The agent preset includes cachesplit, which rewrites the system block — so any system-side token change lands in the numerator uncancelled where the denominator is blind to it. Same bytes here, so it is one block boundary, and the sign is against the PR. But this test will be run against a preset containing toolfilter eventually, and then it silently reports a large fabricated f. Asserting system and tools byte-identical across arms makes the instrument defend itself.

---

Keep-alive: correctly untouched, and worth saying so

keepaliveSavedUSD counts n := e.CacheRead, or e.KeepAliveRefreshed when smaller — both provider-reported. Provider's count times provider's rate; the units already match, so there is nothing for f to fix. Same for cache_saved_usd. Keep-alive is the one savings figure on the page that never had #240.

That deserves a line in §6. A reader who has just been told every savings figure was wrong will go looking, and the silence reads as an oversight rather than a decision. Note it is indirectly affected on non-Anthropic models: with CacheWrite == Input now, the miss − CacheRead spread in both keepaliveSavedUSD and cachesplitSavedUSD drops from 1.15× to 0.9× input, about 22% lower. Correct, undocumented.

Minor

table.go now overrides an operator's own cache_write_frac from prices.yaml to 1.0 whenever the model name lacks "claude"/"anthropic". The modelinfo.go fallback fix is clearly right — nobody wrote that 1.25 down. But prices.yaml is a value the operator typed, and second-guessing it by substring match with no escape hatch is a different thing.

---

Verdict

The defect is real, the direction is settled, and the correction is applied to the right quantity — I tried to break that and could not. Before merge I would want 1 (the read paths silently backfill, making two views of one window disagree by 69% and disarming the reconciliation meant to catch exactly this) and 2 (the factor never round-trips, so the auditability the design rests on is not there). 3 is a line in §6 plus a decision about TotalSavedUSD.

I would not block on the measurement questions — the shipped constants are the conservative choice between the two instruments. But the size-dependence visible in the pooled-vs-median gap, the third and fourth independent readings both landing near 1.50, and the english_prose artifact in the corroborating instrument together suggest 1.6857 sits at the top of what the evidence supports for the rows carrying the money. Given this project's history with figures that were confidently wrong in the flattering direction, that belongs in the report rather than left for a reader to notice that pooled sits below median.

🤖 Generated with Claude Code