From d9dc02d7e9bcd902d5a440e0fcf1d39ac5832562 Mon Sep 17 00:00:00 2001 From: Osher-Elhadad Date: Mon, 7 Sep 2026 08:17:12 +0000 Subject: [PATCH] feat(kvcache): ship a fixed reuse model, and put the budget arm on the page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `BudgetPolicy` has been unreachable from every surface since it landed. The reason was structural rather than an oversight: it is built from a `Predictor`, no `Predictor` existed outside a test, and the registry's promise is that every name in it resolves. A name resolving to an arm with a nil `Predictor` would have been worse than no name — `Decide` returns `write_5m` when it has no opinion, so that arm never pings, and the page would have shown a flat write policy under a label promising a learned one. `kvcache.ReuseModelV1` removes the premise instead of the promise. It is a fixed discrete-time survival model — one coefficient per feature, one per one-hot level, compiled in and never re-fitted at runtime — so `NewKeepAliveBudget` builds the arm from nothing, `keepalive-budget` joins `Registry()`, and the KV-cache page picks it up with no dashboard change at all: `KVCacheArms()` probes the registry, so the checkbox, the baseline select and the per-(user,hour) suggest sweep all gain it for free. The fit is a logistic regression rather than `fit()`'s gradient-boosted pair, and it gives up 0.087 of dollar-AUC to be one. That is the price of shipping: thirty-odd coefficients can be read in a diff, so a re-fit that broke the model would be visible, and the live pinger consults the same model on the request path where the standing rule is "a rule, or a portable logistic-regression dot product -- never an embedded model or a call out of the hot path". `--db`/`--model-out`/`--go-out` are the wiring the Python's own docstring said was missing, so the numbers can be re-derived rather than quoted. Fitted on 18.4 days of the hosted capture: 219,650 idle spans, 37,418 of which reach the first sweep, holdout AUC 0.9065 by row and 0.7837 by dollar. Measured through the page's own scorer against `fixed-5m`: arm pings off the bill net per ping optimal (unreachable) 1,041 8.054% 3.069 keepalive-budget (cap 5) 8,790 2.065% 0.0932 keepalive-budget (cap 8) 9,003 2.059% 0.0907 keepalive-5m flat k=5 59,139 1.860% 0.0125 keepalive-budget (cap 2) 7,573 1.850% 0.0969 keepalive-5m flat k=2 25,586 1.808% 0.0280 keepalive-5m flat k=8 90,779 1.381% 0.0060 stop-reason-gated 14,969 0.710% 0.0188 On this window the arm beats every reachable alternative on money, which is NOT what the previous measurement found -- that one had a constant beating it by 0.89 pp. Two things changed and only one is the corpus; the other is that three specification defects were fixed, each of which is recorded in the doc because each is a trap worth not re-entering: - `sweep_k`, `log_elapsed` and `log_secs_to_deadline` are exact functions of the sweep index (the third is a constant), which a tree ensemble tolerates and a linear model cannot. The first fit produced -4.77 and +4.09 on two collinear features, a flat ~14% hazard at every sweep against a measured 4.86% falling to 0.24%, and an arm that bought its maximum budget on 97% of decisions and saved less than a flat MaxPings=2. The sweep index is a one-hot now: a free baseline hazard per sweep, whose recovered shape is not monotone. - the person-period frame labels the last sweep's survival against `inf`, so it is identically 0. Harmless as a truncation, fatal once the curve is composed into a CDF: F jumps to 1.0 at the tail and `Windows` reads h_maxk = (1-F)/(1-F) = 100%, which the induction propagates back through every earlier sweep. `gap_s > t_maxk + interval` is observable, so nothing needed censoring. - `cache_ttl` is self-consistent offline but a train/serve skew once wired in behind the seam: `Observation.TTL` is the tier the arm under test chose on the previous turn, not the tier the corpus recorded, so the feature is the model's own prior output fed back as an input. Dropped. `user_id` is dropped too -- a per-tenant dummy generalises to no new tenant and would put tenant identifiers in the binary, and `stat_p`/`log_stat_n` already carry that history. The fit weight is the one place an argument lost to a measurement, and both are recorded. Fitting unweighted should be right: the arm compares a probability against an 8.70% break-even, so it needs calibration rather than ranking, and a stake-weighted fit reports the hazard of a dollar rather than of a conversation -- fourfold different here. Measured, `stake` is worth 0.22 pp more of the bill and `none` is 11% better per ping. `stake` ships; `none` stays selectable, because the argument is good and the gap is near this corpus's seed noise. Two guards, because they cover different things. The existing `--fixture` one feeds a hand-written step CDF, so the coefficients, the feature derivation and the standardisation are all outside it. `kv_ttl_reusemodel_drift_test.go` drives `--score` over the checked-in `reusemodel_v1.json` -- the artifact the Go table was generated from, so a hand-edit of the generated file and an uncommitted regeneration both fail -- with a fixture carrying raw `Observation` fields rather than derived features, so a log1p applied to milliseconds on one side and seconds on the other is caught rather than agreed on. It compares the per-sweep survival vector, the CDF on and off the knot grid, the per-window h/s, and the budget. `TestUnbudgetedArmsAreUnaffectedByTheSeam` becomes an allow-list rather than "none". SILENTLY was the load-bearing word in its message and it still is; naming the set also catches the other direction, an arm dropping off the budget path and quietly running on the flat cap, which an assertion of "none" never could. No default changes outside the page's own arm set. `DefaultMaxPings` stays at 2, which still caps the arm below the 8 sweeps it reasons over -- it wins there anyway, and the doc says what the cap costs. Verification: go vet ./... clean gofmt -l . clean go build ./... clean go test ./... pass go test -race ./kvcache/ ./deploy/harbor/ ./dash/ pass (dash 479s) mkdocs build --strict clean kv_ttl_keepalive_policy.py --self-test pass Signed-off-by: Osher-Elhadad --- dash/kvcachesim.go | 14 +- deploy/harbor/kv_ttl_keepalive_policy.py | 994 +++++++++++++++++- deploy/harbor/kv_ttl_reusemodel_drift_test.go | 274 +++++ deploy/harbor/reusemodel_v1.json | 156 +++ docs/how-to/kv-cache-keepalive-budget.md | 211 +++- kvcache/keepalive_test.go | 27 +- kvcache/registry.go | 40 + kvcache/reusemodel.go | 551 ++++++++++ kvcache/reusemodel_test.go | 240 +++++ kvcache/reusemodel_v1_gen.go | 92 ++ 10 files changed, 2576 insertions(+), 23 deletions(-) create mode 100644 deploy/harbor/kv_ttl_reusemodel_drift_test.go create mode 100644 deploy/harbor/reusemodel_v1.json create mode 100644 kvcache/reusemodel.go create mode 100644 kvcache/reusemodel_test.go create mode 100644 kvcache/reusemodel_v1_gen.go diff --git a/dash/kvcachesim.go b/dash/kvcachesim.go index a04a22df9..6023faeab 100644 --- a/dash/kvcachesim.go +++ b/dash/kvcachesim.go @@ -77,6 +77,7 @@ const ( KVStrategyHistorical = kvcache.StrategyHistorical KVStrategyStopReasonGated = kvcache.StrategyStopReasonGated KVStrategyStickySession1h = kvcache.StrategyStickySession1h + KVStrategyKeepAliveBudget = kvcache.StrategyKeepAliveBudget KVStrategyOptimal = kvcache.StrategyOptimal // KVStrategyCustom is the one arm that is NOT in kvcache's registry, because it cannot be // built from a name: it carries the page's own thresholds and the in-process Predictor @@ -139,9 +140,20 @@ func isUnbuildable(name string) bool { return name == kvcache.StrategyReplay } // the only figure that says how much headroom exists at all, and an arm nobody sees by default is // one nobody compares against — and it is marked Unreachable so no surface can present it as a // result. +// +// `keepalive-budget` is the eleventh, and it is in the DEFAULT set rather than merely on the +// picker because the comparison it belongs in is the one this list already draws: it is the only +// arm here that spends a DIFFERENT number of pings per conversation, and against the flat +// keep-alive arms beside it that is the whole question. Measured on 18.4 days of the hosted +// capture it takes 2.07% off the bill where the best flat arm takes 1.86%, and does it with 8,790 +// pings against 59,139 — a figure that only means anything read next to those arms, which is what +// a default set is for. `Config.MaxPings` still bounds it like every other arm: at the page's +// default of 2 it is capped well below the 8 sweeps it reasons over, and +// docs/how-to/kv-cache-keepalive-budget.md says what that costs. var KVCacheDefaultStrategies = []string{KVStrategyNoCache, KVStrategyFixed5m, KVStrategyFixed1h, KVStrategyKeepAlive5m, KVStrategyKeepAlive5mOnce, KVStrategyKeepAlive1hOnce, - KVStrategyObserved, KVStrategyHistorical, KVStrategyStopReasonGated, KVStrategyOptimal} + KVStrategyObserved, KVStrategyHistorical, KVStrategyStopReasonGated, + KVStrategyKeepAliveBudget, KVStrategyOptimal} // kvCacheDefaultBaseline is the arm every saving is measured against when the caller names none, // and it comes from the REGISTRY rather than from a constant here. diff --git a/deploy/harbor/kv_ttl_keepalive_policy.py b/deploy/harbor/kv_ttl_keepalive_policy.py index b08641d37..7329bf040 100644 --- a/deploy/harbor/kv_ttl_keepalive_policy.py +++ b/deploy/harbor/kv_ttl_keepalive_policy.py @@ -9,14 +9,30 @@ code, not a query parameter". No dashboard page renders it.) kv_ttl_keepalive_policy.py --fixture f.json # the drift test's interface, stdlib only + kv_ttl_keepalive_policy.py --score f.json # the MODEL's drift interface, stdlib only kv_ttl_keepalive_policy.py --self-test # asserts this module's claims, stdlib only + kv_ttl_keepalive_policy.py --db cg.db --prices prices.yaml \ + --model-out reusemodel_v1.json --go-out ../../kvcache/reusemodel_v1_gen.go -There is deliberately no `--db` yet, unlike kv_ttl_cost_model.py and kv_ttl_predictor_arms.py: -`fit()` and `person_periods()` below are the fit AS IT WAS RUN, kept so the method is -reviewable, but they are not wired to an entry point — so the figures quoted in this docstring -cannot be reproduced from this file alone. Wiring `--db`/`--prices` the way the siblings do, -and reusing kv_ttl_survival_predictor's person-period expansion instead of the copy below, is -the next thing to do here. +`--db` fits the SHIPPED model and writes the two checked-in artifacts. It is the wiring this +docstring used to say was missing, and it exists so the numbers below can be re-derived rather +than quoted. + +Two models live here and the difference is the point: + + * `fit()` is the reference — a gradient-boosted pair, the fit AS IT WAS RUN, kept because it + is the method the measurements above were made with. It is NOT shippable: ~37 trees cannot + be reviewed in a diff, cannot be a dot product on the live pinger's request path, and have + no coefficients to port. + * `fit_portable()` is what ships — the same person-period frame, the same stake weights, the + same conditional target, fitted as a logistic regression so it exports as ~30 coefficients + that compile into `kvcache.ReuseModelV1`. `--db` prints both models' metrics side by side, + so the cost of going linear is measured on every re-fit rather than assumed once. + +That is what puts the arm on a dashboard page. `kvcache.BudgetPolicy` was kept out of +`kvcache.Registry()` because it "cannot be built from a name alone — it needs a `Predictor`"; +with the model compiled in, it can, and the registry keeps its promise that every name in it +resolves. See docs/how-to/kv-cache-keepalive-budget.md. WHAT PROBLEM THIS SOLVES THAT THE SURVIVAL PREDICTOR DOES NOT ------------------------------------------------------------- @@ -102,6 +118,8 @@ import argparse import json +import math +import os import sys from dataclasses import dataclass, field @@ -115,6 +133,13 @@ DEFAULT_LIFE_S = 300.0 DEFAULT_MAX_K = 8 +#: Where the sibling modules keep these, so a `--db` run reads the same corpus and the same +#: rates the cost model and the arm study did. +DEFAULT_DB = "/var/lib/context-guru/cg.db" +DEFAULT_PRICES = "/etc/context-guru/prices.yaml" +#: The checked-in fitted model, beside this file. `--score` reads it; nothing at runtime does. +DEFAULT_MODEL = "reusemodel_v1.json" + # ── the arithmetic the Go side must match, byte for byte ───────────────────── # @@ -247,6 +272,285 @@ def budget_for(cdf, cached_tokens: int, rates: Rates, *, return ping_budget(h, s, rates.rescue_value(cached_tokens), rates.ping_cost(cached_tokens)) +# ── the shipped model: a frozen dot product, and the CDF it composes ───────── +# +# Also stdlib-only and pure, and for the same reason: `--score` drives it from +# kv_ttl_reusemodel_drift_test.go, and a guard that needs numpy is a guard that does not run +# in CI. Everything the FIT needs stays below the divider; everything the SHIPPED model needs +# to be evaluated is here. +# +# WHY A LOGISTIC REGRESSION WHEN `fit()` BELOW IS A GRADIENT-BOOSTED ENSEMBLE +# --------------------------------------------------------------------------- +# Because the ensemble cannot be shipped. Three constraints meet here and only one shape +# satisfies all of them: +# +# * `kvcache.Predictor` is consulted inside `BudgetPolicy.Windows`, which is on the request +# path in the live pinger. proxy/keepalivestrategy.go's own rule for that path is +# explicit: "a rule, or ... a portable logistic-regression dot product — never an embedded +# model or a call out of the hot path." +# * The model has to be REVIEWABLE. A `HistGradientBoostingClassifier` is ~37 trees of +# thresholds; a reviewer cannot read a diff of it and say whether it got worse. +# * It has to be FIXED. A model that is re-fitted at startup makes yesterday's dashboard +# number unreproducible, and the kv-cache page's whole claim is that a historical window +# replays identically every time. +# +# So the shipped model is the same person-period survival model as `fit()`'s `survival` arm — +# the same frame, the same stake weights, the same conditional target — fitted as a logistic +# regression so that it exports as coefficients. What that costs against the ensemble is +# measured, not assumed: `--fit-report` prints both, and the figure is in +# docs/how-to/kv-cache-keepalive-budget.md. It is not free, and it is not large. +# +# `user_id` is deliberately NOT among the shipped features even though `FEATURES` below has +# it. A one-hot over tenant ids is a per-tenant dummy: it does not generalise to a tenant the +# fit never saw, it grows the artifact with every account, and it puts tenant identifiers in +# the shipped binary. `stat_p`/`log_stat_n` already carry that tenant's own history in a +# non-identifying form — which is what the accumulator is for. + +#: The shipped model's NUMERIC features, in coefficient order. The order is a wire contract: +#: Go reads the same list from the generated table and must apply it index for index. +#: `STATE_NUM` above carries `sweep_k`, `log_elapsed` and `log_secs_to_deadline`; none of the +#: three is here, and this is the one place the shipped model's SPECIFICATION differs from the +#: reference rather than only its estimator. All three are exact functions of the sweep index — +#: `log_elapsed` is log1p(k*interval), and `log_secs_to_deadline` is log1p(life - interval) at +#: EVERY sweep, a constant. In a tree ensemble that is merely redundant. In a linear model it is +#: perfect collinearity, and the first fit showed what that costs: coefficients of -4.77 and +#: +4.09 on two features that move together, summing to a hazard of ~14% at every sweep against +#: a measured 4.86% at sweep 1 falling to 0.24% by sweep 7. The arm then bought its maximum +#: budget on 97% of decisions and saved LESS than a flat MaxPings=2. +#: +#: The sweep index is in `SHIPPED_CAT` instead, one-hot — a free baseline hazard per sweep. That +#: is the standard discrete-time survival specification and it is strictly more general than the +#: linear term it replaces: the shape of the hazard over k is read off the corpus rather than +#: assumed monotone, and the measured shape is not monotone. +SHIPPED_NUM = ["log_prefix", "log_prev_gap", "turn", "hour_sin", "hour_cos", "dow_sin", + "dow_cos", "stat_p", "log_stat_n"] + +#: The shipped model's ONE-HOT features. Provider-level, so they generalise. +#: +#: `cache_ttl` is in `SPAN_CAT` above and deliberately NOT here, and the reason only becomes +#: visible once the model is wired in behind `kvcache.Predictor` — which is what this PR does +#: and what the reference fit never did: +#: +#: * Offline, `cache_ttl` is the tier the corpus RECORDED, read straight off the row. Fitting +#: on it and scoring on it is self-consistent, which is why the reference fit is fine. +#: * At serve time the seam hands over `Observation.TTL`, and kvcache/simulate.go sets that +#: from `st.tier` — the tier the ARM UNDER TEST chose on the previous turn. On a replay of +#: `keepalive-1h` every row reads `ephemeral_1h` no matter what the corpus recorded. +#: +#: So the feature means one thing in training and another in serving, and worse, the second +#: meaning is the model's own prior output fed back as an input. Neither is fixable by +#: re-fitting: the tier is endogenous to the arm. Dropped, and the ~0.13 spread its +#: coefficients carried is left on the table on purpose. +#: +#: `sweep` is the one-hot baseline hazard described above `SHIPPED_NUM`. Its level is the sweep +#: index as a decimal string, so it needs no machinery of its own beyond what `model` already +#: uses. The level for sweep `max_k + 1` is never fitted — there is no label past the horizon — +#: and never read either: `windows()` looks no further than `t_max_k + life`, which sits below +#: `t_(max_k+1)`. +SHIPPED_CAT = ["model", "sweep"] + +#: Hour and day-of-week are encoded as sin/cos pairs rather than as the raw integers +#: `STATE_NUM` carries. A linear model reads raw hour 23 and hour 0 as 23 units apart, which +#: is not a fact about time; the ensemble did not care because it splits. The pair costs one +#: extra coefficient each and makes the encoding honest. Measured effect on this corpus is +#: inside the seed-noise floor either way — see the negative results in the module docstring. +_HOUR_PERIOD = 24.0 +_DOW_PERIOD = 7.0 + + +def _level_of(span: dict, name: str, k: int) -> str: + """A one-hot level. `sweep` comes from the sweep index; everything else off the span. + + Model ids are case-folded. The corpus carries the same upstream under more than one + spelling — `Azure/gpt-4o` beside `azure/gpt-5.5` — because the id is whatever the client + sent, and two levels for one model split its evidence in half. `PriceBook` case-folds its + match keys for the same reason. Go's ReuseModel does `strings.ToLower` here too. + """ + if name == "sweep": + return str(int(k)) + return str(span.get(name) or "").lower() + + +def span_from_raw(*, ts_ms: int, cached_tokens: int, since_last_ms: int, turn: int, + stat_p: float, stat_n: int, model: str) -> dict: + """The span-level feature dict, derived from what `Observation` actually carries. + + The ONE place the derivation lives, so `load_spans` below and the `--score` guard cannot + disagree about it. Go's `ReuseModel.spanOf` is the counterpart and is compared against this + by kv_ttl_reusemodel_drift_test.go — which is why the guard's fixture carries the RAW fields + and not the logs: a fixture carrying precomputed features could not catch a log1p applied to + milliseconds on one side and seconds on the other. + """ + return { + "ts_ms": int(ts_ms), + "log_prefix": math.log1p(max(int(cached_tokens), 0)), + "log_prev_gap": math.log1p(max(int(since_last_ms), 0) / 1000.0), + "turn": float(turn), + "stat_p": float(stat_p), + "stat_n": int(stat_n), + "model": model or "", + } + + +def _sigmoid(z: float) -> float: + """1/(1+e^-z), written so a large-magnitude z cannot overflow either way.""" + if z >= 0.0: + return 1.0 / (1.0 + math.exp(-z)) + e = math.exp(z) + return e / (1.0 + e) + + +def _interp(x: float, xs: list[float], ys: list[float]) -> float: + """`numpy.interp` for a sorted grid: linear inside, CLAMPED to the endpoints outside. + + Clamping matters and is not a detail. A horizon past the last knot must read as the last + cumulative value, not extrapolate past 1.0; a horizon before the first must read 0. Go's + ReuseModel.cdf does the same, and kv_ttl_reusemodel_drift_test.go compares the two on + horizons deliberately placed outside the grid. + """ + if not xs: + return 0.0 + if x <= xs[0]: + return ys[0] + if x >= xs[-1]: + return ys[-1] + lo, hi = 0, len(xs) - 1 + while hi - lo > 1: + mid = (lo + hi) // 2 + if xs[mid] <= x: + lo = mid + else: + hi = mid + span = xs[hi] - xs[lo] + if span <= 0.0: + return ys[hi] + return ys[lo] + (ys[hi] - ys[lo]) * (x - xs[lo]) / span + + +@dataclass +class ReuseModel: + """A frozen discrete-time survival model over the sweep grid, as a portable dot product. + + One coefficient per shipped feature, one per one-hot level, one intercept, and the + standardisation the fit applied — because Go has to apply the identical transform or the + coefficients mean nothing. `kvcache.ReuseModelV1` is the generated Go twin of this, and + `--score` exists so a test can prove the twins agree. + + The model answers the CONDITIONAL question `s_k = P(still idle at t_(k+1) | idle at t_k)`, + which is what the person-period frame labels. The cumulative CDF `windows()` wants is + composed forwards from those, exactly as `FitResult.cdf_for` does it — see that method for + the two caveats about the grid, which apply here identically. + """ + + intercept: float + num_coef: dict[str, float] = field(default_factory=dict) + num_center: dict[str, float] = field(default_factory=dict) + num_scale: dict[str, float] = field(default_factory=dict) + #: feature -> level -> coefficient. The empty-string level is the catch-all an unseen + #: level falls to, so a new model id scores as "like the others" rather than as zero. + cat_coef: dict[str, dict[str, float]] = field(default_factory=dict) + interval_s: float = DEFAULT_INTERVAL_S + life_s: float = DEFAULT_LIFE_S + max_k: int = DEFAULT_MAX_K + #: Provenance. Not decoration: a shipped model whose training window nobody can name is a + #: number nobody can argue with. + trained: dict = field(default_factory=dict) + + # -- the feature vector, which must match _state_frame's state block ----- + + def state_features(self, span: dict, k: int) -> dict: + """One span's shipped features AS SEEN AT SWEEP k. + + The span block is carried; only the state block moves with k. Deliberately the same + arithmetic as `_state_frame` below, on the same fields, so that the shipped model and + the reference ensemble are fitted on the same frame and differ ONLY in the estimator. + """ + t_k = k * self.interval_s + secs = span["ts_ms"] / 1000.0 + t_k + hour = int(secs // 3600 % 24) + dow = int((secs // 86400 + 4) % 7) + return { + "log_prefix": float(span["log_prefix"]), + "log_prev_gap": float(span["log_prev_gap"]), + "turn": float(span["turn"]), + "hour_sin": math.sin(2.0 * math.pi * hour / _HOUR_PERIOD), + "hour_cos": math.cos(2.0 * math.pi * hour / _HOUR_PERIOD), + "dow_sin": math.sin(2.0 * math.pi * dow / _DOW_PERIOD), + "dow_cos": math.cos(2.0 * math.pi * dow / _DOW_PERIOD), + "stat_p": float(span["stat_p"]), + "log_stat_n": math.log1p(float(span["stat_n"])), + } + + def survival(self, span: dict, k: int) -> float: + """`s_k` for one span: the dot product, standardised, through the logistic.""" + f = self.state_features(span, k) + z = self.intercept + for name in SHIPPED_NUM: + scale = self.num_scale.get(name) or 1.0 + z += self.num_coef.get(name, 0.0) * (f[name] - self.num_center.get(name, 0.0)) / scale + for name in SHIPPED_CAT: + levels = self.cat_coef.get(name) or {} + level = _level_of(span, name, k) + # An unseen level falls to the catch-all rather than to zero. Zero is not + # "unknown", it is "exactly the reference level", which is a different claim. + z += levels.get(level, levels.get("", 0.0)) + return _sigmoid(z) + + def cdf_for(self, span: dict, *, interval_s: float | None = None, + life_s: float | None = None, max_k: int | None = None): + """A CUMULATIVE return-time CDF closure for one span, for `budget_for`/`windows`. + + `F(t_(j+1)) = 1 - prod_(i<=j) s_i`, on the grid {t_j} u {t_j + life}, linearly + interpolated between knots and clamped outside. F(t_1) = 0 by construction: this CDF + is already conditioned on having reached the first sweep, which is exactly the + population `windows()` divides by. + """ + iv = self.interval_s if interval_s is None else interval_s + life = self.life_s if life_s is None else life_s + mk = self.max_k if max_k is None else max_k + if mk <= 0: + mk = DEFAULT_MAX_K + surv, cum = 1.0, {0.0: 0.0} + for j in range(1, mk + 2): + t_j = j * iv + cum[t_j] = 1.0 - surv + surv *= self.survival(span, j) + cum[t_j + life] = 1.0 - surv + xs = sorted(cum) + ys = [cum[x] for x in xs] + return lambda seconds: _interp(float(seconds), xs, ys) + + # -- serialisation ------------------------------------------------------- + + def to_dict(self) -> dict: + return {"intercept": self.intercept, "num_coef": self.num_coef, + "num_center": self.num_center, "num_scale": self.num_scale, + "cat_coef": self.cat_coef, "interval_s": self.interval_s, + "life_s": self.life_s, "max_k": self.max_k, "trained": self.trained, + "shipped_num": SHIPPED_NUM, "shipped_cat": SHIPPED_CAT} + + @classmethod + def from_dict(cls, d: dict) -> "ReuseModel": + """Rebuild from `--model-out`'s JSON, refusing a feature list this code cannot score. + + A silently-ignored extra feature is a model scored on a subset of what it was fitted + on, which produces plausible numbers and no error at all. + """ + for key, want in (("shipped_num", SHIPPED_NUM), ("shipped_cat", SHIPPED_CAT)): + got = d.get(key) + if got is not None and list(got) != want: + raise ValueError(f"{key} mismatch: model has {got}, this code scores {want}") + return cls(intercept=float(d["intercept"]), + num_coef=dict(d.get("num_coef") or {}), + num_center=dict(d.get("num_center") or {}), + num_scale=dict(d.get("num_scale") or {}), + cat_coef={k: dict(v) for k, v in (d.get("cat_coef") or {}).items()}, + interval_s=float(d.get("interval_s", DEFAULT_INTERVAL_S)), + life_s=float(d.get("life_s", DEFAULT_LIFE_S)), + max_k=int(d.get("max_k", DEFAULT_MAX_K)), + trained=dict(d.get("trained") or {})) + + # ── the fit ────────────────────────────────────────────────────────────────── # # numpy/pandas/sklearn are imported INSIDE these functions, so that everything above stays @@ -411,6 +715,545 @@ def build(): "seed": seed}) +# ── fitting the shipped model, and where its training data comes from ──────── +# +# This block is what the module docstring called "the next thing to do here": the `--db` +# wiring the fit never had, so the figures can be reproduced rather than quoted. + + +#: How many closed gaps a stats cell needs before it is trusted over its parent. The same 6 +#: as kvcache/strategy.go's `minCell`, and it has to be: `stat_p`/`stat_n` are read from +#: `Observation.Stats` at serve time, so a fit that computed them with a different floor would +#: be training on features the model is never shown. +MIN_CELL = 6 + +#: The four six-hour bands of kvcache.BucketOf, which `Observation.Bucket` carries. +_BUCKETS = ("night", "morning", "afternoon", "evening") + + +def _bucket_at(ts_ms: int) -> str: + """kvcache.BucketAt in Python: the UTC six-hour band an instant falls in.""" + hour = int((ts_ms // 3_600_000) % 24) + if hour >= 18: + return "evening" + if hour >= 12: + return "afternoon" + if hour >= 6: + return "morning" + return "night" + + +class _History: + """kvcache.History in Python: the leak-free stats accumulator, gap by closing gap. + + A faithful port, not an approximation, because `stat_p`/`stat_n` are served to the model + at request time by the Go original. Same five fallback levels, same "count the gap in + EVERY level", same `MIN_CELL` floor, same global-cell last resort. `windows()` and the + break-even are pinned to Go by a drift test; this is pinned to Go by + TestReuseModelStatsMatchTheAccumulator, which replays one trajectory through both. + """ + + def __init__(self) -> None: + self._gaps: dict[tuple, list[float]] = {} + + @staticmethod + def _keys(user: str, model: str, bucket: str) -> list[tuple]: + return [(user, model, bucket), (user, model, ""), (user, "", ""), + ("", model, ""), ("", "", "")] + + def observe(self, user: str, model: str, bucket: str, gap_s: float) -> None: + """Record one CLOSED gap. Never called for a gap that has not happened yet.""" + if gap_s < 0.0: + gap_s = 0.0 + for k in self._keys(user, model, bucket): + self._gaps.setdefault(k, []).append(gap_s) + + def reuse_within(self, user: str, model: str, bucket: str, + horizon_s: float) -> tuple[float, int]: + """The share of this cell's closed gaps no longer than `horizon_s`, and the count.""" + ks = self._keys(user, model, bucket) + cell = None + for k in ks: + g = self._gaps.get(k) + if g is not None and len(g) >= MIN_CELL: + cell = g + break + if cell is None: + cell = self._gaps.get(ks[-1]) or None + if not cell: + return 0.0, 0 + hits = sum(1 for g in cell if g <= horizon_s) + return hits / len(cell), len(cell) + + +#: A successor that could not have read the entry however warm it was. The same two reasons +#: kvcache's stepCost excludes from `hit`, so a "rescue" here means what it means there. +_FULL_MISS = ("prefix_change", "cold_start") + + +def load_spans(db_path: str, prices_path: str, *, since_ms: int | None = None, + until_ms: int | None = None, horizon_s: float = 300.0): + """One row per IDLE SPAN, with the features the shipped model reads, read-only from cg.db. + + An idle span is opened by every request and closed by its successor in the same + conversation. `gap_s` is the seconds until that successor, and it is `inf` — an + unrescuable span — in two cases that are the same case: there is no successor inside the + window, or the successor was a full miss (`prefix_change`/`cold_start`), which could not + have read the entry however warm a ping kept it. + + `stake` is the DOLLARS at risk on the span: what one successful rescue avoids, at the + price list's own rates for that row's model. It is the fit's sample weight, and the + reason is in `fit()`'s docstring. + + Trajectory reading is delegated to `kv_ttl_cost_model.load_trajectories`, which owns the + read-only-URI quoting and the `keepalive = 0` exclusion — a ping counted as a turn invents + traffic and breaks the very gap it sits inside. + """ + import numpy as np + import pandas as pd + + from kv_ttl_cost_model import PriceBook, load_trajectories + + rows = load_trajectories(db_path, since_ms=since_ms, until_ms=until_ms) + book = PriceBook.from_operator_file(prices_path) + + # Group into trajectories so a successor is only ever this conversation's own. Sorted by + # (ts, id) for the same reason Simulate sorts: a tied timestamp must break deterministically. + by_conv: dict[tuple, list] = {} + for r in rows: + by_conv.setdefault(r.key, []).append(r) + for g in by_conv.values(): + g.sort(key=lambda r: (r.ts_ms, r.request_id)) + + hist = _History() + prev_ts: dict[tuple, int] = {} + prev_gap: dict[tuple, float] = {} + turn: dict[tuple, int] = {} + nxt: dict[int, object] = {} + for g in by_conv.values(): + for a, b in zip(g, g[1:]): + nxt[a.request_id] = b + + out = [] + # The SAME chronological walk Simulate does, in the same order, for the same reason: the + # accumulator must only ever hold gaps that closed at or before the decision instant. + for r in sorted(rows, key=lambda r: (r.ts_ms, r.request_id)): + key = r.key + # 1. close the previous span — the gap becomes history HERE and only here, bucketed at + # the PREDECESSOR's timestamp and priced at the current row's model, exactly as + # kvcache/simulate.go does it. + if key in prev_ts: + gap_s = max(0.0, (r.ts_ms - prev_ts[key]) / 1000.0) + hist.observe(r.user, r.model, _bucket_at(prev_ts[key]), gap_s) + prev_gap[key] = gap_s + # 2. the decision instant: everything below was knowable now. + turn[key] = turn.get(key, 0) + 1 + stat_p, stat_n = hist.reuse_within(r.user, r.model, _bucket_at(r.ts_ms), horizon_s) + prefix = int(r.cached_context or 0) + price = book.for_model(r.model) if hasattr(book, "for_model") else book.for_(r.model) + succ = nxt.get(r.request_id) + if succ is None or (succ.miss_reason or "") in _FULL_MISS: + gap = float("inf") + else: + gap = max(0.0, (succ.ts_ms - r.ts_ms) / 1000.0) + rescue_rate = (price.write_5m - price.cache_read) if price.known else 0.0 + row = span_from_raw(ts_ms=r.ts_ms, cached_tokens=prefix, + since_last_ms=int(1000.0 * prev_gap.get(key, 0.0)), + turn=turn[key], stat_p=stat_p, stat_n=stat_n, model=r.model) + row.update({"gap_s": gap, "stake": prefix * rescue_rate, "user_id": r.user, + "cache_ttl": (r.ttl_recorded or ""), "prefix": prefix}) + out.append(row) + prev_ts[key] = r.ts_ms + return pd.DataFrame(out) + + +def _shipped_design(spans, featuriser: ReuseModel, *, interval_s: float, life_s: float, + max_k: int, levels: dict | None = None, min_level_rows: int = 200): + """The person-period design matrix for the SHIPPED feature set. + + Same layout as `person_periods` — one row per (span, sweep it ACTUALLY reached) — and the + numeric block is produced by `ReuseModel.state_features` itself rather than by a second + copy of that arithmetic. That is deliberate: the featuriser used to FIT is the same code + path that scores at serve time, so the two cannot drift apart in a way a test would have + to notice. + + Returns (X, y, w, levels). `levels` maps each one-hot feature to its ordered level list, + with "" LAST as the catch-all that rare and unseen levels share — the same role + `OneHotEncoder(min_frequency=...)` plays in the reference fit. + """ + import numpy as np + + if levels is None: + levels = {} + for name in SHIPPED_CAT: + if name == "sweep": + # Enumerated rather than counted out of the frame: every sweep the induction can + # reach needs its own baseline, and a sweep that happened to be rare in this + # window must not be folded into the catch-all, which would hand it another + # sweep's hazard. + levels[name] = [str(j) for j in range(1, max_k + 1)] + [""] + continue + counts = spans[name].fillna("").astype(str).str.lower().value_counts() + keep = [str(v) for v, n in counts.items() if n >= min_level_rows and str(v) != ""] + levels[name] = sorted(keep) + [""] + + rows_num, rows_cat, ys, ws = [], [], [], [] + for k in range(1, max_k + 1): + t_k = k * interval_s + reached = spans[spans.gap_s > t_k] + if reached.empty: + break + # `t_k + interval_s` at EVERY sweep, including the last. + # + # `person_periods` above uses `inf` at k == max_k, which makes `y_survival` identically 0 + # there — "nobody survives past the horizon". For the reference's own purpose that is + # only a truncation. For a model whose survival curve is COMPOSED INTO A CDF it is a + # corruption of the tail, and it is not subtle: s_max_k fits to ~0, so F jumps to 1.0 at + # t_max_k + life, and `windows()` then computes + # + # h_max_k = (F(t_max_k + life) - F(deadline)) / (1 - F(t_max_k)) = (1-F)/(1-F) = 1.0 + # + # — a 100% hazard at the last sweep. The backward induction propagates that certainty + # back through every earlier sweep, so V_j > 0 everywhere and the arm buys its maximum + # budget on 100% of decisions, byte-identical to a flat MaxPings. Measured exactly that + # before this line changed. + # + # Nothing needs censoring here: `gap_s > t_max_k + interval_s` is observable for every + # span in the frame, so the label is available and the honest thing is to use it. + nxt_t = t_k + interval_s + for rec in reached.to_dict("records"): + f = featuriser.state_features(rec, k) + rows_num.append([f[n] for n in SHIPPED_NUM]) + onehot = [] + for name in SHIPPED_CAT: + lv = levels[name] + val = _level_of(rec, name, k) + idx = lv.index(val) if val in lv else len(lv) - 1 + onehot.extend(1.0 if i == idx else 0.0 for i in range(len(lv))) + rows_cat.append(onehot) + ys.append(1 if rec["gap_s"] > nxt_t else 0) + ws.append(max(0.0, float(rec["stake"]))) + if not rows_num: + raise ValueError("no span reached the first sweep; nothing to fit") + xn = np.asarray(rows_num, dtype=float) + xc = np.asarray(rows_cat, dtype=float) + y = np.asarray(ys, dtype=int) + w = np.asarray(ws, dtype=float) + # The same stake normalisation `fit()` uses, and the same epsilon: a zero-stake row still + # carries a little weight, because dropping it would silently change the population. + w = 1e-9 + w / (w.mean() or 1.0) + return np.hstack([xn, xc]), y, w, levels + + +def _weights(w, mode: str): + """The sample weight for a given `--weight` mode. + + `stake` is what `fit()` uses and its docstring defends: rows are not what the bill is made + of. That argument is about RANKING, and it is right about ranking. + + `none` exists because of an argument that turned out to be WRONG, and it is kept, and + still selectable, because the argument is a good one and the next corpus may vindicate it. + It runs: this model is not used for ranking. `BudgetPolicy` compares the probability against + the rates' own break-even — 8.70% — so what the arm needs is a CALIBRATED probability for + the span in front of it, and a stake-weighted fit is deliberately miscalibrated, reporting + the hazard of a dollar rather than of a conversation. The two differ fourfold on this corpus + (21.10% stake-weighted at sweep 1 against 4.86% by row) and the decision turns entirely on + which side of 8.70% the number lands. `log_prefix` is a covariate precisely so the size + effect survives without the weighting. + + Measured on the 18.4-day corpus, both fits replayed through the real arm against the same + fixed-5m baseline: + + weight pings off the bill net per ping budget 0 on + stake 9,003 2.059% 0.0907 41.4% + none 7,250 1.836% 0.1004 56.4% + + So `stake` is worth 0.22 pp more of the bill, and `none` is 11% better per ping. `stake` + ships because the money is the question the page is asked; the calibration argument bought + efficiency, not savings. What it did NOT do is what the argument predicted — being + better-calibrated against the break-even did not produce better decisions, it produced + FEWER of them. That is worth remembering before re-deriving the same argument. + """ + import numpy as np + + if mode == "stake": + return w + if mode == "none": + return np.ones_like(w) + raise ValueError(f"unknown weight mode {mode!r}; want stake or none") + + +def fit_portable(spans, *, interval_s: float = DEFAULT_INTERVAL_S, + life_s: float = DEFAULT_LIFE_S, max_k: int = DEFAULT_MAX_K, + seed: int = 0, weight: str = "stake") -> tuple: + """Fit the SHIPPED model: the survival arm, as a logistic regression that exports. + + Same frame and same stake weights as `fit()`; the estimator is the only difference, and + the whole reason is portability — see the shipped-model block above the fit divider. + + Returns (ReuseModel, report). The report carries the metrics the docs quote, plus the + reference ensemble's own metrics on the identical frame, so the cost of going linear is a + measured number rather than a claim. + """ + import numpy as np + from sklearn.ensemble import HistGradientBoostingClassifier + from sklearn.linear_model import LogisticRegression + from sklearn.metrics import log_loss, roc_auc_score + + shape = ReuseModel(intercept=0.0, interval_s=interval_s, life_s=life_s, max_k=max_k) + x, y, stake, levels = _shipped_design(spans, shape, interval_s=interval_s, life_s=life_s, + max_k=max_k) + w = _weights(stake, weight) + n_num = len(SHIPPED_NUM) + center, scale, degenerate = _standardise(x[:, :n_num]) + xs = x.copy() + xs[:, :n_num] = (xs[:, :n_num] - center) / scale + # A feature with no variance in the frame carries no information, and standardising it + # would divide by ~0 — which is not a rounding concern but a correctness one: the + # coefficient then multiplies float noise by 1/eps, so Python and Go disagree on the + # fourteenth decimal and the CDF disagrees in the second. `log_secs_to_deadline` is + # exactly this: with interval < life, `deadline - t_k` is `life - interval` at EVERY + # sweep, so the term is a constant the intercept already carries. Zeroed, listed, and + # kept in the feature vector so the wire contract with Go does not move. + xs[:, :n_num][:, degenerate] = 0.0 + + lr = LogisticRegression(max_iter=2000, C=1.0, random_state=seed) + lr.fit(xs, y, sample_weight=w) + coef = lr.coef_[0].copy() + coef[:n_num][degenerate] = 0.0 + + model = ReuseModel( + intercept=float(lr.intercept_[0]), + num_coef={n: float(coef[i]) for i, n in enumerate(SHIPPED_NUM)}, + num_center={n: float(center[i]) for i, n in enumerate(SHIPPED_NUM)}, + num_scale={n: float(scale[i]) for i, n in enumerate(SHIPPED_NUM)}, + cat_coef={}, interval_s=interval_s, life_s=life_s, max_k=max_k) + at = n_num + for name in SHIPPED_CAT: + lv = levels[name] + model.cat_coef[name] = {lv[i]: float(coef[at + i]) for i in range(len(lv))} + at += len(lv) + + def scored(p): + # Both AUCs, because they answer different questions and the module docstring is built + # on the difference: by ROW the model looks strong, by DOLLAR it is much weaker, and + # reporting only the first is how that gets missed. + # auc_dollar and the loss are always STAKE-weighted, whatever `--weight` fitted with: + # they answer "how well does this rank/price dollars", which does not change because the + # estimator stopped optimising for it. Reporting them under the fit's own weights would + # make the two modes' numbers incomparable, which is the one thing they are for. + return {"auc_row": float(roc_auc_score(y, p)), + "auc_dollar": float(roc_auc_score(y, p, sample_weight=stake)), + "logloss_dollar": float(log_loss(y, p, sample_weight=stake)), + "logloss_row": float(log_loss(y, p)), + "mean_pred": float(p.mean()), "mean_actual": float(y.mean())} + + gbm = HistGradientBoostingClassifier(max_iter=200, learning_rate=0.08, random_state=seed) + gbm.fit(xs, y, sample_weight=w) + report = { + "n_rows": int(len(y)), "n_spans": int(len(spans)), + "n_reached_first_sweep": int((spans.gap_s > interval_s).sum()), + "positive_rate": float(y.mean()), + "weight": weight, + "degenerate": [SHIPPED_NUM[i] for i, d in enumerate(degenerate) if d], + "in_sample": {"logistic": scored(lr.predict_proba(xs)[:, 1]), + "reference_gbm": scored(gbm.predict_proba(xs)[:, 1])}, + "holdout": _holdout(spans, interval_s=interval_s, life_s=life_s, max_k=max_k, + seed=seed, weight=weight), + "levels": {k: v for k, v in levels.items()}, + } + return model, report + + +def _standardise(xn): + """Per-column mean and standard deviation, with the no-variance columns flagged. + + The floor is relative to the column's own centre rather than absolute, because a feature + measured in log-seconds and one measured in turns do not share a scale at which "no + variance" means the same thing. + """ + import numpy as np + + center = xn.mean(axis=0) + scale = xn.std(axis=0) + degenerate = scale <= 1e-8 * (1.0 + np.abs(center)) + scale = scale.copy() + scale[degenerate] = 1.0 + center = np.where(degenerate, 0.0, center) + return center, scale, degenerate + + +def _holdout(spans, *, interval_s: float, life_s: float, max_k: int, seed: int, + weight: str = "none", train_frac: float = 0.7) -> dict: + """Fit on the EARLY part of the window and score the late part. + + Split by time, never at random. A random split puts the same conversation's sweeps on both + sides — the person-period layout means one span contributes up to `max_k` rows — so a + random holdout is scored on rows whose siblings it trained on, and reports a number the + deployment will not see. Rolling origin is what the reference measurement used; one + time-ordered cut is the cheap version of it, and it is the number to argue with. + """ + import numpy as np + from sklearn.linear_model import LogisticRegression + from sklearn.metrics import log_loss, roc_auc_score + + cut = float(spans.ts_ms.quantile(train_frac)) + early, late = spans[spans.ts_ms <= cut], spans[spans.ts_ms > cut] + if early.empty or late.empty: + return {"skipped": "window too short to split"} + shape = ReuseModel(intercept=0.0, interval_s=interval_s, life_s=life_s, max_k=max_k) + xtr, ytr, str_, levels = _shipped_design(early, shape, interval_s=interval_s, + life_s=life_s, max_k=max_k) + xte, yte, ste, _ = _shipped_design(late, shape, interval_s=interval_s, life_s=life_s, + max_k=max_k, levels=levels) + wtr = _weights(str_, weight) + n_num = len(SHIPPED_NUM) + center, scale, degenerate = _standardise(xtr[:, :n_num]) + + def prep(x): + out = x.copy() + out[:, :n_num] = (out[:, :n_num] - center) / scale + out[:, :n_num][:, degenerate] = 0.0 + return out + + lr = LogisticRegression(max_iter=2000, C=1.0, random_state=seed) + lr.fit(prep(xtr), ytr, sample_weight=wtr) + p = lr.predict_proba(prep(xte))[:, 1] + if len(np.unique(yte)) < 2: + return {"skipped": "holdout has one class only"} + return {"train_rows": int(len(ytr)), "test_rows": int(len(yte)), + "cut_ts_ms": int(cut), + "auc_row": float(roc_auc_score(yte, p)), + "auc_dollar": float(roc_auc_score(yte, p, sample_weight=ste)), + "logloss_dollar": float(log_loss(yte, p, sample_weight=ste)), + "logloss_row": float(log_loss(yte, p)), + "mean_pred": float(p.mean()), "mean_actual": float(yte.mean())} + + +def _go_float(v: float) -> str: + """A Go float literal that round-trips: 17 significant digits, never an integer.""" + s = repr(float(v)) + if "e" not in s and "E" not in s and "." not in s and "inf" not in s and "nan" not in s: + s += ".0" + return s + + +def emit_go(model: ReuseModel, report: dict, *, package: str = "kvcache") -> str: + """Render the fitted model as the Go source that ships it. + + Generated rather than hand-typed because a 30-coefficient table transcribed by hand is a + table with a typo in it, and checked in rather than loaded at runtime because a model read + from a file at startup makes yesterday's dashboard figure unreproducible. + """ + tr = dict(model.trained or {}) + ins = report.get("in_sample") or {} + lg, gb = ins.get("logistic") or {}, ins.get("reference_gbm") or {} + ho = report.get("holdout") or {} + degenerate = report.get("degenerate") or [] + lines = [ + "// Code generated by deploy/harbor/kv_ttl_keepalive_policy.py --go-out. DO NOT EDIT.", + "//", + "// Regenerate with:", + "//", + "//\tcd deploy/harbor && python3 kv_ttl_keepalive_policy.py \\", + "//\t --db /var/lib/context-guru/cg.db --prices /etc/context-guru/prices.yaml \\", + "//\t --model-out reusemodel_v1.json --go-out ../../kvcache/reusemodel_v1_gen.go", + "//", + "// Provenance. This is the argument for trusting the coefficients, so it is generated", + "// from the fit rather than written by hand — including the parts that are unflattering.", + "//", + f"//\twindow (UTC) {tr.get('window', '?')} ({tr.get('days', '?')} days)", + f"//\trequests {tr.get('requests', '?')}", + f"//\tidle spans {report.get('n_spans', '?')}", + f"//\t reaching sweep 1 {report.get('n_reached_first_sweep', '?')}" + " <- the population the decision is actually faced on", + f"//\tperson-periods {report.get('n_rows', '?')}", + f"//\tschedule interval {model.interval_s:.0f}s / life {model.life_s:.0f}s" + f" / max_k {model.max_k}", + f"//\tsurvived to sweep+1 {report.get('positive_rate', 0.0) * 100:.2f}% of person-periods", + "//", + f"//\tfit weight {report.get('weight', '?')}", + "//", + "//\t AUC by row AUC by dollar logloss mean p / actual", + f"//\tholdout (late 30%) {ho.get('auc_row', float('nan')):.4f} " + f"{ho.get('auc_dollar', float('nan')):.4f} " + f"{ho.get('logloss_row', float('nan')):.4f} " + f"{ho.get('mean_pred', float('nan')):.4f} / {ho.get('mean_actual', float('nan')):.4f}", + f"//\tin-sample {lg.get('auc_row', float('nan')):.4f} " + f"{lg.get('auc_dollar', float('nan')):.4f} " + f"{lg.get('logloss_row', float('nan')):.4f} " + f"{lg.get('mean_pred', float('nan')):.4f} / {lg.get('mean_actual', float('nan')):.4f}", + f"//\tin-sample, ref GBM {gb.get('auc_row', float('nan')):.4f} " + f"{gb.get('auc_dollar', float('nan')):.4f} " + f"{gb.get('logloss_row', float('nan')):.4f} " + f"{gb.get('mean_pred', float('nan')):.4f} / {gb.get('mean_actual', float('nan')):.4f}", + "//", + "// The HOLDOUT row is the one to argue with: it is fitted on the early 70% of the", + "// window and scored on the late 30%, split by time rather than at random because one", + "// span contributes up to max_k person-periods and a random split scores a row against", + "// its own siblings. The in-sample rows are here only to show the gap; the reference", + "// GBM's in-sample figures in particular are an upper bound on nothing.", + "//", + "// Neither AUC is a saving. What this arm is worth in money is the KV-cache page's own", + "// replay of a window under it, against the same baseline as every other arm — which is", + "// what shipping it as a registry arm is for. docs/how-to/kv-cache-keepalive-budget.md", + "// carries the measured table and says plainly where a flat MaxPings still beats it.", + "//", + ] + if degenerate: + lines += [ + f"// Zero-variance in this frame, so the coefficient is exactly 0: " + f"{', '.join(degenerate)}.", + "// Kept in the vector rather than removed, so the feature order Go reads does not", + "// move when a future re-fit on a different schedule makes the term informative.", + "//", + ] + # gofmt folds a comment block that abuts `package` into the package doc and strips the + # trailing "//". Emit the blank line it wants, so the generated file is already canonical + # and `gofmt -l` in CI stays silent without a formatting pass this script cannot run. + while lines and lines[-1] == "//": + lines.pop() + lines += [ + "", + f"package {package}", + "", + "// ReuseModelV1 is the fitted, frozen reuse model the `keepalive-budget` arm reads.", + "//", + "// See reusemodel.go for what the fields mean and for the CDF this composes. The", + "// numbers are a logistic survival fit on the corpus named above; the Python that", + "// produced them, the feature arithmetic they assume, and this table are pinned", + "// together by TestReuseModelV1AgreesWithThePort.", + "var ReuseModelV1 = &ReuseModel{", + f"\tVersion: {_go_str(tr.get('version', 'v1'))},", + f"\tTrainedOn: {_go_str(tr.get('window', ''))},", + f"\tIntervalS: {_go_float(model.interval_s)},", + f"\tLifeS: {_go_float(model.life_s)},", + f"\tMaxK: {model.max_k},", + f"\tIntercept: {_go_float(model.intercept)},", + "\tNumeric: []ReuseTerm{", + ] + for name in SHIPPED_NUM: + lines.append(f"\t\t{{Name: {_go_str(name)}, Coef: {_go_float(model.num_coef.get(name, 0.0))}, " + f"Center: {_go_float(model.num_center.get(name, 0.0))}, " + f"Scale: {_go_float(model.num_scale.get(name, 1.0))}}},") + lines.append("\t},") + lines.append("\tCategorical: []ReuseCategory{") + for name in SHIPPED_CAT: + lines.append(f"\t\t{{Name: {_go_str(name)}, Levels: []ReuseLevel{{") + for level, c in sorted((model.cat_coef.get(name) or {}).items()): + lines.append(f"\t\t\t{{Level: {_go_str(level)}, Coef: {_go_float(c)}}},") + lines.append("\t\t}},") + lines.append("\t},") + lines.append("}") + lines.append("") + return "\n".join(lines) + + +def _go_str(s) -> str: + """A Go string literal. json.dumps is exactly Go's quoting for the characters here.""" + return json.dumps("" if s is None else str(s)) + + # ── the drift test's interface ─────────────────────────────────────────────── @@ -499,21 +1342,156 @@ def f(x: float) -> float: hs, _ = windows(stay, max_k=2) assert hg[1] > 5 * hs[1], f"not conditional: {hg[1]} vs {hs[1]}" - print(f"self-test OK break-even {100 * be:.2f}% 13 features " - f"interval {DEFAULT_INTERVAL_S:.0f}s / life {DEFAULT_LIFE_S:.0f}s / max_k " + print(f"self-test OK break-even {100 * be:.2f}% " + f"{len(FEATURES)} reference / {len(SHIPPED_NUM) + len(SHIPPED_CAT)} shipped features" + f" interval {DEFAULT_INTERVAL_S:.0f}s / life {DEFAULT_LIFE_S:.0f}s / max_k " f"{DEFAULT_MAX_K}") return 0 +def _run_score(model_path: str, fixture_path: str) -> int: + """Score SPANS through a fitted model and print JSON. Stdlib only, so the guard runs anywhere. + + This is the model layer's counterpart to `--fixture`. `--fixture` feeds a hand-written step + CDF and therefore pins only the break-even/windows/induction arithmetic; the fitted model + was never on either side of it. `--score` closes that hole: it drives the SAME coefficients + the Go table carries, so a codegen slip, a feature-order change, a standardisation that Go + forgot to apply, or a divergent sin/cos convention shows up as a number, not as a + plausible-looking dashboard row. + + A fixture is {"cases":[{ts_ms, cached_tokens, since_last_ms, turn, stat_p, stat_n, model, + input_rate, min_prefix (optional), horizons:[seconds, …]}]} — RAW fields, exactly what + `kvcache.Observation` carries, so the feature DERIVATION is pinned too and not merely the + dot product. The output carries the per-sweep survival vector, the CDF at each requested + horizon, the per-window h/s, and the resulting budget — all four, because a budget that + agrees for the wrong reason is a guard that has already stopped working. + """ + with open(model_path, encoding="utf-8") as fh: + model = ReuseModel.from_dict(json.load(fh)) + with open(fixture_path, encoding="utf-8") as fh: + spec = json.load(fh) + interval = float(spec.get("interval_s", model.interval_s)) + life = float(spec.get("life_s", model.life_s)) + max_k = int(spec.get("max_k", model.max_k)) + surv, cdfs, budgets, wins = [], [], [], [] + for case in spec["cases"]: + span = span_from_raw( + ts_ms=case["ts_ms"], cached_tokens=case["cached_tokens"], + since_last_ms=case.get("since_last_ms", 0), turn=case.get("turn", 1), + stat_p=case.get("stat_p", 0.0), stat_n=case.get("stat_n", 0), + model=case.get("model", "")) + surv.append([model.survival(span, j) for j in range(1, max_k + 2)]) + cdf = model.cdf_for(span, interval_s=interval, life_s=life, max_k=max_k) + cdfs.append([cdf(float(h)) for h in case.get("horizons", [])]) + h, s = windows(cdf, interval_s=interval, life_s=life, max_k=max_k) + wins.append({"h": h, "s": s}) + rates = Rates.from_input_rate(float(case.get("input_rate", 3.0e-6))) + budgets.append(budget_for(cdf, int(case.get("cached_tokens", 0)), rates, + interval_s=interval, life_s=life, max_k=max_k, + min_prefix=int(case.get("min_prefix", 0)))) + print(json.dumps({"survival": surv, "cdf": cdfs, "windows": wins, "budgets": budgets})) + return 0 + + +def _run_fit(args) -> int: + """Fit the shipped model on a real corpus and write the two artifacts. + + The JSON is the fit's own output and the Go file is generated from it, so the pair can be + checked against each other — which is what `--score` plus + TestReuseModelV1AgreesWithThePort do. Neither is loaded at runtime: the Go table is + compiled in. + """ + import datetime as _dt + + spans = load_spans(args.db, args.prices, since_ms=args.since, until_ms=args.until) + model, report = fit_portable(spans, interval_s=args.interval, life_s=args.life, + max_k=args.max_k, seed=args.seed, weight=args.weight) + lo, hi = int(spans.ts_ms.min()), int(spans.ts_ms.max()) + fmt = "%Y-%m-%d %H:%M" + + def _utc(ms): + return _dt.datetime.fromtimestamp(ms / 1000.0, _dt.timezone.utc).strftime(fmt) + + model.trained = { + "version": args.version, + # The basename, not the path. The window and the row counts are the provenance that + # matters; whose home directory the snapshot happened to sit in is noise in a checked-in + # artifact, and one more thing to scrub before sharing it. + "corpus": os.path.basename(args.db), + "window": f"{_utc(lo)} -> {_utc(hi)} UTC", + "requests": int(len(spans)), + "days": round((hi - lo) / 86_400_000.0, 1), + "seed": args.seed, + "weight": args.weight, + } + if args.model_out: + payload = model.to_dict() + payload["report"] = report + with open(args.model_out, "w", encoding="utf-8") as fh: + json.dump(payload, fh, indent=1, sort_keys=True) + fh.write("\n") + print(f"wrote {args.model_out}", file=sys.stderr) + if args.go_out: + with open(args.go_out, "w", encoding="utf-8") as fh: + fh.write(emit_go(model, report)) + print(f"wrote {args.go_out}", file=sys.stderr) + lg = report["in_sample"]["logistic"] + gb = report["in_sample"]["reference_gbm"] + ho = report["holdout"] + print(f"fit OK spans {report['n_spans']} reaching sweep 1 " + f"{report['n_reached_first_sweep']} person-periods {report['n_rows']} " + f"window {model.trained['days']}d") + def line(tag, d): + return (f" {tag:19s} AUC row {d['auc_row']:.4f} AUC dollar {d['auc_dollar']:.4f}" + f" logloss {d['logloss_row']:.4f} mean p {d['mean_pred']:.4f}" + f" vs actual {d['mean_actual']:.4f}") + + print(f" fit weight {report['weight']}") + if "skipped" in ho: + print(f" holdout skipped: {ho['skipped']}") + else: + print(line("holdout (late 30%)", ho) + " <- the number to use") + print(line("in-sample", lg)) + print(line("in-sample, ref GBM", gb) + " (not shippable)") + if report["degenerate"]: + print(f" zero-variance, coefficient forced to 0: {', '.join(report['degenerate'])}") + return 0 + + def main(argv=None) -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--fixture", help="score a JSON fixture and print it (drift test)") ap.add_argument("--self-test", action="store_true", help="assert the module's claims on hand-written distributions") + ap.add_argument("--score", metavar="FIXTURE", + help="score spans through --model and print it (model drift test)") + ap.add_argument("--model", metavar="JSON", default=DEFAULT_MODEL, + help=f"the fitted model --score reads (default {DEFAULT_MODEL})") + ap.add_argument("--db", help="fit the shipped model on this cg.db (read-only)") + ap.add_argument("--prices", default=DEFAULT_PRICES, + help=f"the operator price list (default {DEFAULT_PRICES})") + ap.add_argument("--since", type=int, help="only requests at or after this epoch ms") + ap.add_argument("--until", type=int, help="only requests before this epoch ms") + ap.add_argument("--interval", type=float, default=DEFAULT_INTERVAL_S, + help="keep-alive cadence in seconds") + ap.add_argument("--life", type=float, default=DEFAULT_LIFE_S, + help="the tier's lifetime in seconds") + ap.add_argument("--max-k", type=int, default=DEFAULT_MAX_K, + help="how far the induction looks ahead") + ap.add_argument("--seed", type=int, default=0) + ap.add_argument("--weight", choices=("none", "stake"), default="stake", + help="fit sample weight; `stake` is the measured winner, see _weights") + ap.add_argument("--version", default="v1", help="the version label recorded in the model") + ap.add_argument("--model-out", help="write the fitted model as JSON") + ap.add_argument("--go-out", help="write the fitted model as Go source") args = ap.parse_args(argv) if args.fixture: return _run_fixture(args.fixture) + if args.score: + return _run_score(args.model, args.score) + if args.db: + return _run_fit(args) if args.self_test: return _self_test() ap.print_help() diff --git a/deploy/harbor/kv_ttl_reusemodel_drift_test.go b/deploy/harbor/kv_ttl_reusemodel_drift_test.go new file mode 100644 index 000000000..103debc7c --- /dev/null +++ b/deploy/harbor/kv_ttl_reusemodel_drift_test.go @@ -0,0 +1,274 @@ +package harbor + +// The shipped reuse model must mean the same thing in Go as in the Python that fitted it. +// +// kv_ttl_keepalive_drift_test.go beside this file already pins the DECISION arithmetic — the +// break-even, `windows`, the backward induction — by feeding both sides a hand-written step CDF. +// That guard never touches a fitted model: `--fixture` supplies the distribution, so the +// coefficients, the feature derivation and the standardisation are all outside it. Its own +// docstring says so ("the drift guard covers NEITHER"). +// +// This is that hole. `kvcache.ReuseModelV1` is a table generated from +// deploy/harbor/reusemodel_v1.json, and between the two sit a dozen chances to disagree +// silently: a log1p applied to milliseconds on one side and seconds on the other, a +// standardisation Go forgot, a sin/cos convention, a one-hot level that case-folds on one side +// only, a sweep index off by one, an interpolation that extrapolates instead of clamping. Every +// one of those produces a plausible number and no error at all. +// +// So the fixture carries RAW `Observation` fields, never derived features, and the comparison +// covers four layers rather than only the last: the per-sweep survival vector, the CDF at +// horizons deliberately placed on and off the knot grid, the per-window h/s that +// BudgetPolicy.Windows derives, and the budget itself. A budget that agrees for the wrong reason +// is a guard that has already stopped working. +// +// When they disagree, Go is right — it is what a replay is scored by. + +import ( + "encoding/json" + "math" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/rossoctl/context-guru/kvcache" +) + +// reuseCase is one span, in the terms Observation actually carries. Nothing here is a derived +// feature: that is the point (see the file comment). +type reuseCase struct { + TsMs int64 `json:"ts_ms"` + CachedTokens int64 `json:"cached_tokens"` + SinceLastMs int64 `json:"since_last_ms"` + Turn int `json:"turn"` + StatP float64 `json:"stat_p"` + StatN int `json:"stat_n"` + Model string `json:"model"` + InputRate float64 `json:"input_rate"` + MinPrefix int64 `json:"min_prefix,omitempty"` + Horizons []float64 `json:"horizons"` + + why string `json:"-"` +} + +type reuseFixture struct { + IntervalS float64 `json:"interval_s"` + LifeS float64 `json:"life_s"` + MaxK int `json:"max_k"` + Cases []reuseCase `json:"cases"` +} + +type reusePyResult struct { + Survival [][]float64 `json:"survival"` + CDF [][]float64 `json:"cdf"` + Windows []budgetWindows `json:"windows"` + Budgets []int `json:"budgets"` +} + +// fixedStats is a Stats that answers with whatever the case says, so the two sides read the +// same stat_p/stat_n without either having to replay a trajectory to get there. +type fixedStats struct { + p float64 + n int +} + +func (f fixedStats) ReuseWithin(_, _ string, _ kvcache.Bucket, _ time.Duration) (float64, int, string) { + return f.p, f.n, kvcache.LevelGlobal +} + +func (f fixedStats) MedianIdle(_, _ string, _ kvcache.Bucket) (time.Duration, int, string) { + return 0, f.n, kvcache.LevelGlobal +} + +// reuseDriftCases is the case table. Each row sits on a branch the two implementations could +// plausibly diverge on rather than being chosen to look varied. +// +// The horizon lists matter as much as the spans. `windows()` reads four horizons per sweep — +// t_j, the standing deadline, t_j + life and t_j + interval — and only two of those are knots. +// The rest are interpolated, and one of them (t_j + interval for the last sweep) sits past the +// end of the grid, where the two sides must CLAMP rather than extrapolate. +func reuseDriftCases() []reuseCase { + // On and off the knot grid at interval 280 / life 300: knots are 0, 280, 560, 580, 840, + // 860, ... so 300, 420, 700 and 2600 are all interpolated, 0 is the lower clamp and 99999 + // is past the last knot. + hz := []float64{0, 1, 280, 300, 420, 560, 580, 700, 840, 860, 1120, 1140, 2240, 2260, 2520, + 2540, 2600, 99999} + return []reuseCase{ + {why: "a large prefix on a known model, the population the money sits on", + TsMs: 1787654871634, CachedTokens: 124845, SinceLastMs: 0, Turn: 1, + StatP: 0.0, StatN: 0, Model: "aws/claude-opus-5", InputRate: 3.8e-06, Horizons: hz}, + {why: "the same span with a rich history, so stat_p and log_stat_n both move", + TsMs: 1787654871634, CachedTokens: 124845, SinceLastMs: 45_000, Turn: 37, + StatP: 0.93, StatN: 412, Model: "aws/claude-opus-5", InputRate: 3.8e-06, Horizons: hz}, + {why: "a model id the fit never saw, which must fall to the catch-all level", + TsMs: 1787654871634, CachedTokens: 60000, SinceLastMs: 10_000, Turn: 5, + StatP: 0.5, StatN: 20, Model: "nosuch/model-9", InputRate: 3.8e-06, Horizons: hz}, + {why: "a model id differing only in case, which must fold to the same level", + TsMs: 1787654871634, CachedTokens: 60000, SinceLastMs: 10_000, Turn: 5, + StatP: 0.5, StatN: 20, Model: "AWS/Claude-Opus-5", InputRate: 3.8e-06, Horizons: hz}, + {why: "a tiny prefix, where the ping's fixed overhead dominates the break-even", + TsMs: 1787654871634, CachedTokens: 500, SinceLastMs: 1_000, Turn: 2, + StatP: 0.2, StatN: 8, Model: "aws/claude-sonnet-5", InputRate: 1.52e-06, Horizons: hz}, + {why: "a 5x different rate scale, to prove the threshold does not move with it", + TsMs: 1787654871634, CachedTokens: 124845, SinceLastMs: 0, Turn: 1, + StatP: 0.0, StatN: 0, Model: "aws/claude-opus-5", InputRate: 1.9e-05, Horizons: hz}, + {why: "a MinPrefix gate above the prefix: a decision, not an absence of one", + TsMs: 1787654871634, CachedTokens: 5_000, SinceLastMs: 0, Turn: 1, + StatP: 0.0, StatN: 0, Model: "aws/claude-opus-5", InputRate: 3.8e-06, + MinPrefix: 20_000, Horizons: hz}, + {why: "an hour and weekday at the far side of both cycles, for the sin/cos convention", + TsMs: 1787698799000, CachedTokens: 90_000, SinceLastMs: 600_000, Turn: 12, + StatP: 0.75, StatN: 64, Model: "claude-opus-5", InputRate: 3.8e-06, Horizons: hz}, + {why: "turn and gap far out in the tail, where standardisation errors show up", + TsMs: 1787654871634, CachedTokens: 250_000, SinceLastMs: 7_200_000, Turn: 4_000, + StatP: 0.99, StatN: 5_000, Model: "azure/gpt-5.5", InputRate: 2.5e-06, Horizons: hz}, + {why: "a zero prefix, which the arm must decline rather than price", + TsMs: 1787654871634, CachedTokens: 0, SinceLastMs: 0, Turn: 1, + StatP: 0.0, StatN: 0, Model: "aws/claude-opus-5", InputRate: 3.8e-06, Horizons: hz}, + } +} + +// observationOf builds the Observation the Go side scores, from the same raw fields the fixture +// hands the Python. +func observationOf(c reuseCase) kvcache.Observation { + return kvcache.Observation{ + User: "u", Conversation: "c", Model: c.Model, RequestID: 1, + Now: c.TsMs, + HourUTC: time.UnixMilli(c.TsMs).UTC().Hour(), + Bucket: kvcache.BucketAt(c.TsMs), + CachedTokens: c.CachedTokens, + SinceLastMs: c.SinceLastMs, + TTL: kvcache.TTL5m, + ExpiresAt: c.TsMs + int64(kvcache.TTL5m.Lifetime()/time.Millisecond), + Turn: c.Turn, + Stats: fixedStats{p: c.StatP, n: c.StatN}, + Pricing: kvcache.Pricing{ + Known: true, Input: c.InputRate, Output: c.InputRate * 5.0, + CacheRead: c.InputRate * 0.10, Write5m: c.InputRate * 1.25, + Write1h: c.InputRate * 2.00, + PingInputTokens: 1, + PingOutputTokens: 1, + }, + } +} + +func TestReuseModelV1AgreesWithThePort(t *testing.T) { + if err := kvcache.ReuseModelV1.Validate(); err != nil { + t.Fatalf("the shipped model is not scoreable by this build: %v", err) + } + py := pythonBinFor(t, "kv_ttl_keepalive_policy") + if py == "" { + t.Skip("no python3 that can import kv_ttl_keepalive_policy") + } + model := kvcache.ReuseModelV1 + fx := reuseFixture{IntervalS: model.IntervalS, LifeS: model.LifeS, MaxK: model.MaxK, + Cases: reuseDriftCases()} + blob, err := json.Marshal(fx) + if err != nil { + t.Fatalf("marshal: %v", err) + } + path := filepath.Join(t.TempDir(), "reuse_fixture.json") + if err := os.WriteFile(path, blob, 0o600); err != nil { + t.Fatalf("write fixture: %v", err) + } + // The model the Python scores is the CHECKED-IN artifact the Go table was generated from. + // That is the comparison worth making: it catches a hand-edit of the generated file and a + // regeneration that was never committed, neither of which a shared in-memory model would. + out, err := exec.Command(py, "kv_ttl_keepalive_policy.py", "--score", path, + "--model", "reusemodel_v1.json").Output() + if err != nil { + t.Fatalf("python --score: %v", err) + } + var got reusePyResult + if err := json.Unmarshal(out, &got); err != nil { + t.Fatalf("parse python output %q: %v", strings.TrimSpace(string(out)), err) + } + if len(got.Budgets) != len(fx.Cases) { + t.Fatalf("python answered %d cases, fixture had %d", len(got.Budgets), len(fx.Cases)) + } + + // 1e-9 absolute. The two sides do the same arithmetic in the same order in IEEE754 doubles, + // so anything above float noise is a real divergence rather than a tolerance question — a + // loose tolerance here would hide exactly the bugs this exists to catch. + const tol = 1e-9 + for i, c := range fx.Cases { + o := observationOf(c) + pol := kvcache.BudgetPolicy{Predictor: model, Interval: time.Duration(model.IntervalS) * + time.Second, MaxK: model.MaxK, MinPrefix: c.MinPrefix} + + // (a) the CDF, at horizons on and off the knot grid. + for j, hz := range c.Horizons { + want := got.CDF[i][j] + have, ok := model.ReuseProbability(o, time.Duration(hz*float64(time.Second))) + if !ok { + t.Errorf("case %d (%s): Go declined horizon %gs, python answered %.12f", + i, c.why, hz, want) + continue + } + if math.Abs(have-want) > tol { + t.Errorf("case %d (%s): F(%gs) Go %.12f vs python %.12f (delta %.3g)", + i, c.why, hz, have, want, have-want) + } + } + + // (b) the per-window hazard and survival BudgetPolicy derives from that CDF. Compared + // because a CDF that agrees and windows that do not means the schedule is being read + // differently, which no budget comparison alone would localise. + h, s, ok := pol.Windows(o) + if !ok { + if c.CachedTokens > 0 { + t.Errorf("case %d (%s): Windows declined a priced observation", i, c.why) + } + continue + } + for j := range got.Windows[i].H { + if j >= len(h) { + t.Errorf("case %d (%s): Go returned %d windows, python %d", + i, c.why, len(h), len(got.Windows[i].H)) + break + } + if math.Abs(h[j]-got.Windows[i].H[j]) > tol { + t.Errorf("case %d (%s): h[%d] Go %.12f vs python %.12f", + i, c.why, j, h[j], got.Windows[i].H[j]) + } + if math.Abs(s[j]-got.Windows[i].S[j]) > tol { + t.Errorf("case %d (%s): s[%d] Go %.12f vs python %.12f", + i, c.why, j, s[j], got.Windows[i].S[j]) + } + } + + // (c) the decision. + budget, bok := pol.PingBudget(o) + if !bok { + // The Python has no cap to fall back to, so it reports 0 where Go reports + // (0, false) — see budget_for's own docstring. Only a nonzero python budget is a + // disagreement here. + if got.Budgets[i] != 0 { + t.Errorf("case %d (%s): Go had no opinion, python chose %d", + i, c.why, got.Budgets[i]) + } + continue + } + if budget != got.Budgets[i] { + t.Errorf("case %d (%s): budget Go %d vs python %d\n Go h=%v\n py h=%v", + i, c.why, budget, got.Budgets[i], h, got.Windows[i].H) + } + } +} + +// TestReuseModelV1IsValid keeps a re-fit that adds or renames a feature from shipping silently. +// +// Validate refuses a table naming a term this build cannot evaluate, and the failure it prevents +// is the quiet one: an unknown feature would otherwise be scored against zero, which is not an +// error, not a crash, and not the model that was fitted. +func TestReuseModelV1IsValid(t *testing.T) { + if err := kvcache.ReuseModelV1.Validate(); err != nil { + t.Fatalf("shipped model invalid: %v", err) + } + if kvcache.ReuseModelV1.Version == "" || kvcache.ReuseModelV1.TrainedOn == "" { + t.Error("the shipped model carries no provenance; a saving attributed to it could not " + + "be traced to a corpus or a window") + } +} diff --git a/deploy/harbor/reusemodel_v1.json b/deploy/harbor/reusemodel_v1.json new file mode 100644 index 000000000..54bc337f3 --- /dev/null +++ b/deploy/harbor/reusemodel_v1.json @@ -0,0 +1,156 @@ +{ + "cat_coef": { + "model": { + "": 0.044073416904029585, + "aws/claude-haiku-4-5": 0.01059219829200029, + "aws/claude-opus-5": 0.5891045430667132, + "aws/claude-sonnet-5": 0.4672028093860629, + "azure/gpt-4o": 0.04754489687167924, + "azure/gpt-5.5": 0.43532292998608124, + "claude-haiku-4-5": 0.05693310295387887, + "claude-opus-4-6": 0.8914374258675521, + "claude-opus-4-7": 0.20280481953179896, + "claude-opus-4-8": 0.5671313782762598, + "claude-opus-5": 0.759769916393038, + "claude-sonnet-4-6": 0.6687443839651331, + "claude-sonnet-5": 0.24100295418844414 + }, + "sweep": { + "": 0.0, + "1": -0.8103139873353238, + "2": -0.04573723713715996, + "3": 0.6255276686681279, + "4": 0.8279466337110583, + "5": 0.35705180846007445, + "6": 0.6740656989771248, + "7": 1.8645461417180886, + "8": 1.4885780486208022 + } + }, + "intercept": 5.018342744954828, + "interval_s": 280.0, + "life_s": 300.0, + "max_k": 8, + "num_center": { + "dow_cos": -0.27736442403982214, + "dow_sin": 0.5289808564926407, + "hour_cos": -0.12480005679454068, + "hour_sin": -0.03876981083782035, + "log_prefix": 4.146090828434305, + "log_prev_gap": 0.8069032678784968, + "log_stat_n": 5.707126141180795, + "stat_p": 0.8669557007725781, + "turn": 93.21953452687274 + }, + "num_coef": { + "dow_cos": 0.06955802004724103, + "dow_sin": -0.044776344705750655, + "hour_cos": 0.05351815398273302, + "hour_sin": -0.11807319204324197, + "log_prefix": -2.0617767376699336, + "log_prev_gap": -0.051073693614662456, + "log_stat_n": -0.028663327452458465, + "stat_p": 0.3068298285865256, + "turn": 0.17222201868549095 + }, + "num_scale": { + "dow_cos": 0.5331507814789217, + "dow_sin": 0.5991648136759256, + "hour_cos": 0.7038772883647778, + "hour_sin": 0.698196684692585, + "log_prefix": 4.7037498066187835, + "log_prev_gap": 1.7739566028829084, + "log_stat_n": 1.7859544188184382, + "stat_p": 0.13350760774270665, + "turn": 539.4799156608967 + }, + "report": { + "degenerate": [], + "holdout": { + "auc_dollar": 0.783682670246469, + "auc_row": 0.906474222098225, + "cut_ts_ms": 1788300247681, + "logloss_dollar": 0.239557851020535, + "logloss_row": 0.03452489585405714, + "mean_actual": 0.9918776371308017, + "mean_pred": 0.9868032053406215, + "test_rows": 75840, + "train_rows": 200481 + }, + "in_sample": { + "logistic": { + "auc_dollar": 0.7597075790054804, + "auc_row": 0.8854908284878285, + "logloss_dollar": 0.2849597379074432, + "logloss_row": 0.05733733212388787, + "mean_actual": 0.9853467525088575, + "mean_pred": 0.9826050730741852 + }, + "reference_gbm": { + "auc_dollar": 0.8704567592804243, + "auc_row": 0.9572464839215526, + "logloss_dollar": 0.23746905140427646, + "logloss_row": 0.05027501032065765, + "mean_actual": 0.9853467525088575, + "mean_pred": 0.9775838466946333 + } + }, + "levels": { + "model": [ + "aws/claude-haiku-4-5", + "aws/claude-opus-5", + "aws/claude-sonnet-5", + "azure/gpt-4o", + "azure/gpt-5.5", + "claude-haiku-4-5", + "claude-opus-4-6", + "claude-opus-4-7", + "claude-opus-4-8", + "claude-opus-5", + "claude-sonnet-4-6", + "claude-sonnet-5", + "" + ], + "sweep": [ + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "" + ] + }, + "n_reached_first_sweep": 37418, + "n_rows": 276321, + "n_spans": 219650, + "positive_rate": 0.9853467525088575, + "weight": "stake" + }, + "shipped_cat": [ + "model", + "sweep" + ], + "shipped_num": [ + "log_prefix", + "log_prev_gap", + "turn", + "hour_sin", + "hour_cos", + "dow_sin", + "dow_cos", + "stat_p", + "log_stat_n" + ], + "trained": { + "corpus": "cg.db", + "days": 18.4, + "requests": 219650, + "seed": 0, + "version": "v1", + "weight": "stake", + "window": "2026-08-17 11:48 -> 2026-09-04 21:45 UTC" + } +} diff --git a/docs/how-to/kv-cache-keepalive-budget.md b/docs/how-to/kv-cache-keepalive-budget.md index b606458c2..e8bc369b8 100644 --- a/docs/how-to/kv-cache-keepalive-budget.md +++ b/docs/how-to/kv-cache-keepalive-budget.md @@ -249,11 +249,156 @@ a 5× volume ramp across the window, so the weekday label is partly a label for date. **Separating a weekly cycle from a date effect needs 8–12 weeks**, and until then no feature question on this corpus is answerable. -## How to use it +## Turning it on + +Pick it on the **KV-cache** page. It is the `keepalive-budget` checkbox, it is in the default +arm set, and it is scored against the same baseline, the same rates and the same savings +arithmetic as every other arm. + +That is new. `BudgetPolicy` used to be unreachable from any page, and the reason was structural +rather than an oversight: it is built from a `Predictor`, no `Predictor` existed outside a test, +and the registry's promise is that every name in it resolves. A name resolving to an arm with a +nil `Predictor` would have been worse than no name at all — `Decide` returns `write_5m` when it +has no opinion, so that arm never pings, and the page would have shown a flat write policy under +a label promising a learned one. + +[`kvcache.ReuseModelV1`](#the-shipped-model) removed the premise instead of the promise. The +model is a table of coefficients compiled into the binary, so `NewKeepAliveBudget` builds the arm +from nothing and the name resolves on every process. Nothing else about the seam changed: a +caller with its own predictor still constructs `BudgetPolicy` directly and gets its own label. + +!!! warning "`Config.MaxPings` still caps it, and the page defaults to 2" + A budget can ask for fewer pings than the cap, never more. The arm reasons over 8 sweeps, so + at the page's default `k=2` it is bounded to a quarter of its own horizon. It still wins + there — see the table below — but if you want to see the schedule it would actually choose, + set the keep-alive cap to 8. + +### The shipped model + +`ReuseModelV1` is a **fixed** discrete-time survival model: one coefficient per shipped feature, +one per one-hot level, one intercept, compiled in and never re-fitted at runtime. The +coefficients and their full provenance are in `kvcache/reusemodel_v1_gen.go`, generated; the +arithmetic that reads them is in `kvcache/reusemodel.go`, hand-written. + +| | | +|---|---| +| corpus | 18.4 days of the hosted capture, 2026-08-17 → 2026-09-04 UTC | +| 219,650 idle spans | of which **37,418 reach the first sweep** — the population the decision is faced on | +| 276,321 person-periods | one row per (span, sweep it actually reached) | +| holdout AUC | **0.9065** by row, **0.7837** by dollar, fitted on the early 70% and scored on the late 30% | +| reference GBM, in-sample | 0.9572 by row, 0.8705 by dollar — the ceiling a shippable model gives up | + +Three things about it are worth arguing with. + +**It is a logistic regression, not the gradient-boosted pair `fit()` uses.** Not because the +ensemble is worse — it is better, and the generated file prints by how much on every re-fit — +but because it cannot ship. Thirty-seven trees of thresholds cannot be reviewed in a diff, so +nobody could tell a re-fit that improved the model from one that broke it; and the live pinger +consults the same model on the request path, where `proxy/keepalivestrategy.go`'s own rule is "a +rule, or … a portable logistic-regression dot product — never an embedded model or a call out of +the hot path." + +**It is fixed, and that is a feature.** A model re-fitted at startup would make yesterday's +dashboard figure unreproducible, and this page's whole claim is that replaying a historical +window gives the same answer every time. + +**The sweep index is a one-hot, not a linear term.** This is the one place the shipped model's +specification departs from the reference's, and it was not a preference. `sweep_k`, +`log_elapsed` and `log_secs_to_deadline` are all exact functions of the sweep index — the third +is `life - interval` at *every* sweep, a constant. A tree ensemble treats that as redundancy; a +linear model cannot, and the first fit showed what it costs: coefficients of −4.77 and +4.09 on +two features that move together, summing to a hazard of ~14% at every sweep where the corpus +measures 4.86% at the first and 0.24% by the seventh. The arm bought its maximum budget on 97% +of decisions and saved *less than a flat `MaxPings=2`*. A free coefficient per sweep recovers the +real shape, which rises from −1.25 at the first sweep to +1.83 at the seventh and then falls — +not monotone, which is exactly why assuming monotone was wrong. + +`user_id` and `cache_ttl` are both in the reference's 13 features and neither is shipped. +`user_id` is a per-tenant dummy: it does not generalise to a tenant the fit never saw and it +would put tenant identifiers in the binary, while `stat_p`/`log_stat_n` already carry that +tenant's own history in a non-identifying form. `cache_ttl` is subtler and only becomes wrong +once the model is actually wired in behind `Predictor`: offline it is the tier the corpus +*recorded*, but at serve time the seam hands over `Observation.TTL`, which `simulate.go` sets +from the tier **the arm under test chose on the previous turn**. Training on one and serving the +other is a train/serve skew, and the second meaning is the model's own prior output fed back as +an input. Dropping it cost 0.0001 of holdout dollar-AUC. + +### What it is worth, measured + +Same corpus, replayed through the real arm on the KV-cache page's own scorer, against the +`fixed-5m` baseline the page uses. `optimal` reads the future and is a ceiling, never a result. + +| arm | pings | off the bill | of the ceiling | net per ping | +|---|---:|---:|---:|---:| +| `optimal` *(unreachable)* | 1,041 | **8.054%** | 100.0% | 3.069 | +| **`keepalive-budget`, cap 5** | **8,790** | **2.065%** | 25.6% | **0.0932** | +| **`keepalive-budget`, cap 8** | **9,003** | **2.059%** | 25.6% | **0.0907** | +| `keepalive-5m` flat `k=5` | 59,139 | 1.860% | 23.1% | 0.0125 | +| **`keepalive-budget`, cap 2** | **7,573** | **1.850%** | 23.0% | **0.0969** | +| `keepalive-5m` flat `k=6` | 69,864 | 1.817% | 22.6% | 0.0103 | +| `keepalive-5m` flat `k=2` | 25,586 | 1.808% | 22.4% | 0.0280 | +| `keepalive-5m-once` | 13,538 | 1.516% | 18.8% | 0.0444 | +| `keepalive-5m` flat `k=8` | 90,779 | 1.381% | 17.1% | 0.0060 | +| `stop-reason-gated` | 14,969 | 0.710% | 8.8% | 0.0188 | +| `fixed-5m` *(baseline)* | 0 | 0.000% | — | — | +| `historical-probability` | 0 | −0.128% | — | — | + +**On this window the arm beats every reachable alternative on money, which is not what the +earlier measurement found.** The version of this page written against the previous corpus +reported a flat constant beating the arm by 0.89 pp; here `keepalive-budget` takes 2.07% against +the best flat arm's 1.86%, and does it with **8,790 pings against 59,139** — 7.5× the value per +ping. Two things changed and only one of them is the corpus: the window is longer (18.4 days +against 16.9), and the specification defects above were fixed. Read the earlier figure as +measuring a differently-specified model. + +The budget genuinely varies per conversation, which is the entire point of the arm and the one +thing a flat `MaxPings` cannot do: + +| budget chosen | share of decisions | +|---:|---:| +| 0 | 41.4% | +| 1 | 31.9% | +| 2 | 22.5% | +| 5 | 1.2% | +| 6 | 3.0% | +| 8 | 0.0% | + +Note the shape: **41.4% of conversations are worth no ping at all**, and the arm declines them. +That is where the 6.6× reduction in ping volume comes from, and it is why the per-ping column +moves so much more than the money column. + +### The fit weight: an argument that lost + +The fit is weighted by **dollars at risk**, and the reasoning said it should not be. A +stake-weighted fit is deliberately miscalibrated — it reports the hazard of a dollar, not of a +conversation, and the two differ fourfold here (21.10% stake-weighted at the first sweep against +4.86% by row). `BudgetPolicy` compares that probability against a break-even of 8.70%, so the +whole decision turns on which side of 8.70% the number lands, and `log_prefix` is a covariate +precisely so the size effect survives without the weighting. + +Both were fitted and both were replayed through the arm: + +| `--weight` | pings | off the bill | net per ping | budget 0 on | +|---|---:|---:|---:|---:| +| `stake` | 9,003 | **2.059%** | 0.0907 | 41.4% | +| `none` | 7,250 | 1.836% | **0.1004** | 56.4% | + +`stake` ships, because money is the question the page is asked. What the calibration argument +predicted did not happen: being better calibrated against the break-even did not produce better +decisions, it produced **fewer** of them. `--weight none` is still selectable, and the argument +is still a good one — worth re-testing on a longer window rather than re-deriving. -`BudgetPolicy` is deliberately **not** in `Registry()`. Like `Custom`, it cannot be built from -a name alone — it needs a `Predictor` — and the registry's promise is that every name in it -resolves. Construct it and hand it to `Simulate`: +## How to use it from Go + +The arm is in the registry, so a name is enough: + +```go +s, _ := kvcache.NewStrategy(kvcache.StrategyKeepAliveBudget, nil, cfg) +result := kvcache.Simulate(reqs, s, cfg) +``` + +With your own predictor instead of the shipped one, construct it directly and hand it to +`Simulate`: ```go pol := kvcache.BudgetPolicy{ @@ -314,14 +459,28 @@ budgeted by `Config.MaxPings` exactly as before. ```sh cd deploy/harbor python3 kv_ttl_keepalive_policy.py --self-test # asserts the module's claims, stdlib only -python3 kv_ttl_keepalive_policy.py --fixture f.json +python3 kv_ttl_keepalive_policy.py --fixture f.json # the decision arithmetic's guard +python3 kv_ttl_keepalive_policy.py --score f.json # the shipped MODEL's guard + +# re-fit the shipped model and regenerate both checked-in artifacts +python3 kv_ttl_keepalive_policy.py \ + --db /var/lib/context-guru/cg.db --prices /etc/context-guru/prices.yaml \ + --model-out reusemodel_v1.json --go-out ../../kvcache/reusemodel_v1_gen.go ``` +`--db` is what this page used to say was missing. The fit reads the corpus read-only, builds the +person-period frame itself, and writes two artifacts: `reusemodel_v1.json`, which is the fit's own +output, and `kvcache/reusemodel_v1_gen.go`, generated from it. Both are checked in, and the pair +being checked against each other is the point — see `--score` below. The fit needs pandas and +scikit-learn; everything else here does not. + Everything above the `── the fit ──` divider is **stdlib-only and pure**: `break_even`, `windows`, `ping_budget`, `Rates`, `budget_for`. The scientific stack is imported inside the fitting functions only, so the drift guard runs in a plain CI container. A guard that cannot run is an absent guard. +There are **two** guards, because they cover different things. + `kv_ttl_keepalive_drift_test.go` drives that `--fixture` entry point and compares the budget **and** both intermediate vectors across 11 cases chosen to sit on branches the two implementations could plausibly diverge on — either side of the break-even to within 2 basis @@ -330,14 +489,33 @@ where most have already returned, a small prefix where the ping's fixed overhead a rate scale 5× different to prove the threshold does not move. Deleting the survivor divide from the Python makes it fail with `port 0.05` against `Go 0.50`, which is the check working. +What that guard cannot see is the **model**: `--fixture` supplies the distribution as a +hand-written step CDF, so the coefficients, the feature derivation and the standardisation are all +outside it. `kv_ttl_reusemodel_drift_test.go` closes that hole. It drives `--score` over the +checked-in `reusemodel_v1.json` — the artifact the Go table was generated from, so a hand-edit of +the generated file or a regeneration that was never committed both fail — and its fixture carries +**raw `Observation` fields, never derived features**, so a `log1p` applied to milliseconds on one +side and seconds on the other is caught rather than agreed on. It compares four layers: the +per-sweep survival vector, the CDF at horizons deliberately placed on and off the knot grid, the +per-window `h`/`s` that `Windows` derives, and the budget. Tolerance is 1e-9 absolute, because +both sides do the same arithmetic in the same order and anything above float noise is a real +divergence rather than a tolerance question. + ## Where to look next -The measured blocker is specific: **the model cannot rank dollars.** Row AUC 0.93, falling to -0.57 on the spans holding 87% of the stake — the whole-population dollar AUC is the table cell -flagged unverified above, and is NOT 0.70, a figure the same table's own arithmetic rules out -by a wider margin than the 0.635 it disputes. Everything downstream of that — the threshold -tuning, the four feature families — was an attempt to work around it and none succeeded, which -is what makes it the blocker rather than one finding among several. +The blocker this page named — **the model cannot rank dollars** — is still the blocker, and it is +now a number rather than an inference: holdout dollar-AUC **0.7837** against row-AUC 0.9065. The +gap is the finding. Row AUC says the model knows which conversations come back; dollar AUC says it +knows much less about which *dollars* do, and the dollars are concentrated hard enough that this is +the difference that matters. The reference ensemble reaches 0.8705 by dollar, so roughly half the +remaining gap is the estimator and half is the data. + +What has changed is that this no longer blocks *shipping the arm*. The measurement above says the +arm beats every reachable alternative on money on this window even with a dollar-AUC of 0.78 — +because the induction only needs the probability to land on the right side of 8.70%, which is a +weaker requirement than ranking. The blocker is now about how much MORE there is, not about +whether there is any: the arm reaches 25.6% of the `optimal` ceiling, and the other 74% is what +better dollar discrimination would buy. Two things follow, in order: @@ -352,3 +530,14 @@ Two things follow, in order: What is **not** worth doing is adding more columns and measuring them on 18 days. That returns sign-flipping answers indefinitely, and this work produced four of them. + +Two more, specific to the shipped model: + +3. **Re-test `--weight none` on a longer window.** The calibration argument above is sound and + lost by 0.22 pp on 18 days, which is within shouting distance of this corpus's ±0.42 pp seed + noise. It is not settled, it is merely decided for now. +4. **The cap interacts with the budget and nobody has looked at why.** `keepalive-budget` scores + *better* at `MaxPings=5` (2.065%) than at 8 (2.059%). The difference is small and may be noise, + but the arm asking for more pings than it should on a thin tail is exactly the shape the + backward induction is supposed to prevent, so a cap that improves on it is worth a look rather + than a shrug. diff --git a/kvcache/keepalive_test.go b/kvcache/keepalive_test.go index f9b20952c..d883f2d9f 100644 --- a/kvcache/keepalive_test.go +++ b/kvcache/keepalive_test.go @@ -449,6 +449,7 @@ func TestUnbudgetedArmsAreUnaffectedByTheSeam(t *testing.T) { reqs, cfg := dataset(t) cfg.MaxPings = 6 covered := 0 + budgeters := map[string]bool{} for _, spec := range Registry() { s, err := NewStrategy(spec.Name, reqs, cfg) if err != nil { @@ -458,9 +459,7 @@ func TestUnbudgetedArmsAreUnaffectedByTheSeam(t *testing.T) { } name := spec.Name if _, isBudgeter := s.(PingBudgeter); isBudgeter { - t.Errorf("%s implements PingBudgeter: the registered arms are meant to stay on the "+ - "Config.MaxPings path, and one that opted in silently would change behaviour", - name) + budgeters[name] = true } before := Simulate(reqs, s, cfg) Simulate(reqs, budgetedStub{label: "interloper", budget: 1, ok: true}, cfg) @@ -476,6 +475,28 @@ func TestUnbudgetedArmsAreUnaffectedByTheSeam(t *testing.T) { before.TotalUSD, before.Pings, after.TotalUSD, after.Pings) } } + // The set of registered arms on the budget path is an ALLOW-LIST, not "none". + // + // It was "none" until keepalive-budget joined the registry, and the message then read "one + // that opted in silently would change behaviour" — which is still the failure being + // guarded, because SILENTLY is the load-bearing word. Comparing against a named set keeps + // that and adds the other direction: an arm dropping off this list is also a behaviour + // change nobody asked for, and an assertion of "none" could never have caught it. + want := map[string]bool{StrategyKeepAliveBudget: true} + for name := range budgeters { + if !want[name] { + t.Errorf("%s implements PingBudgeter but is not on the allow-list: the other "+ + "registered arms are meant to stay on the Config.MaxPings path, and one that "+ + "opted in silently would change behaviour", name) + } + } + for name := range want { + if !budgeters[name] { + t.Errorf("%s is on the PingBudgeter allow-list but does not implement it; the arm "+ + "still resolves, so it now silently runs on Config.MaxPings instead of on its "+ + "own per-conversation budget", name) + } + } // A registry that stopped resolving would make the loop above vacuous, so say how much of // it was actually replayed. One arm (replay) legitimately has no name-based constructor. if covered < len(Registry())-1 { diff --git a/kvcache/registry.go b/kvcache/registry.go index be25d18c9..1373392fe 100644 --- a/kvcache/registry.go +++ b/kvcache/registry.go @@ -100,6 +100,8 @@ var registry = []StrategySpec{ {Name: StrategyStopReasonGated, Description: StopReasonGated{}.Describe()}, {Name: StrategyStickySession1h, Description: NewStickySession1h().Describe(), PartialReplayUnsafe: true}, + {Name: StrategyKeepAliveBudget, Description: NewKeepAliveBudget(Config{}).Describe() + + " Reads the compiled-in " + ReuseModelV1.Describe() + "."}, {Name: StrategyReplay, Description: "Replay an explicit action supplied per request. The " + "seam a policy decided elsewhere — an offline predictor, a hand-written experiment — " + "is scored through.", NeedsDataset: true}, @@ -151,6 +153,8 @@ func NewStrategy(name string, reqs []*Request, cfg Config) (Strategy, error) { return StopReasonGated{}, nil case StrategyStickySession1h: return NewStickySession1h(), nil + case StrategyKeepAliveBudget: + return NewKeepAliveBudget(cfg), nil case StrategyOptimal: return NewOptimal(reqs, cfg), nil case StrategyReplay: @@ -281,6 +285,42 @@ func (StopReasonGated) Describe() string { "revisits, the deliberate decision already in pingable() to keep pinging on end_turn." } +// ── the learned budget arm ───────────────────────────────────────────────── + +// NewKeepAliveBudget builds the learned keep-alive arm from the model compiled into this +// binary. +// +// This constructor is why `keepalive-budget` is IN the registry, and the reason is worth +// stating because the arm's own doc comment says the opposite. BudgetPolicy was excluded on +// the grounds that it "cannot be built from a name alone — it needs a Predictor", and while +// the only Predictor was an injected one that was exactly right: a name in the registry has +// to resolve, and one that resolved to an arm with a nil Predictor would resolve to an arm +// that never pings — a silently different policy wearing the right label. +// +// ReuseModelV1 removes the premise rather than the promise. The model is a fixed table of +// coefficients in this binary, so the name resolves to the same arm on every process, in +// every replay, without anything being handed in. Nothing else about the exclusion argument +// changes: a caller with its OWN predictor still constructs BudgetPolicy directly and hands +// it to Simulate, exactly as before, and gets its own label. +// +// MinPrefix is deliberately left at 0 — no small-prefix gate. The arithmetic does not need +// one: PingBudget's ping cost is Pricing.KeepAliveCost, which already carries the fixed +// ping_input/ping_output terms that do not scale with the prefix, so the break-even it +// compares against is the true one at 500 tokens as much as at 125k. MinPrefix is an operator +// dial for a deployment that wants a floor anyway, not a correction for a bias, and choosing +// a number here would be inventing one. +func NewKeepAliveBudget(cfg Config) BudgetPolicy { + return BudgetPolicy{ + Predictor: ReuseModelV1, + // Simulate overrides Interval to Config.PingIdle regardless — see its doc comment — + // so this matters only for a direct PingBudget/Windows call. Seeded from the config + // rather than from the model so the two agree wherever the caller has an opinion. + Interval: cfg.PingIdle, + MaxK: ReuseModelV1.MaxK, + Semantics: cfg.Semantics, + } +} + // ── the explicit-action seam ─────────────────────────────────────────────── // Replay scores an action list decided somewhere else. diff --git a/kvcache/reusemodel.go b/kvcache/reusemodel.go new file mode 100644 index 000000000..95c0c63a1 --- /dev/null +++ b/kvcache/reusemodel.go @@ -0,0 +1,551 @@ +package kvcache + +import ( + "fmt" + "math" + "sort" + "strconv" + "strings" + "time" +) + +// The FIXED reuse model: a frozen survival fit, compiled in, answering Predictor. +// +// # Why this file exists +// +// `Predictor` has been the seam a learned model plugs into since the simulator was written, +// and until now nothing implemented it outside a test. That had a consequence bigger than a +// missing feature: `BudgetPolicy` — the arm that turns reuse probabilities into a +// per-conversation keep-alive budget — was kept out of `Registry()` because "it cannot be +// built from a name alone: it needs a Predictor", so no dashboard page could reach it and no +// operator could ask what it would have done. The registry's promise is that every name in it +// resolves; an arm needing an injected dependency could not keep it. +// +// A model compiled in as constants resolves from a name. That is the whole idea here, and it +// is also why the model is a dot product rather than the gradient-boosted ensemble the +// reference fit uses (deploy/harbor/kv_ttl_keepalive_policy.py's `fit()`): +// +// - It has to be REVIEWABLE. Thirty-odd coefficients can be read in a diff and argued with. +// Thirty-seven trees of thresholds cannot, so nobody could tell a re-fit that improved the +// model from one that broke it. +// - It has to be FIXED. Re-fitting at startup would make yesterday's dashboard figure +// unreproducible, and the KV-cache page's whole claim is that replaying a historical +// window gives the same answer every time. +// - It has to be cheap enough for the request path, because the live pinger consults the +// same model. proxy/keepalivestrategy.go states that rule for itself: "a rule, or ... a +// portable logistic-regression dot product — never an embedded model or a call out of the +// hot path." +// +// What going linear costs against the ensemble is measured on every re-fit and printed beside +// the coefficients in reusemodel_v1_gen.go, rather than asserted once here. +// +// # What it answers, and why that is not the obvious thing +// +// The model is fitted on CONDITIONAL survival over the keep-alive sweep grid — +// `s_k = P(still idle at sweep k+1 | still idle at sweep k)` — not on "will this conversation +// come back within five minutes". That is because of the population: a keep-alive fires only +// after a conversation has already gone quiet for one interval, so the decision is never faced +// by the ~92.5% of gaps that close inside five minutes. Elapsed silence is not a feature here, +// it is the sample definition. `ReuseProbability` composes those conditionals forwards into the +// cumulative answer the interface promises. +// +// # Where the numbers live +// +// The coefficients are in reusemodel_v1_gen.go, generated. This file is the arithmetic that +// reads them, and it is hand-written and reviewed. The two are pinned to the Python that +// produced them by TestReuseModelV1AgreesWithThePort, which drives +// `kv_ttl_keepalive_policy.py --score` over the checked-in reusemodel_v1.json and compares the +// per-sweep survival vector, the CDF at horizons on and off the knot grid, and the budget that +// results — all three, because a budget that agrees for the wrong reason is a guard that has +// already stopped working. + +// ReuseTerm is one numeric feature's coefficient and the standardisation the fit applied. +// +// Center/Scale are on the wire because the coefficient is meaningless without them: the fit +// standardised every numeric column, so Go has to apply the identical transform or a +// coefficient of -1.0 means something different on each side. A Scale of 0 is read as 1, +// which is the same guard the fit applies — see the zero-variance note in the generated file. +type ReuseTerm struct { + Name string + Coef float64 + Center float64 + Scale float64 +} + +// ReuseLevel is one one-hot level's coefficient. The empty Level is the CATCH-ALL: a level the +// fit never saw scores as that rather than as zero, because zero is not "unknown" — it is +// "exactly the reference level", which is a different and much stronger claim. +type ReuseLevel struct { + Level string + Coef float64 +} + +// ReuseCategory is one one-hot feature and its levels. +type ReuseCategory struct { + Name string + Levels []ReuseLevel +} + +// ReuseModel is a frozen discrete-time survival model over the keep-alive sweep grid. +// +// Immutable after construction and safe for concurrent use: every method takes the model by +// pointer but writes nothing through it, which matters because one *ReuseModel is shared by +// every dashboard request and every live ping sweep. +type ReuseModel struct { + // Version and TrainedOn are provenance, and they are on the type rather than only in a + // comment so that a result can carry them: a saving attributed to "the model" is not + // checkable unless the reader can tell WHICH model produced it. + Version string + TrainedOn string + + // IntervalS, LifeS and MaxK are the schedule the model was FITTED on. The grid the + // conditionals live on is a property of the fit, not of the caller — see grid(). + IntervalS float64 + LifeS float64 + MaxK int + + Intercept float64 + Numeric []ReuseTerm + Categorical []ReuseCategory +} + +// The shipped feature names. Strings rather than struct fields because they are a wire +// contract with the Python: the generated table names each term, and a term this code cannot +// evaluate is a fitted coefficient silently applied to zero. Refused loudly instead — see +// Validate. +const ( + featLogPrefix = "log_prefix" + featLogPrevGap = "log_prev_gap" + featTurn = "turn" + featHourSin = "hour_sin" + featHourCos = "hour_cos" + featDowSin = "dow_sin" + featDowCos = "dow_cos" + featStatP = "stat_p" + featLogStatN = "log_stat_n" + + featCategoricalModel = "model" + // featCategoricalSweep is the BASELINE HAZARD, one coefficient per sweep index, and its + // level is the index as a decimal string. + // + // It is a one-hot rather than a numeric `sweep_k` term, and that is the one place the + // shipped model's specification departs from the reference ensemble's. The reference reads + // `sweep_k`, `log_elapsed` and `log_secs_to_deadline` — all three exact functions of the + // sweep index, and the third a constant at any schedule with interval < life. A tree + // ensemble treats that as redundancy. A linear model cannot: it is exact collinearity, the + // coefficients are unidentified, and the first fit here produced -4.77 and +4.09 on two + // features that move together, summing to a hazard of ~14% at every sweep where the corpus + // measures 4.86% at the first and 0.24% by the seventh. The arm bought its maximum budget + // on 97% of decisions and saved less than a flat MaxPings=2. + // + // A free coefficient per sweep is strictly more general than the linear term, and the shape + // it recovers is not monotone — which is exactly why assuming monotone was wrong. + featCategoricalSweep = "sweep" +) + +// hourPeriod and dowPeriod are the cycle lengths the sin/cos encoding uses. +// +// Hour and day-of-week are encoded as a sin/cos PAIR rather than as the raw integers the +// reference ensemble reads, and the reason is that this model is linear: a linear term on raw +// hour reads 23:00 and 00:00 as 23 units apart, which is not a fact about time. The ensemble +// did not care because it splits rather than multiplies. The pair costs one extra coefficient +// each; the measured effect on this corpus is inside the seed-noise floor either way. +const ( + hourPeriod = 24.0 + dowPeriod = 7.0 +) + +// reuseSpan is the half of the feature vector that does NOT move with the sweep index. +// +// Hoisted out of the per-sweep loop deliberately: composing the CDF walks the whole grid, and +// `Windows` asks for four horizons per sweep, so the span block would otherwise be recomputed +// tens of times per decision — including the Stats lookup, which is the only part that touches +// a map. +type reuseSpan struct { + nowMs int64 + logPrefix float64 + logPrevGap float64 + turn float64 + statP float64 + logStatN float64 + model string +} + +// spanOf reads the span-level features off an Observation. +// +// Every line here has a counterpart in the Python's `load_spans`, and the pairs have to agree +// or the model is served features it was not fitted on: +// +// log_prefix log1p(CachedTokens) = log1p(cache_read + cache_write) offline +// log_prev_gap log1p(SinceLastMs/1000) = log1p(the gap that already closed), 0 at turn 1 +// turn Turn = requests served so far, including this one +// stat_p/n Stats.ReuseWithin(5m) = the same leak-free accumulator, same 6-gap floor +// model lower(Model) = case-folded, because the corpus spells one +// upstream two ways and two levels for one model +// split its evidence in half +// +// A nil Stats yields (0, 0), which is exactly what the accumulator returns early in a replay +// before any gap has closed — so it is a value the fit saw, not a missing one. +func (m *ReuseModel) spanOf(o Observation) reuseSpan { + var p float64 + var n int + if o.Stats != nil { + p, n, _ = o.Stats.ReuseWithin(o.User, o.Model, o.Bucket, Horizon5m) + } + return reuseSpan{ + nowMs: o.Now, + logPrefix: math.Log1p(float64(o.CachedTokens)), + logPrevGap: math.Log1p(float64(o.SinceLastMs) / 1000.0), + turn: float64(o.Turn), + statP: p, + logStatN: math.Log1p(float64(n)), + model: strings.ToLower(o.Model), + } +} + +// reuseState is the half of the feature vector that DOES move with the sweep index. +type reuseState struct { + sweep int + hourSin float64 + hourCos float64 + dowSin float64 + dowCos float64 +} + +// stateAt is the span's state as seen at sweep k, mirroring the Python's `state_features`. +// +// The calendar fields are the hour and weekday AT THE SWEEP INSTANT, not at the request: +// a conversation that goes quiet at 23:58 is decided about at 00:03, and Observation.HourUTC +// is the wrong one of those two. That is why this recomputes them from Now rather than reading +// the field the Observation already carries. +func (m *ReuseModel) stateAt(sp reuseSpan, k int) reuseState { + tk := float64(k) * m.interval() + // Epoch seconds at the sweep. Floored rather than truncated-toward-zero: the corpus is + // post-1970 so the two agree, but the Python uses `//` and this has to be the same + // function, not the same answer on the data that happens to be in front of it. + secs := float64(sp.nowMs)/1000.0 + tk + hour := math.Mod(math.Floor(secs/3600.0), 24.0) + dow := math.Mod(math.Floor(secs/86400.0)+4.0, 7.0) + return reuseState{ + sweep: k, + hourSin: math.Sin(2.0 * math.Pi * hour / hourPeriod), + hourCos: math.Cos(2.0 * math.Pi * hour / hourPeriod), + dowSin: math.Sin(2.0 * math.Pi * dow / dowPeriod), + dowCos: math.Cos(2.0 * math.Pi * dow / dowPeriod), + } +} + +// value resolves one named feature. An unknown name yields (0, false) so Validate can refuse +// the model rather than scoring a fitted coefficient against a silent zero. +func value(sp reuseSpan, st reuseState, name string) (float64, bool) { + switch name { + case featLogPrefix: + return sp.logPrefix, true + case featLogPrevGap: + return sp.logPrevGap, true + case featTurn: + return sp.turn, true + case featStatP: + return sp.statP, true + case featLogStatN: + return sp.logStatN, true + case featHourSin: + return st.hourSin, true + case featHourCos: + return st.hourCos, true + case featDowSin: + return st.dowSin, true + case featDowCos: + return st.dowCos, true + } + return 0, false +} + +// levelOf reads the one-hot level for a categorical feature. +func levelOf(sp reuseSpan, st reuseState, name string) (string, bool) { + switch name { + case featCategoricalModel: + return sp.model, true + case featCategoricalSweep: + return strconv.Itoa(st.sweep), true + } + return "", false +} + +// catMovesWithSweep says whether a categorical feature's LEVEL depends on the sweep index, so +// that only the ones that do are re-resolved per sweep. The numeric counterpart is +// movesWithSweep; both are switches rather than sets so that adding a feature without +// classifying it fails to compile rather than being silently evaluated at the wrong sweep. +func catMovesWithSweep(name string) bool { return name == featCategoricalSweep } + +// coefFor is a level's coefficient, falling back to the catch-all. +// +// A linear scan, not a map: there are a dozen levels, the slice is contiguous, and a map here +// would allocate at construction for no measurable gain on a lookup this small. +func (c ReuseCategory) coefFor(level string) float64 { + fallback := 0.0 + for _, l := range c.Levels { + if l.Level == level { + return l.Coef + } + if l.Level == "" { + fallback = l.Coef + } + } + return fallback +} + +// Validate refuses a model this code cannot evaluate exactly. +// +// It exists because the failure it prevents is silent. A generated table naming a feature this +// build does not implement would otherwise score that coefficient against zero, which is not +// an error and not a crash — it is a slightly different model, producing slightly different +// budgets, with nothing to notice. TestReuseModelV1IsValid calls this on the shipped table, so +// a re-fit that adds a feature fails the build's tests rather than the deployment. +func (m *ReuseModel) Validate() error { + if m == nil { + return fmt.Errorf("kvcache: nil reuse model") + } + if len(m.Numeric) == 0 && len(m.Categorical) == 0 { + return fmt.Errorf("kvcache: reuse model %q has no terms", m.Version) + } + var sp reuseSpan + st := m.stateAt(sp, 1) + for _, t := range m.Numeric { + if _, ok := value(sp, st, t.Name); !ok { + return fmt.Errorf("kvcache: reuse model %q has unknown numeric feature %q; this "+ + "build cannot score it, and scoring it as zero would be a different model", + m.Version, t.Name) + } + } + for _, c := range m.Categorical { + if _, ok := levelOf(sp, st, c.Name); !ok { + return fmt.Errorf("kvcache: reuse model %q has unknown categorical feature %q; "+ + "this build cannot score it, and scoring it as zero would be a different model", + m.Version, c.Name) + } + } + if m.IntervalS <= 0 || m.LifeS <= 0 { + return fmt.Errorf("kvcache: reuse model %q has a non-positive schedule "+ + "(interval %gs, life %gs)", m.Version, m.IntervalS, m.LifeS) + } + return nil +} + +func (m *ReuseModel) interval() float64 { + if m.IntervalS > 0 { + return m.IntervalS + } + return float64(DefaultPingIdle / time.Second) +} + +func (m *ReuseModel) life() float64 { + if m.LifeS > 0 { + return m.LifeS + } + return float64(TTL5m.Lifetime() / time.Second) +} + +func (m *ReuseModel) maxK() int { + if m.MaxK > 0 { + return m.MaxK + } + return DefaultBudgetMaxK +} + +// survival is `s_k`: the dot product, standardised, through the logistic. +// +// zSpan is passed in rather than recomputed because the span block is the same at every sweep +// and this is the innermost loop of the whole arm. +func (m *ReuseModel) survival(sp reuseSpan, zSpan float64, k int) float64 { + st := m.stateAt(sp, k) + z := zSpan + for _, t := range m.Numeric { + if !movesWithSweep(t.Name) { + continue + } + v, ok := value(sp, st, t.Name) + if !ok { + continue + } + z += t.Coef * (v - t.Center) / scaleOf(t) + } + for _, c := range m.Categorical { + if !catMovesWithSweep(c.Name) { + continue + } + level, ok := levelOf(sp, st, c.Name) + if !ok { + continue + } + z += c.coefFor(level) + } + return sigmoid(z) +} + +// zSpanOf is the intercept plus every term that does not move with the sweep index. +func (m *ReuseModel) zSpanOf(sp reuseSpan) float64 { + z := m.Intercept + var st reuseState // unused by the span-level features, by construction + for _, t := range m.Numeric { + if movesWithSweep(t.Name) { + continue + } + v, ok := value(sp, st, t.Name) + if !ok { + continue + } + z += t.Coef * (v - t.Center) / scaleOf(t) + } + for _, c := range m.Categorical { + if catMovesWithSweep(c.Name) { + continue + } + level, ok := levelOf(sp, st, c.Name) + if !ok { + continue + } + z += c.coefFor(level) + } + return z +} + +// movesWithSweep says whether a feature is in the state block. A switch rather than a set +// lookup, so adding a feature to value() without classifying it here fails to compile the +// same day rather than scoring at the wrong sweep. +func movesWithSweep(name string) bool { + switch name { + case featHourSin, featHourCos, featDowSin, featDowCos: + return true + } + return false +} + +func scaleOf(t ReuseTerm) float64 { + if t.Scale == 0 { + return 1.0 + } + return t.Scale +} + +// sigmoid is 1/(1+e^-z), arranged so a large-magnitude z cannot overflow in either direction. +func sigmoid(z float64) float64 { + if z >= 0 { + return 1.0 / (1.0 + math.Exp(-z)) + } + e := math.Exp(z) + return e / (1.0 + e) +} + +// ReuseProbability implements Predictor: P(this conversation has returned within horizon). +// +// CUMULATIVE, as the interface promises, composed forwards from the fitted conditionals: +// +// F(t_(j+1)) = 1 - prod_(i<=j) s_i +// +// on the grid {t_j} u {t_j + life}, linearly interpolated between knots and CLAMPED outside +// it. Two properties of that are worth stating because BudgetPolicy depends on both: +// +// - F(t_1) = 0 by construction. This CDF is already conditioned on having reached the first +// sweep, which is exactly the population `Windows` divides by — so the conditioning is not +// applied twice. +// - the grid is the FIT's, not the caller's. A caller replaying on a different interval gets +// the model's own knots interpolated to its horizons rather than a silent re-fit, which is +// the honest failure mode: the numbers degrade visibly instead of the model pretending to +// have been trained on a schedule it was not. +// +// ok is false only when there is no model to consult. Everything else a decision needs to be +// refused — unknown rates, no prefix, a schedule the arm cannot price — is BudgetPolicy's own +// business and is already checked there; duplicating it here would put the same policy in two +// places and let them disagree. +func (m *ReuseModel) ReuseProbability(o Observation, horizon time.Duration) (float64, bool) { + if m == nil || (len(m.Numeric) == 0 && len(m.Categorical) == 0) { + return 0, false + } + seconds := horizon.Seconds() + sp := m.spanOf(o) + zSpan := m.zSpanOf(sp) + iv, life, mk := m.interval(), m.life(), m.maxK() + + // The knot grid, built exactly as the Python builds it: a map keyed by the knot, so that a + // schedule where t_(j+1) coincides with t_j+life resolves the collision the same way on + // both sides. Sorted afterwards, because {t_j} and {t_j + life} interleave whenever + // life > interval — which is the shipped case (300 > 280). + cum := make(map[float64]float64, 2*mk+3) + cum[0] = 0 + surv := 1.0 + for j := 1; j <= mk+1; j++ { + tj := float64(j) * iv + cum[tj] = 1.0 - surv + surv *= m.survival(sp, zSpan, j) + cum[tj+life] = 1.0 - surv + // Early exit once the grid brackets the horizon. Every later knot is larger, so it + // could only ever be selected by the upper clamp — which cannot fire now that a knot + // above the horizon exists. Same answer, up to 8x less work per call, and `Windows` + // makes four calls per sweep. + if tj > seconds { + break + } + } + xs := make([]float64, 0, len(cum)) + for at := range cum { + xs = append(xs, at) + } + sort.Float64s(xs) + ys := make([]float64, len(xs)) + for i, at := range xs { + ys[i] = cum[at] + } + p := interp(seconds, xs, ys) + // Clamped, not trusted. A non-monotone composition cannot arise from a product of + // probabilities, but the interpolation is floating point and BudgetPolicy divides by + // 1-F: a p of 1+1e-16 there is a negative survivor share and a budget with no meaning. + if p < 0 { + p = 0 + } + if p > 1 { + p = 1 + } + return p, true +} + +// interp is numpy.interp for a sorted grid: linear inside, CLAMPED to the endpoints outside. +// +// The clamping is load-bearing rather than defensive. A horizon past the last knot must read +// as the last cumulative value, not extrapolate past 1.0 — `Windows` asks about +// t_maxk + life, which is the last knot, and about t_(maxk+1), which is not. The Python's +// `_interp` is the same function, and the drift test compares them on horizons deliberately +// placed outside the grid. +func interp(x float64, xs, ys []float64) float64 { + if len(xs) == 0 { + return 0 + } + if x <= xs[0] { + return ys[0] + } + if x >= xs[len(xs)-1] { + return ys[len(ys)-1] + } + lo, hi := 0, len(xs)-1 + for hi-lo > 1 { + mid := (lo + hi) / 2 + if xs[mid] <= x { + lo = mid + } else { + hi = mid + } + } + span := xs[hi] - xs[lo] + if span <= 0 { + return ys[hi] + } + return ys[lo] + (ys[hi]-ys[lo])*(x-xs[lo])/span +} + +// Describe is the one-sentence provenance a result carries beside the arm's numbers. +func (m *ReuseModel) Describe() string { + if m == nil { + return "no reuse model" + } + return fmt.Sprintf("reuse model %s, fitted on %s", m.Version, m.TrainedOn) +} diff --git a/kvcache/reusemodel_test.go b/kvcache/reusemodel_test.go new file mode 100644 index 000000000..b15a95552 --- /dev/null +++ b/kvcache/reusemodel_test.go @@ -0,0 +1,240 @@ +package kvcache + +// Each test here asserts one PROPERTY the arm depends on, not one code path. +// +// The Python-parity guard lives in deploy/harbor/kv_ttl_reusemodel_drift_test.go and pins the +// numbers. These pin the shape: that the CDF is a distribution, that a horizon outside the knot +// grid clamps rather than extrapolates, that an unseen level falls to the catch-all rather than to +// the reference level, that the standardisation is actually applied, and that a model this build +// cannot score is refused instead of quietly scored against zeros. + +import ( + "math" + "testing" + "time" +) + +// tinyModel is a hand-written model with values chosen so the expected answers can be computed by +// hand rather than read off a previous run of the code under test. +func tinyModel() *ReuseModel { + return &ReuseModel{ + Version: "test", TrainedOn: "hand-written", + IntervalS: 280, LifeS: 300, MaxK: 3, + Intercept: 0, + Numeric: []ReuseTerm{ + // (log_prefix - 10) / 2, so a prefix whose log1p is 10 contributes exactly 0 and one + // at 12 contributes exactly the coefficient. + {Name: featLogPrefix, Coef: 1, Center: 10, Scale: 2}, + }, + Categorical: []ReuseCategory{ + {Name: featCategoricalSweep, Levels: []ReuseLevel{ + {Level: "1", Coef: 2}, {Level: "2", Coef: 2}, {Level: "3", Coef: 2}, + {Level: "4", Coef: 2}, {Level: "", Coef: 2}, + }}, + {Name: featCategoricalModel, Levels: []ReuseLevel{ + {Level: "known", Coef: 1}, {Level: "", Coef: -3}, + }}, + }, + } +} + +func obs(prefix int64, model string) Observation { + return Observation{ + User: "u", Model: model, Now: 1_700_000_000_000, CachedTokens: prefix, + Turn: 1, TTL: TTL5m, Bucket: BucketAt(1_700_000_000_000), + } +} + +func TestReuseModelCDFIsADistribution(t *testing.T) { + m := tinyModel() + o := obs(60_000, "known") + prev := -1.0 + // Every horizon from before the first knot to well past the last one. + for sec := 0; sec <= 3000; sec += 10 { + p, ok := m.ReuseProbability(o, time.Duration(sec)*time.Second) + if !ok { + t.Fatalf("declined at %ds", sec) + } + if p < 0 || p > 1 { + t.Fatalf("F(%ds) = %g is not a probability", sec, p) + } + if p < prev-1e-12 { + t.Fatalf("F(%ds) = %g fell below F(%ds) = %g; a cumulative distribution cannot "+ + "decrease, and BudgetPolicy divides by 1-F so a decrease is a negative "+ + "survivor share and a budget with no meaning", sec, p, sec-10, prev) + } + prev = p + } +} + +func TestReuseModelIsAlreadyConditionedOnReachingTheFirstSweep(t *testing.T) { + // F(t_1) = 0 by construction: the CDF starts at the first sweep, because that is the + // population the decision is faced by. Windows divides by 1-F(t_1), so a nonzero value here + // would condition on the same thing twice. + m := tinyModel() + p, ok := m.ReuseProbability(obs(60_000, "known"), time.Duration(m.IntervalS)*time.Second) + if !ok { + t.Fatal("declined at the first sweep") + } + if p != 0 { + t.Errorf("F(t_1) = %g, want exactly 0", p) + } +} + +func TestReuseModelClampsPastTheGridRatherThanExtrapolating(t *testing.T) { + m := tinyModel() + o := obs(60_000, "known") + last := float64(m.MaxK+1)*m.IntervalS + m.LifeS + end, _ := m.ReuseProbability(o, time.Duration(last)*time.Second) + for _, sec := range []float64{last, last + 1, last * 10, 86_400} { + p, ok := m.ReuseProbability(o, time.Duration(sec)*time.Second) + if !ok { + t.Fatalf("declined at %gs", sec) + } + if math.Abs(p-end) > 1e-12 { + t.Errorf("F(%gs) = %g but the last knot is %g; extrapolating past the grid is how a "+ + "probability climbs past 1", sec, p, end) + } + } +} + +func TestReuseModelAnUnseenLevelFallsToTheCatchAllNotToZero(t *testing.T) { + // The distinction is the whole reason ReuseLevel has an empty-string entry. Zero is not + // "unknown", it is "exactly the reference level" — a much stronger claim about a model id the + // fit never saw. tinyModel gives the catch-all -3 against the known level's +1, so the two + // are far apart and a fallback to 0 would sit between them and be caught either way. + m := tinyModel() + known, _ := m.ReuseProbability(obs(60_000, "known"), 560*time.Second) + unseen, _ := m.ReuseProbability(obs(60_000, "never-trained-on"), 560*time.Second) + zeroish := &ReuseModel{Version: "z", TrainedOn: "x", IntervalS: 280, LifeS: 300, MaxK: 3, + Numeric: m.Numeric, Categorical: []ReuseCategory{m.Categorical[0], + {Name: featCategoricalModel, Levels: []ReuseLevel{{Level: "known", Coef: 1}}}}} + fellToZero, _ := zeroish.ReuseProbability(obs(60_000, "never-trained-on"), 560*time.Second) + if math.Abs(unseen-known) < 1e-9 { + t.Error("an unseen model id scored identically to a trained one; the level is not being read") + } + if math.Abs(unseen-fellToZero) < 1e-9 { + t.Error("the catch-all coefficient had no effect: an unseen level is being scored as 0, " + + "which claims it behaves exactly like the reference level") + } +} + +func TestReuseModelCaseFoldsTheModelLevel(t *testing.T) { + // The corpus carries `Azure/gpt-4o` beside `azure/gpt-5.5`, because the id is whatever the + // client sent. Two levels for one model halve its evidence, so both sides fold. + m := tinyModel() + lower, _ := m.ReuseProbability(obs(60_000, "known"), 560*time.Second) + upper, _ := m.ReuseProbability(obs(60_000, "KNOWN"), 560*time.Second) + if math.Abs(lower-upper) > 1e-12 { + t.Errorf("%q scored %g and %q scored %g; the level is not case-folded", + "known", lower, "KNOWN", upper) + } +} + +func TestReuseModelAppliesTheStandardisation(t *testing.T) { + // A coefficient means nothing without the Center/Scale the fit used. Same coefficient, same + // features, different standardisation must give a different answer — otherwise Go is ignoring + // two thirds of every ReuseTerm and the Python's coefficients are being applied to raw + // values. + a := tinyModel() + b := tinyModel() + b.Numeric[0].Scale = 20 // ten times wider, so the same feature contributes a tenth as much + pa, _ := a.ReuseProbability(obs(1_000_000, "known"), 560*time.Second) + pb, _ := b.ReuseProbability(obs(1_000_000, "known"), 560*time.Second) + if math.Abs(pa-pb) < 1e-6 { + t.Errorf("Scale had no effect (%g vs %g); the standardisation is not being applied", pa, pb) + } +} + +func TestReuseModelHasNoOpinionWithoutATable(t *testing.T) { + // ok=false is reserved for "there is no model". Everything else a decision needs refused — + // unknown rates, no prefix, an unpriceable schedule — is BudgetPolicy's own business and is + // checked there; duplicating it here would be the same policy in two places. + var nilModel *ReuseModel + if _, ok := nilModel.ReuseProbability(obs(60_000, "known"), time.Minute); ok { + t.Error("a nil model claimed an opinion") + } + empty := &ReuseModel{Version: "e", IntervalS: 280, LifeS: 300, MaxK: 3} + if _, ok := empty.ReuseProbability(obs(60_000, "known"), time.Minute); ok { + t.Error("a model with no terms claimed an opinion") + } +} + +func TestReuseModelValidateRefusesWhatThisBuildCannotScore(t *testing.T) { + // The failure this prevents is silent: an unknown feature would be scored against zero, + // which is not an error and not a crash — it is a different model producing different + // budgets with nothing to notice. + for _, tc := range []struct { + name string + m *ReuseModel + }{ + {"nil", nil}, + {"no terms", &ReuseModel{Version: "x", IntervalS: 280, LifeS: 300}}, + {"unknown numeric", &ReuseModel{Version: "x", IntervalS: 280, LifeS: 300, MaxK: 3, + Numeric: []ReuseTerm{{Name: "sweep_k", Coef: 1, Scale: 1}}}}, + {"unknown categorical", &ReuseModel{Version: "x", IntervalS: 280, LifeS: 300, MaxK: 3, + Categorical: []ReuseCategory{{Name: "cache_ttl", + Levels: []ReuseLevel{{Level: "ephemeral_5m", Coef: 1}}}}}}, + {"no schedule", &ReuseModel{Version: "x", MaxK: 3, + Numeric: []ReuseTerm{{Name: featLogPrefix, Coef: 1, Scale: 1}}}}, + } { + if err := tc.m.Validate(); err == nil { + t.Errorf("%s: Validate accepted a model this build cannot score exactly", tc.name) + } + } + if err := tinyModel().Validate(); err != nil { + t.Errorf("a well-formed model was refused: %v", err) + } +} + +func TestReuseModelTheLastSweepsSurvivalCannotChangeABudget(t *testing.T) { + // s_(MaxK+1) is composed into the grid but never read by Windows, which looks no further than + // t_MaxK + life. This is asserted rather than assumed because the Python's `sweep` one-hot has + // no fitted level for MaxK+1 — it falls to the catch-all — so if that value DID reach a + // decision, the arm's answer would depend on an unfitted coefficient. + base := tinyModel() + moved := tinyModel() + for i, l := range moved.Categorical[0].Levels { + if l.Level == "" { + moved.Categorical[0].Levels[i].Coef = -12 // a wildly different unfitted catch-all + } + } + o := obs(60_000, "known") + o.Pricing = Pricing{Known: true, Input: 3.8e-06, Output: 1.9e-05, CacheRead: 3.8e-07, + Write5m: 4.75e-06, Write1h: 7.6e-06, PingInputTokens: 1, PingOutputTokens: 1} + iv := time.Duration(base.IntervalS) * time.Second + pa := BudgetPolicy{Predictor: base, Interval: iv, MaxK: base.MaxK} + pb := BudgetPolicy{Predictor: moved, Interval: iv, MaxK: moved.MaxK} + ka, oka := pa.PingBudget(o) + kb, okb := pb.PingBudget(o) + if oka != okb || ka != kb { + t.Errorf("moving the unfitted sweep-%d coefficient changed the budget from (%d,%v) to "+ + "(%d,%v); Windows is reading past t_MaxK + life", base.MaxK+1, ka, oka, kb, okb) + } +} + +func TestReuseModelV1IsTheArmTheRegistryBuilds(t *testing.T) { + // NewKeepAliveBudget is what makes `keepalive-budget` resolvable from a name, which is the + // whole reason the arm could join the registry. If it stopped reading the compiled-in model + // the name would still resolve — to an arm with a nil Predictor, which never pings at all + // (Decide returns write_5m), so the page would show a silently different policy under the + // right label. + s, err := NewStrategy(StrategyKeepAliveBudget, nil, Config{}) + if err != nil { + t.Fatalf("the registry cannot build its own arm: %v", err) + } + b, isBudget := s.(BudgetPolicy) + if !isBudget { + t.Fatalf("%s resolved to %T, not a BudgetPolicy", StrategyKeepAliveBudget, s) + } + if b.Predictor == nil { + t.Fatal("the arm has no Predictor: it resolves, but it can never ping") + } + if b.Predictor != Predictor(ReuseModelV1) { + t.Errorf("the arm reads %T rather than the compiled-in ReuseModelV1", b.Predictor) + } + if _, isBudgeter := s.(PingBudgeter); !isBudgeter { + t.Error("the arm does not implement PingBudgeter, so Simulate will run it on the flat " + + "Config.MaxPings path instead of its own per-conversation budget") + } +} diff --git a/kvcache/reusemodel_v1_gen.go b/kvcache/reusemodel_v1_gen.go new file mode 100644 index 000000000..9a74f8e5c --- /dev/null +++ b/kvcache/reusemodel_v1_gen.go @@ -0,0 +1,92 @@ +// Code generated by deploy/harbor/kv_ttl_keepalive_policy.py --go-out. DO NOT EDIT. +// +// Regenerate with: +// +// cd deploy/harbor && python3 kv_ttl_keepalive_policy.py \ +// --db /var/lib/context-guru/cg.db --prices /etc/context-guru/prices.yaml \ +// --model-out reusemodel_v1.json --go-out ../../kvcache/reusemodel_v1_gen.go +// +// Provenance. This is the argument for trusting the coefficients, so it is generated +// from the fit rather than written by hand — including the parts that are unflattering. +// +// window (UTC) 2026-08-17 11:48 -> 2026-09-04 21:45 UTC (18.4 days) +// requests 219650 +// idle spans 219650 +// reaching sweep 1 37418 <- the population the decision is actually faced on +// person-periods 276321 +// schedule interval 280s / life 300s / max_k 8 +// survived to sweep+1 98.53% of person-periods +// +// fit weight stake +// +// AUC by row AUC by dollar logloss mean p / actual +// holdout (late 30%) 0.9065 0.7837 0.0345 0.9868 / 0.9919 +// in-sample 0.8855 0.7597 0.0573 0.9826 / 0.9853 +// in-sample, ref GBM 0.9572 0.8705 0.0503 0.9776 / 0.9853 +// +// The HOLDOUT row is the one to argue with: it is fitted on the early 70% of the +// window and scored on the late 30%, split by time rather than at random because one +// span contributes up to max_k person-periods and a random split scores a row against +// its own siblings. The in-sample rows are here only to show the gap; the reference +// GBM's in-sample figures in particular are an upper bound on nothing. +// +// Neither AUC is a saving. What this arm is worth in money is the KV-cache page's own +// replay of a window under it, against the same baseline as every other arm — which is +// what shipping it as a registry arm is for. docs/how-to/kv-cache-keepalive-budget.md +// carries the measured table and says plainly where a flat MaxPings still beats it. + +package kvcache + +// ReuseModelV1 is the fitted, frozen reuse model the `keepalive-budget` arm reads. +// +// See reusemodel.go for what the fields mean and for the CDF this composes. The +// numbers are a logistic survival fit on the corpus named above; the Python that +// produced them, the feature arithmetic they assume, and this table are pinned +// together by TestReuseModelV1AgreesWithThePort. +var ReuseModelV1 = &ReuseModel{ + Version: "v1", + TrainedOn: "2026-08-17 11:48 -> 2026-09-04 21:45 UTC", + IntervalS: 280.0, + LifeS: 300.0, + MaxK: 8, + Intercept: 5.018342744954828, + Numeric: []ReuseTerm{ + {Name: "log_prefix", Coef: -2.0617767376699336, Center: 4.146090828434305, Scale: 4.7037498066187835}, + {Name: "log_prev_gap", Coef: -0.051073693614662456, Center: 0.8069032678784968, Scale: 1.7739566028829084}, + {Name: "turn", Coef: 0.17222201868549095, Center: 93.21953452687274, Scale: 539.4799156608967}, + {Name: "hour_sin", Coef: -0.11807319204324197, Center: -0.03876981083782035, Scale: 0.698196684692585}, + {Name: "hour_cos", Coef: 0.05351815398273302, Center: -0.12480005679454068, Scale: 0.7038772883647778}, + {Name: "dow_sin", Coef: -0.044776344705750655, Center: 0.5289808564926407, Scale: 0.5991648136759256}, + {Name: "dow_cos", Coef: 0.06955802004724103, Center: -0.27736442403982214, Scale: 0.5331507814789217}, + {Name: "stat_p", Coef: 0.3068298285865256, Center: 0.8669557007725781, Scale: 0.13350760774270665}, + {Name: "log_stat_n", Coef: -0.028663327452458465, Center: 5.707126141180795, Scale: 1.7859544188184382}, + }, + Categorical: []ReuseCategory{ + {Name: "model", Levels: []ReuseLevel{ + {Level: "", Coef: 0.044073416904029585}, + {Level: "aws/claude-haiku-4-5", Coef: 0.01059219829200029}, + {Level: "aws/claude-opus-5", Coef: 0.5891045430667132}, + {Level: "aws/claude-sonnet-5", Coef: 0.4672028093860629}, + {Level: "azure/gpt-4o", Coef: 0.04754489687167924}, + {Level: "azure/gpt-5.5", Coef: 0.43532292998608124}, + {Level: "claude-haiku-4-5", Coef: 0.05693310295387887}, + {Level: "claude-opus-4-6", Coef: 0.8914374258675521}, + {Level: "claude-opus-4-7", Coef: 0.20280481953179896}, + {Level: "claude-opus-4-8", Coef: 0.5671313782762598}, + {Level: "claude-opus-5", Coef: 0.759769916393038}, + {Level: "claude-sonnet-4-6", Coef: 0.6687443839651331}, + {Level: "claude-sonnet-5", Coef: 0.24100295418844414}, + }}, + {Name: "sweep", Levels: []ReuseLevel{ + {Level: "", Coef: 0.0}, + {Level: "1", Coef: -0.8103139873353238}, + {Level: "2", Coef: -0.04573723713715996}, + {Level: "3", Coef: 0.6255276686681279}, + {Level: "4", Coef: 0.8279466337110583}, + {Level: "5", Coef: 0.35705180846007445}, + {Level: "6", Coef: 0.6740656989771248}, + {Level: "7", Coef: 1.8645461417180886}, + {Level: "8", Coef: 1.4885780486208022}, + }}, + }, +}