Skip to content

fix(stats): greeks, r_squared and treynor_ratio break on any gap - #553

Closed
WatchTree-19 wants to merge 1 commit into
ranaroussi:mainfrom
WatchTree-19:fix/benchmark-metrics-skip-gaps
Closed

WatchTree-19 wants to merge 1 commit into
ranaroussi:mainfrom
WatchTree-19:fix/benchmark-metrics-skip-gaps

Conversation

@WatchTree-19

Copy link
Copy Markdown

Thanks for picking up #546 and #547 so quickly. Running the new gap handling on main (5a751dd) against the benchmark metrics, one thing breaks: since a missing return now stays NaN, a single gap makes np.cov and linregress return NaN for the whole series.

  • greeks() reports beta and alpha as 0, because the NaN goes through its .fillna(0)
  • r_squared() / r2() return NaN
  • treynor_ratio() falls back to 0 with the "beta is zero" warning
rng = np.random.default_rng(7)
idx = pd.bdate_range("2020-01-01", periods=756)
bench = pd.Series(rng.normal(0.0003, 0.01, 756), index=idx)
strat = 0.8 * bench + rng.normal(0, 0.006, 756)
strat.iloc[[100, 200, 300]] = np.nan

qs.stats.greeks(strat, bench)       # main: beta 0.0, alpha 0.0   (0.0.82: beta 0.76)
qs.stats.r_squared(strat, bench)    # main: nan                   (0.0.82: 0.58)
qs.stats.treynor_ratio(strat, bench)  # main: 0 plus a warning

This is only on main, not in any release. reports.metrics() is not affected, because it fills the frame with 0 before calling these.

The fix estimates over the dates on which both the strategy and the benchmark have an observation, which is the same "skip the gap" rule #546 applies to the single-series statistics. It is a small helper, _paired_observations(), called from greeks() and r_squared(); treynor_ratio() and r2() go through those. Series without gaps give exactly the same numbers as before.

Tests: TestBenchmarkMetricsSkipGaps in tests/test_regressions.py, following the pattern of TestMissingObservationsAreNotZeroReturns. Each metric on a gapped series must match the same metric on the observed dates, and treynor_ratio() must not warn or return 0. All three fail on main and pass with the fix. Full suite: 207 passed. ruff check and ruff format --check are clean. I added a line to the 0.0.83 section of the changelog.

Since missing returns stay NaN (ranaroussi#546), a single gap made np.cov and
linregress return NaN for the whole series: greeks() reported beta and
alpha as 0 through its fillna(0), r_squared() returned NaN, and
treynor_ratio() fell back to 0 with a "beta is zero" warning. Estimate
over the dates on which both the strategy and the benchmark have an
observation.
ranaroussi added a commit that referenced this pull request Sep 26, 2026
…, #553)

Estimate greeks(), r_squared()/r2() and treynor_ratio() over the dates on
which both the strategy and the benchmark were observed. 0.0.83 stopped
filling gaps with 0, and np.cov/linregress propagate NaN through the whole
estimate, so a single missing day reduced beta and alpha to 0 and R-squared
to NaN. greeks() reported that zero without complaint, since its trailing
fillna(0) could not tell "undefined" from "no sensitivity". Reported with a
fix by @WatchTree-19 in #553.

Revert kelly_criterion() to the classic fixed-odds fraction p - q/b. 0.0.82
divided it by the average-loss magnitude, on the argument that a fraction
which does not move when the series is rescaled must be wrong. That quantity
is the growth-optimal leverage for a per-period P&L, not the Kelly fraction:
on daily returns it reaches double and triple digits, and reports showed
figures like 10792% where the fraction reads 21% (#552). The fixed-odds
fraction depends only on the odds by construction, so its scale-invariance
is a property rather than a defect.

Bumps version to 0.0.84.
@ranaroussi

Copy link
Copy Markdown
Owner

Thank you — this was a real regression, and your diagnosis was exactly right. Shipped in v0.0.84.

One correction to the report: this was in a release. 5a751dd is the 0.0.83 release commit, so 0.0.83 on PyPI carried the bug for every user with a benchmark and any gap in their data.

I introduced it in 0.0.83 when I stopped filling missing observations with 0 (#546). That change was right on its own terms — a missing day is not a flat day — but I did not audit the joint estimators, and np.cov/linregress propagate NaN through the entire result. The part that made it dangerous is the one you pointed at: greeks() ends with .fillna(0), which cannot distinguish "undefined" from "no sensitivity to the benchmark", so it reported a confident-looking beta = 0.0.

Reproducing your snippet on 0.0.83:

no gap : {'beta': 0.7649, 'alpha': -0.0006}  r2 0.5779
3 gaps : {'beta': 0.0,    'alpha': 0.0}      r2 nan   treynor 0

And on 0.0.84:

3 gaps : {'beta': 0.7657, 'alpha': -0.0093}  r2 0.5793  treynor -0.4944

I implemented your approach rather than merging the branch directly, because I was also reverting the Kelly change in the same release and wanted one coherent diff; the logic is yours. It is factored into a _paired_observations() helper shared by greeks() and r_squared(), with treynor_ratio() inheriting the fix through greeks(). You are credited in the release notes and the changelog.

I also swept every benchmark-taking function in stats against a gapped series to check for the same exposure — information_ratio, rolling_greeks and compare all came back clean.

One subtlety worth recording, which your approach gets right: estimating over the shared observations is deliberately not the same as calling dropna() first. Dropping the dates makes the strategy look like an irregular-frequency series, so _prepare_benchmark() compounds the benchmark across each missing day (day 101 becomes the two-day move, -0.01142 instead of -0.008699). Leaving them NaN keeps a daily series with three unobserved days and discards the benchmark's move on those days. Both are defensible, but they answer different questions, and the NaN reading is the correct one here. There is a regression test pinning that distinction so nobody "simplifies" it away later.

Closing as shipped. This is the second report of yours that held up under scrutiny (#545 was the other) — genuinely useful, please keep them coming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants