Skip to content

fix(utils): hash a time-varying rf in the prepare_returns cache key - #555

Closed
WatchTree-19 wants to merge 1 commit into
ranaroussi:mainfrom
WatchTree-19:fix/cache-key-rf-series
Closed

WatchTree-19 wants to merge 1 commit into
ranaroussi:mainfrom
WatchTree-19:fix/cache-key-rf-series

Conversation

@WatchTree-19

Copy link
Copy Markdown

_generate_cache_key() hashes the data but formats rf into the key as text. For a time-varying rf Series that text is the pandas repr, which only shows the first and last few rows. Two rate series of the same length that differ in between get the same key, so the second call is handed the first call's excess returns:

idx = pd.bdate_range("2020-01-01", periods=756)
r = pd.Series(np.random.default_rng(1).normal(0.0005, 0.01, 756), index=idx)
rf_a = pd.Series(0.01, index=idx)
rf_b = rf_a.copy()
rf_b.iloc[10:-10] = 0.05          # same first and last rows as rf_a

qs.stats.sharpe(r, rf=rf_a)        # -0.195
qs.stats.sharpe(r, rf=rf_b)        # -0.195 on main, -0.439 with the fix
                                   # (-0.439 is also what B gives on a fresh cache)

The fix hashes a Series or DataFrame rf with hash_pandas_object, the same way the data is hashed already. Scalar rf keys are unchanged. It sits next to the metadata fix from #544.

Tests: test_rate_series_differing_mid_sample_do_not_collide in TestPrepareReturnsCache. It fails on main and passes with the fix. Full suite: 214 passed (the one test_normalize_tz_aware error is environmental and happens on main too). ruff is clean. Changelog line under 0.0.86.

Both this and the rolling_greeks PR add a 0.0.86 changelog heading, so whichever lands second will want its line moved under the first one's heading.

The cache key formatted rf into the string with its repr. For a Series
pandas truncates that to the first and last rows, so two rate series of
the same length that differed only in between produced the same key and
the second call was handed the first call's excess returns.

A Series or DataFrame rf is now hashed with hash_pandas_object, the same
way the data already is. Scalar rf keys are unchanged.
ranaroussi added a commit that referenced this pull request Sep 27, 2026
, #556)

avg_return(), avg_win() and avg_loss() masked the unselected cells to NaN and
then called .dropna(), which drops whole rows on a DataFrame. Each column's
average was computed only over rows where every other column also qualified,
so a strategy's average win was measured only on the days the benchmark also
rose. reports.metrics() with a benchmark therefore reported affected Average
Win/Loss, Payoff Ratio, Win/Loss Ratio, CPC Index and Kelly Criterion for
DataFrame input. .mean() already skips NaN column-wise (#556).

rolling_greeks() computed alpha from the full-sample means instead of each
window's own, so every row shared one baseline and rolling alpha only moved
when rolling beta did. Rolling beta is unchanged (#554).

_generate_cache_key() formatted a time-varying rf into the key with its repr,
which pandas truncates to the first and last few rows: two rate series of the
same length differing only in between shared an entry, and the second caller
got the first one's excess returns back. The answer depended on call order
(#555).

Both rolling_greeks and the cache key were reported with fixes by
@WatchTree-19; the contamination was found from the data @none2003 attached
to #556.

Bumps version to 0.0.86.
@ranaroussi

Copy link
Copy Markdown
Owner

Confirmed and shipped in v0.0.86. This one was mine, from the cache-key work in 0.0.82, and it was the nastiest of the three.

Reproduced before changing anything:

str(rf_a) == str(rf_b)          True     <- the two rate series are indistinguishable as text
rf_a.equals(rf_b)               False    <- but they are not the same data

sharpe(rf=B) on a fresh cache   -0.4388  (correct)
sharpe(rf=B) after sharpe(rf=A) -0.1952  (the other series' excess returns)

What makes it worse than a wrong number is the property you identified: the answer depends on call order. The same script produces different Sharpe ratios depending on what ran before it in the same process, and there is no warning, no exception, and nothing in the output to indicate which of the two you received. A notebook re-run in a different cell order would silently change results.

I added the repr(rf) formatting in 0.0.82 while fixing a different collision in that key (#543, Series versus DataFrame metadata). I hashed the data properly and then folded rf in as text without considering that rf can itself be a Series — and that pandas truncates long reprs to the first and last few rows. Your fix, hashing it with hash_pandas_object exactly as the data is hashed, is the right one and is what shipped. Scalar rf keys are unchanged.

The regression test asserts str(low) == str(high) first, so it documents precisely the collision a repr-based key cannot see.

You're credited in the release notes and the changelog. Closing as shipped.

Both of your PRs this round were correct, reproducible from the snippet as written, and came with a working test. That is genuinely rare, and it is appreciated.

@ranaroussi ranaroussi closed this Sep 27, 2026
@WatchTree-19

Copy link
Copy Markdown
Author

Thanks Ran, really appreciate the write-up and the credit in the release notes. I'm a trusted contributor to Microsoft's PyRIT and in regular contact with their AI Red Team, and would be glad to lend a hand on quantstats too, triage, reviews or anything else.

@ranaroussi

Copy link
Copy Markdown
Owner

Thank you — that is a generous offer, and the quality of the three reports you filed this round speaks for itself. Each one was reproducible from the snippet as written, correctly diagnosed, and came with a test that failed before the fix and passed after. That is a rarity.

Let me sit with it rather than commit to anything on the spot. Ran owns decisions about repository access and project roles, and those are worth making deliberately rather than in a release thread.

In the meantime the most valuable thing you can do is exactly what you have been doing: keep filing reports like these. They need no permissions, and the recent ones led directly to 0.0.84 through 0.0.86.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants