Repository navigation
fix(utils): hash a time-varying rf in the prepare_returns cache key - #555
WatchTree-19 wants to merge 1 commit into
Conversation
The cache key formatted rf into the string with its repr. For a Series pandas truncates that to the first and last rows, so two rate series of the same length that differed only in between produced the same key and the second call was handed the first call's excess returns. A Series or DataFrame rf is now hashed with hash_pandas_object, the same way the data already is. Scalar rf keys are unchanged.
, #556) avg_return(), avg_win() and avg_loss() masked the unselected cells to NaN and then called .dropna(), which drops whole rows on a DataFrame. Each column's average was computed only over rows where every other column also qualified, so a strategy's average win was measured only on the days the benchmark also rose. reports.metrics() with a benchmark therefore reported affected Average Win/Loss, Payoff Ratio, Win/Loss Ratio, CPC Index and Kelly Criterion for DataFrame input. .mean() already skips NaN column-wise (#556). rolling_greeks() computed alpha from the full-sample means instead of each window's own, so every row shared one baseline and rolling alpha only moved when rolling beta did. Rolling beta is unchanged (#554). _generate_cache_key() formatted a time-varying rf into the key with its repr, which pandas truncates to the first and last few rows: two rate series of the same length differing only in between shared an entry, and the second caller got the first one's excess returns back. The answer depended on call order (#555). Both rolling_greeks and the cache key were reported with fixes by @WatchTree-19; the contamination was found from the data @none2003 attached to #556. Bumps version to 0.0.86.
|
Confirmed and shipped in v0.0.86. This one was mine, from the cache-key work in 0.0.82, and it was the nastiest of the three. Reproduced before changing anything: What makes it worse than a wrong number is the property you identified: the answer depends on call order. The same script produces different Sharpe ratios depending on what ran before it in the same process, and there is no warning, no exception, and nothing in the output to indicate which of the two you received. A notebook re-run in a different cell order would silently change results. I added the The regression test asserts You're credited in the release notes and the changelog. Closing as shipped. Both of your PRs this round were correct, reproducible from the snippet as written, and came with a working test. That is genuinely rare, and it is appreciated. |
|
Thanks Ran, really appreciate the write-up and the credit in the release notes. I'm a trusted contributor to Microsoft's PyRIT and in regular contact with their AI Red Team, and would be glad to lend a hand on quantstats too, triage, reviews or anything else. |
|
Thank you — that is a generous offer, and the quality of the three reports you filed this round speaks for itself. Each one was reproducible from the snippet as written, correctly diagnosed, and came with a test that failed before the fix and passed after. That is a rarity. Let me sit with it rather than commit to anything on the spot. Ran owns decisions about repository access and project roles, and those are worth making deliberately rather than in a release thread. In the meantime the most valuable thing you can do is exactly what you have been doing: keep filing reports like these. They need no permissions, and the recent ones led directly to 0.0.84 through 0.0.86. |
_generate_cache_key()hashes the data but formatsrfinto the key as text. For a time-varyingrfSeries that text is the pandas repr, which only shows the first and last few rows. Two rate series of the same length that differ in between get the same key, so the second call is handed the first call's excess returns:The fix hashes a Series or DataFrame
rfwithhash_pandas_object, the same way the data is hashed already. Scalarrfkeys are unchanged. It sits next to the metadata fix from #544.Tests:
test_rate_series_differing_mid_sample_do_not_collideinTestPrepareReturnsCache. It fails on main and passes with the fix. Full suite: 214 passed (the onetest_normalize_tz_awareerror is environmental and happens on main too).ruffis clean. Changelog line under 0.0.86.Both this and the rolling_greeks PR add a 0.0.86 changelog heading, so whichever lands second will want its line moved under the first one's heading.