Skip to content

About

Find the four ways a backtest lies: look-ahead, ledger invariants, fee/funding double-counting, mutation blind spots. Dependency-free checkers + audit report format.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

backtest-audit-kit

"I verify whether your strategy is real — or whether the backtest lied to you."

A tiny, dependency-free toolkit that audits a backtest the way an accountant audits a ledger and a QA engineer audits a test suite. It does not sell you returns; it finds the four ways a backtest most often lies.

Case library → — twelve reproducible ways a backtest lies, each with the mechanism, the number, and the repro.

回测审计 / 策略代码复核 — 一条路径,一份书面报告,48 小时。 ¥199 / $29,查不出可复现的问题不收钱。给我一份交易 CSV、一段回测脚本、或一个引擎文件都行。 详情与边界 → PORTFOLIO.md · 邮箱 gbbggjvf@gmail.com 交付物长什么样 → 一页真实样例(已匿名) 另一种形态:整批验证(37 个公开策略一次扫完)→ 一页样例

Backtest audit / strategy-code review — one path, one written report, 48 hours. ¥199 / $29, no reproducible finding, no fee. Send a trade CSV, a backtest script, or one engine file. Scope and limits → PORTFOLIO.md · gbbggjvf@gmail.com What the deliverable looks like → one-page sample, real audit, anonymised Another shape: a whole-leaderboard pass (37 public strategies, one screen) → one-page sample

一条命令(3 分钟)

python3 -m auditkit.cli your.csv --trials 50     # 不用装,纯标准库
pip install -e . && auditkit your.csv            # 或者装上,用 auditkit 命令

给它一份交易/净值 CSV(date + equity,或 date + pnl / return),它只回三件事:

检查 问的问题 什么时候红
look-ahead 时间戳和成交价物理上可能吗 时间戳倒退、entry==exit 零持有、入场价落在该 bar 的 high/low 之外 = FAIL
multiple testing 这个 Sharpe 能和「试了 N 次里最好的那次」分开吗 校正后 p ≥ 0.05 = FAIL(--trials 必须填你实际搜过的参数组数;在多个时间框架上各调过参数,再加 --timeframes N,N_eff = trials × timeframes;不填则标 UNVERIFIED,不冒充通过)
out-of-sample decay 后半段还剩多少边 后半 SR ≤ 0 = FAIL,衰减 ≥ 50% = WARN

多尺度搜索会单独标一行:同一策略在 5 个时间框架上各调 100 组 = N_eff 500,门槛按 500 算;若按 100 会判 PASS、按 500 判 FAIL,会额外出一条 WARN 说明膨胀。

退出码:0 全过 · 1 有 FAIL · 2 有缺输入判不了的项。加 --ohlc bars.csv(date, high, low)才能判价格级前视;--json 给机器读;--split 0.6 改样本内比例。

One command (3 minutes)

python3 -m auditkit.cli your.csv --trials 50 prints three verdicts — look-ahead, multiple-testing (deflated Sharpe), out-of-sample decay — from a plain CSV (date + equity, or pnl / return). Exit code 0 = pass, 1 = FAIL, 2 = needs input. Without --trials the multiple-testing line says UNVERIFIED instead of inventing a pass. Tuned the same grid on several time frames? Add --timeframes N — effective trials become trials × N, and the tool prints the inflated N and, when it flips the verdict, a WARN line. --ohlc bars.csv unlocks the price-level look-ahead check.


English

What it is. A pure standard-library (Python 3.9+) kit of four independent checkers for a trading backtest:

Checker Question it answers
checks/lookahead.py Does any decision on bar i see bars i+1...? (AST-based future-function lint)
checks/invariants.py Does the ledger obey accounting invariants? (equity identity, wd_cum carry, monotonic counters, non-negative balances)
checks/mutants.py Which bugs would your test suite not catch? (mutation testing; survivors = coverage blind spots)
checks/fees.py Are fees two-sided, funding charged per bar, and slippage counted once per fill?
checks/bust_revival.py After a ruin check, does the account get reset to 1% and keep trading? (see cases/case10_bust_revival.md)

Who it is for. Quant developers, prop-desk researchers, and anyone who has ever shipped a strategy that looked great in a notebook and died in production. If you are reviewing someone else's backtest — or your own — this gives you a checklist, reproduction scripts, and before/after numbers instead of opinions.

What I deliver. ① A ranked findings list plus a runnable reproduction script for each problem. ② A minimal fix patch. ③ A before/after reconciliation of the headline numbers (win rate, net PnL, multiple, MDD, t-stat).

Three sample cases — this page summarizes three of the fifteen write-ups in cases/:

New: Case 15 — a +90.2% annualised claim, recomputed. The monthly cells do compound to 90.2%, but the figure is hypothetical (the vendor's own disclaimer), three months carry it (remove them → 34.8%), and monthly peak-to-trough is −21.8%. Arithmetic consistent; not an audited real-money return.

New: Case 14 — a peer audit of a live account. 1,701 setups → median DSR 0.00, PBO 0.03–0.19; 1,041 live trades → −$1,308.74 realized, 78.29% win rate against an 83.47% break-even, and a $131.44 equity gap that turned out to be financing (plus $979.82 of spread). Owner's permission to name him is quoted in the file.

New: Case 13 — half the tickers, twice the PBO. A retraction: I published PBO 73.33% against a README claim of 20%, but the run had loaded 63 of 75 tickers — the missing 12 shifted every cross-sectional rank. Full universe gives 46.67% / 26.67%, matching the maintainer. Same commit, same scripts; only the loaded count changed.

New: Case 12 — the same 333 trades, three verdicts. One trade list, one data file; it passes --trials 20 --timeframes 1 (p=0.024) and fails the same line with --timeframes 5 (N_eff=100, p=0.12). The third block counts the search we actually ran, 576 configurations, and fails on a single time frame. Three outputs, one screen, nothing hidden.

  1. Look-ahead — the exit scan ran on the entry bar, so every signal knew its outcome. Fixed: 91 trades / 90.1% / +$214.37 → 26 trades / 65.4% / -$0.12; a tick-level re-run gives 27 trades / 66.7% / +$0.37. The leak was worth ≈ $214 of imaginary PnL — the whole profit.
  2. Withdrawal accounting — the bust path dropped wd_cum (10 return sites). It only fires when a run busts: at lev 1.2 the published config never busts, so 7.82x → 7.82x; at lev 5 it busts once and 0.00x → 5.97x. The ~27x gap between 215.27x (no withdrawal) and 7.82x (double-keep-half) is withdrawal policy, not the bug — an earlier version of this page paired those two and mis-attributed it.
  3. Drawdown metric — the circuit-breaker reset the running peak, so the "reported MDD" was only the local drawdown. Same run, same config: investor-view raw drawdown 98.7% vs reported 53.1% — a 45.6 pp gap. Conclusion: MDD must be written twice (breaker-local and raw). (Bonus case 4: 5-minute bar clustering inflated n ~11x and turned t = 15.60 into t = 1.75 after a 24h cooldown.)

Two public cases, both answered and followed by a change — threads outside this repo; one-page PDF: audits/2026-11-01-two-public-cases.pdf.

Thread The number I questioned What happened
mr-scrooge-v6 #9 -$1,308.74 over 1,041 trades (t ~ -3.0) accepted; ratchet re-tightened, config bug fixed (B-142 to B-145)
pineforge-engine #664 341 of 7,989 probes graded at a 100x wider PnL tolerance accepted; regrade of all 7,989 due 2026-11-04

Full write-up: audits/2026-11-01-two-public-cases.md. Not paid work, and not a track record: 23 public threads opened, 7 answered (30%), 0 paid.

What a deliverable looks like - audits/2026-11-01-sample-verification-report.pdf. One page, on a public live record recomputed from its own daily file: six numbers side by side, two convention gaps explained, one command to re-run. That result was no error found - which is also a result.

Install.

pip install backtest-audit-kit

Use it in code (four checkers, pure standard library):

from auditkit.checks import lookahead
findings = lookahead.detect_file("strategy.py")   # future-data lint
print(lookahead.format_findings(findings))

How to run.

cd portfolio/backtest-audit-kit
python3 -m unittest discover tests      # stdlib only, no install
# or, if you have pytest:
python3 -m pytest tests/ -q
python3 examples/run_demo.py            # see all four checkers in action

Honesty note. The look-ahead checker is a heuristic linter, not a proof engine: it flags the shapes that leak, and a stand-alone forward-simulation pass is reported MEDIUM for human review rather than being declared safe. Mutation testing reports blind spots; it never claims your strategy is profitable. No checker here predicts returns.


中文

是什么。 一个纯 Python 标准库(3.9+)实现的回测审计工具包,四个互相独立 的检查器:

检查器 回答的问题
checks/lookahead.py 在 bar i 上的决策,是否读到了 i+1... 的数据?(基于 AST 的前视/未来函数检测)
checks/invariants.py 账本是否满足会计不变量?(权益恒等式、wd_cum 结转、计数器单调、余额非负)
checks/mutants.py 你的测试套件抓不到哪些 bug?(变异测试;存活的变异体 = 覆盖盲区)
checks/fees.py 手续费是否双边计、资金费是否逐 bar 计、滑点是否每笔只算一次?
checks/bust_revival.py 爆仓检查之后,账户是否被重置成 1% 继续交易?(见 cases/case10_bust_revival.md)

给谁用。 量化开发者、自营研究岗、以及任何"回测很美、实盘就死"过的 人。带看别人的回测(或自己的)时,它给你的是问题清单 + 复现脚本 + 修复前后 的数字,而不是一句"我觉得不太行"。

交付物承诺。 ① 按严重度排序的问题清单,每个问题配一个可运行的复现脚本。 ② 最小修复补丁。③ 修复前后的对账数字(胜率 / 净利 / 倍数 / 回撤 / t 值)。

三个案例摘要(本页只摘三份;cases/ 里另有完整 15 份):

新增:案例 15 —— 复算一个 +90.2% 的年化。月度格子复利确实得到 90.2%,但该数是 hypothetical(C2 自己的免责声明),且由最大的三个月撑起(去掉后 → 34.8%),月度收盘最大回撤 −21.8%。算术自洽,但不是可审计的实盘收益。

新增:案例 14 —— 一个实盘账户的同行审计。1,701 个假设 → 中位 DSR 0.00、PBO 0.03–0.19;1,041 笔实盘 → 已实现 −$1,308.74、胜率 78.29% 对盈亏平衡所需的 83.47%,以及一笔 $131.44 的权益缺口(最终是隔夜融资,另有 $979.82 点差)。Owner 同意署名的原话在文内。

新增:案例 13 —— 半套票,双倍 PBO。一次撤回:我曾公布 PBO 73.33%(README 写 20%),但那次只加载了 75 只里的 63 只 —— 缺的 12 只改变了所有截面排名。满 75/75 复算是 46.67% / 26.67%,与维护者一致。同一 commit、同一脚本,只有加载数不同。

新增:案例 12 —— 同一份 333 笔交易,三种判决。一份交易表、一份数据:--trials 20 --timeframes 1 判 PASS(p=0.024),同一行加 --timeframes 5 判 FAIL(N_eff=100,p=0.12);第三块按我们实际扫过的 576 组算,单时间框架就 FAIL。三段输出同页,不藏的。

  1. 前视 bug —— 出场扫描写在开仓 bar 的循环内,每个信号都提前知道结果。 修复后:91 笔 / 90.1% / +$214.37 → 26 笔 / 65.4% / -$0.12;tick 级 复跑得 27 笔 / 66.7% / +$0.37。这个泄漏值 约 $214 的虚假收益, 即全部利润。
  2. 提现会计 bug —— 爆仓返程丢了 wd_cum(引擎里有 10 处这样的 return)。 它只在爆仓时触发:lev 1.2 的公开配置全程 0 爆仓 ⇒ 7.82x → 7.82x, 一分不变;lev 5 爆仓一次 ⇒ 0.00x → 5.97x。215.27x 与 7.82x 之间那 27 倍 是提现规则本身的代价(不提现 vs 翻 2 留 1),不是 bug —— 早先这一页把 两个配置配成了一对,归因是错的,已改。
  3. 回撤口径 bug —— 熔断触发时把 peak 重置为 cap,"报告 MDD"只是每段熔断 后的局部回撤。同一次 run、同一配置:投资者口径 98.7% vs 报告 53.1%,差 45.6 个百分点。结论:MDD 必须双写(熔断口径 + 原始口径)。 (附赠案例 4:一段暴跌连打十几根 5min bar 让样本量虚增约 11 倍,t 值从 15.60 在 24h 冷却去重后掉到 1.75。)

两个公开案例(都在本仓库之外) —— 对方认账并改了代码;一页 PDF:audits/2026-11-01-two-public-cases.pdf。

帖子 我质疑的数字 结果
mr-scrooge-v6 #9 1,041 笔合计 -$1,308.74(t ≈ -3.0) 认账;收紧 ratchet,修掉配置加载 bug(B-142 至 B-145)
pineforge-engine #664 7,989 个 probe 里 341 个按宽 100 倍的 PnL 容差评分 认账;承诺 2026-11-04 前重跑全部 7,989

完整文字:audits/2026-11-01-two-public-cases.md。不是付费工作,也不是战绩:公开 23 条,7 条有回复(30%),付费 0。

第三份,性质不同:读源码。 APEX V2 [JOAT](开源 Pine,作者 JOAT)——不跑、不看净值,只把描述里的每个数字对回代码,查出六处对不上:面板上的 8.0σ 其实是 12 个标准差(分母带 * 1.5);confluence 分母 9 但输入上限 7;九个票里有俩测的是同一个 VWAP 偏离;Enable Trailing Stop 默认开着却永远触发不了(limit 和 trail_price 同价);Risk Per Trade % 算完没用(仓位走 default_qty_value=100);strategy() 无手续费无滑点。每条都带行号,附 sed 复现命令:audits/2026-11-01-apex-v2-code-audit.md。没有对方回复 —— TradingView 脚本页没有评论区(页面 706KB 里 comment 只出现两次,全是数据字段;相关 API 全 404),所以这份是"结论"而不是"回合"。

交付物长什么样 —— audits/2026-11-01-sample-verification-report.pdf。一页纸,靶子是一份公开的实盘记录,用他本人发布的逐日文件复算:六个数字逐行对照,两处口径差异讲清楚,附一条可重跑的命令。那次结论是「未发现错误」——这也是结果。

怎么跑。

cd portfolio/backtest-audit-kit
python3 -m unittest discover tests      # 只用标准库,无需安装
# 若装了 pytest:
python3 -m pytest tests/ -q
python3 examples/run_demo.py            # 四个检查器全跑一遍

诚实声明。 前视检查器是启发式 linter,不是证明器:它抓的是会泄漏的代码 形态,独立的"向前模拟"循环会被标为 MEDIUM 交人工复核,而不是被判安全。 变异测试只报告覆盖盲区,从不声称你的策略赚钱。本工具包不预测任何收益。


Also in this account · 另见

  • outpost/ — 一个"只放能跑、能查、能被否证的东西"的据点: 单文件表格整理工具(零依赖、离线自检 14 项)+ 一份被自己推翻的回测(做空山寨无 edge)
    • 一次"两条腿都断了"的数据源修复。不写客户名、不写"我能做什么"。

账本审计(ledger):单份或批量

python3 -m auditkit.ledger run examples/ledger/clustered.json
python3 -m auditkit.ledger batch examples/ledger/manifest.json --out report.md

七张卡针对的是「交易账本」而不是净值 CSV:账本恒等、逐笔回撤、样本膨胀、成本 vs 过路费、前视扫描、数值稳定性、止盈是否低于往返费。

两种输入形状都吃:带 pnl_usdt / margin / leverage 的账本,或旧 dump 的 trades + ledger 序列。 缺字段的卡标「未测」,不猜:dump 没有 balance 就不做账本恒等,没有入场时间就不做样本膨胀。

批量输出一张对照表(笔数 / 期末权益 / MDD / 估算成本 / 成本倍数 / 命中卡)加逐卡覆盖统计。 批量提高的是吞吐,不是命中率 —— 同一个策略不会因为跑得多而变简单。

证明什么 / 不证明什么

一条命令:python3 examples/run_demo.py

证明

  • 四个检查器能在同一份劣质输入上各自报出具体违约,并给出文件与行号。
  • 失败样例是内置的、可对照的:tests/fixtures/lookahead_buggy.py 报 4 条(2 HIGH / 2 MEDIUM),lookahead_fixed.py 报 0 条。
  • 变异测试能区分弱套件与强套件:weak suite 杀掉 1/2(50%),strong suite 杀掉 2/2(100%)。它不是在乱报警,也不会放过真漏洞。
  • 账本不变量能定位到具体规则:BUST_WD_CARRY 在 step=1 触发。

不证明

  • 不预测收益,不判断策略好坏。
  • 不覆盖你没有提供的代码路径,只审给到的东西。
  • 不替代人工判断:取舍、严重级别、是否值得改,仍由人决定。
  • 「查不出问题」不等于「没有漏洞」,只等于「在这套检查器下没查出」。

About

Find the four ways a backtest lies: look-ahead, ledger invariants, fee/funding double-counting, mutation blind spots. Dependency-free checkers + audit report format.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages