Skip to content

Multi-head retention-time calibration by default - #81

Merged
RobbinBouwmeester merged 3 commits into
mainfrom
feat/multihead-default
Sep 11, 2026
Merged

Multi-head retention-time calibration by default#81
RobbinBouwmeester merged 3 commits into
mainfrom
feat/multihead-default

Conversation

@RobbinBouwmeester

Copy link
Copy Markdown
Member

Draft until the entrapment arm lands. The empirical-null half of CLAUDE.md's promotion gate is running now; see Gate status below. Everything else is complete and green.

Leaving rt_im_train.multihead_calibration unset now calibrates the DeepLC base model over 80 of its best-correlating LC-setup heads against each run's own confident seed PSMs, instead of using the single head deeplc.predict returns.

Measured

Two acquisitions, six pooled runs each, configurations differing only in this setting (docs/08 §4d):

AIF baseline AIF multi-head Astral baseline Astral multi-head
stripped peptides @1% 80,842 84,725 (+4.8%) 102,942 117,652 (+14.3%)
precursors @1% 100,753 105,603 (+4.8%) 113,961 131,646 (+15.5%)
protein groups @1% 10,707 10,963 (+2.4%) 11,458 12,276 (+7.1%)
empirical decoy fraction 0.0100 0.0100 0.0100 0.0100
PSMs reaching rescore 11.90M 11.27M 7.17M 3.51M

Identical decoy fraction in all four arms, and fewer candidates reaching rescore with more identifications out of them: interference removed, not a threshold traded.

Cost: 1.4×–1.7× wall clock. The calibration is fitted against each run's own anchors, so it cannot be shared across an experiment. multihead_calibration: 0 restores the previous behaviour exactly.

Why the field became Option<usize>

A bare number could not express this default without breaking two working configurations:

  1. A native, Python-free run is supported. A non-zero default would have made "no DeepLC interpreter" a startup error. Unset means automatic, and automatic means nothing when no interpreter resolved.
  2. finetune_deeplc occupies the same slot, and validation refuses the pair — so a config enabling the fine-tune would have been rejected for a conflict it never wrote. The fine-tune now keeps its slot; asking for both explicitly is still an error.

The default is scoped, and three tests caught the absence of that

It applies only where the run's retention times are DeepLC's already: imported library under library_irt = auto/deeplc, or FASTA with rt_predictor = deeplc. My first cut lacked this and would have let a rt_predictor: native run have its RT source silently replaced by DeepLC because an unrelated interpreter happened to be discoverable. An explicit count ignores the scope and is a hard requirement.

config.rs::multihead_heads is the single resolver, so the two orchestrators and the library_irt predicate cannot disagree about whether it ran.

Two consequences worth knowing:

  • the separate library_irt base-model re-prediction no longer runs under the defaults, because multi-head re-predicts the library itself — about 27 minutes saved on a 10.9M-row library, producing a column it would have overwritten;
  • model_identities.rt_predictor now reads multihead-80 rather than deeplc-4.4.0-base. That is how to tell which one a given run used.

Also fixed, because this default made it reachable

An optional numeric setting could not be changed from the desktop settings screen at all. Option<usize> was classified as schema kind other, so the form sent the string "0" and the engine rejected it — the off-switch for the very default this PR ships. The generator now unwraps Option<T> for numeric T (this also fixes extract.windows_in_flight), and a blank optional number means unset rather than an error.

Gate status

CLAUDE.md requires entrapment plus two acquisition contexts before a sensitivity default moves.

  • Two acquisitions: done (AIF and Astral, above).
  • Entrapment: running. The archived entrapment library turned out to be native-rt-v1; native-frag-v1, and a run against it produces 333 confident seeds and 6,401 candidates surviving extract — the degenerate regime, which measures nothing. It is being rebuilt with MS2PIP HCDch2 + DeepLC over the same 5.83M peptidoforms, after which both arms run against it and the FDP goes in this PR and in docs/08.

I will not take this out of draft until that number exists.

Validation

  • engine: cargo fmt --check, clippy --all-targets -D warnings, cargo test --workspace 283 passed
  • desktop: cargo test --lib 87 passed; check_desktop_ui.py; node --check
  • gen_config_reference.py --check, check_doc_refs.py clean

Two pre-existing tests changed meaning rather than breaking, and are updated to assert the new semantics: the base-model re-prediction is now suppressed under the shipped defaults because multi-head is the re-prediction.

🤖 Generated with Claude Code

RobbinBouwmeester and others added 2 commits September 11, 2026 17:35
Leaving `rt_im_train.multihead_calibration` unset now calibrates the DeepLC base
model over 80 of its best-correlating LC-setup heads against each run's own
confident seed PSMs, in place of the single head `deeplc.predict` returns.

Measured on two acquisitions, six pooled runs each, at an unchanged empirical
decoy fraction of 0.0100 in all four arms (docs/08 section 4d): AIF 80,842 ->
84,725 peptides (+4.8%), Astral 102,942 -> 117,652 (+14.3%), precursors +4.8%
and +15.5%, protein groups +2.4% and +7.1%. Fewer candidates reach rescore and
more of them are real, which is interference removed rather than a threshold
traded. It costs 1.4x to 1.7x wall clock, because the calibration is fitted
against each run's own anchors and cannot be shared across an experiment;
`multihead_calibration: 0` restores the previous behaviour exactly.

The field becomes `Option<usize>`, because a bare number could not express this
default without breaking two things:

- a native, Python-free run is a supported configuration, and a default it
  cannot satisfy would have turned "no interpreter" into a startup error. Unset
  means automatic, and automatic means nothing when no interpreter resolved;
- `finetune_deeplc` occupies the same slot, and validation refuses the pair. A
  configuration that enables the fine-tune would have been rejected for a
  conflict it never wrote. The fine-tune now keeps its slot; asking for both
  explicitly is still an error.

The default is also SCOPED, which three existing tests caught the absence of: it
applies only where the run's retention times are DeepLC's already -- an imported
library under `library_irt = auto` or `deeplc`, or FASTA with `rt_predictor =
deeplc`. Without that scope a `rt_predictor: native` run would have had its RT
source silently replaced by DeepLC because an unrelated interpreter happened to
be discoverable. An explicit count ignores the scope and is a hard requirement.

`config.rs::multihead_heads` is the single resolver, so the two orchestrators and
the `library_irt` predicate cannot disagree about whether it ran. Two
consequences: the separate base-model re-prediction no longer runs under the
defaults, because multi-head re-predicts the library itself (27 minutes on a
10.9M-row library, producing a column it would overwrite); and
`model_identities.rt_predictor` now reads `multihead-80` rather than
`deeplc-4.4.0-base`, which is how to tell which one a run used.

Also fixed, because this default made it reachable: an optional numeric setting
could not be changed from the desktop settings screen. `Option<usize>` was
classified as kind "other", so the form sent the STRING "0" and the engine
rejected it -- the off-switch for the very default this commit ships. The
generator now unwraps `Option<T>` for numeric T (`extract.windows_in_flight` too)
and a blank optional number means unset rather than an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLAUDE.md gates a sensitivity default on entrapment plus two acquisition
contexts. The two acquisitions were already recorded; this is the empirical null.

E. coli AIF file against an E. coli + 1:1 human entrapment library, one binary,
two runs differing only in rt_im_train.multihead_calibration:

  real peptides at 1%    7,879 -> 8,216  (+4.28%)
  spike-in peptides        138 -> 144
  empirical FDP          0.995% -> 0.995%

Identical to three decimal places, so the gain is real discoveries rather than a
loosened threshold. At 138 and 144 accepted spike-ins this is about five times
more sensitive than the entrapment arms in docs/28, which rest on 22-30 and
cannot resolve below roughly 0.1 percentage points.

Two things the write-up is careful not to over-read. Decoys accepted at the
entrapment threshold moved 91 to 113; under classifier=entrapment the q column is
the entrapment FDP rather than target-decoy, so that is a side diagnostic and not
the threshold in use. And the 1.06x wall clock here is not the cost of the
default: a single run whose baseline already pays a DeepLC re-prediction displaces
that work rather than adding to it, so the 1.4x-1.7x from the pooled experiments
is what to plan around.

The archived entrapment library could not be used. Its report reads
`native-rt-v1; native-frag-v1`, and a run against it returns 333 confident seeds
and 6,401 candidates surviving extract -- the degenerate regime the HCDch2 finding
describes, in which nothing can be measured. It was rebuilt over the same
5,828,348 peptidoforms with MS2PIP 4.2.0 HCDch2 and DeepLC 4.4.0
(`deeplc-4.4.0-base; ms2pip-4.2.0-HCDch2`, 69,564,520 fragments). Anyone repeating
this benchmark should check the library's model_identity first, which is why that
is written down.

`bench/entrapment_fdp.py` is the analysis, mirroring
`rescore.rs::classify_entrapment` and recomputing `(ratio * entrapment + 1) / real`
rather than trusting a single column.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@RobbinBouwmeester
RobbinBouwmeester marked this pull request as ready for review September 11, 2026 16:48
@RobbinBouwmeester

Copy link
Copy Markdown
Member Author

Entrapment arm: done, gate closed

The empirical null CLAUDE.md requires. E. coli AIF file against an E. coli + 1:1 human entrapment library, one binary, two runs differing only in rt_im_train.multihead_calibration.

baseline multi-head
real peptides @1% 7,879 8,216 (+4.28%)
spike-in peptides accepted 138 144
empirical FDP 0.995% 0.995%
target PSMs @1% 8,832 9,166
wall clock 10:19 10:55

FDP is (ratio * entrapment + 1) / real at entrapment_ratio = 0.560632, the engine's own estimator, and it is identical to three decimal places. The gain is real discoveries at the same measured false-discovery proportion.

At 138 and 144 accepted spike-ins this is roughly five times more sensitive than the entrapment arms in docs/28, which rest on 22-30 and cannot resolve FDP differences below about 0.1 percentage points.

Two things deliberately not over-read:

  • decoys accepted at the entrapment threshold moved 91 → 113 (1.03% → 1.23% of targets). Under classifier = entrapment the q column is the entrapment FDP, not target-decoy, so this is a side diagnostic rather than the threshold in use. Worth watching if it grows; the spike-in FDP is the null and it did not move.
  • the 1.06x wall clock here is not the cost of the default. This is a single run whose baseline arm already pays a DeepLC re-prediction, so multi-head displaces work rather than adding it. The 1.4x–1.7x from the six-run pooled experiments is what to plan around.

The library had to be rebuilt first

The archived entrapment library reads native-rt-v1; native-frag-v1, and a run against it gives 333 confident seeds and 6,401 candidates surviving extract — the degenerate regime the HCDch2 finding describes, in which nothing can be measured. It was rebuilt over the same 5,828,348 peptidoforms with MS2PIP 4.2.0 HCDch2 + DeepLC 4.4.0 (deeplc-4.4.0-base; ms2pip-4.2.0-HCDch2, 69,564,520 fragments). Anyone repeating this benchmark should check model_identity first, which is now written down in docs/08.

bench/entrapment_fdp.py is the analysis: it mirrors rescore.rs::classify_entrapment and recomputes the FDP from the formula rather than trusting one column.

Gate status: both halves now satisfied. Two acquisitions (AIF, Astral) and an empirical null. Out of draft.

# Conflicts:
#	configs/config-schema.json
#	docs/24_config_reference.md
#	rust/mumdia/crates/mumdia/src/python.rs
@RobbinBouwmeester
RobbinBouwmeester merged commit 901b463 into main Sep 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant