fix(error_models): catch mis-stated binned-qual anchors (#263) - #266
Merged
Merged
Conversation
trans is indexed by each unique's mean quality, so binned reads leave mass spikes at the true bins and a thin smear between them. A wrong anchor lands in the smear and fits from a handful of transitions: on NovaSeq 16S, anchors 2,12,23,37 put 241 of 456 rates at the 1e-7 floor. R builds the same model. Three changes: - an anchor inside the observed range with no observations at all is an error (R returns NA there, or stops); - a stern warning when an anchor holds less mass than a non-anchor quality within two of it, naming both masses; - binned-qual warnings print once per run, not once per iteration. Correct-bin models are unchanged and match R to 2e-15 (NovaSeq 16S F and R). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 4, 2026
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #263.
The problem, measured on real data
transis indexed by each unique's mean quality, so binned reads do not produce a cleanly binned matrix. They leave mass spikes at the true bins and a thin smear of averaged qualities between them. On 5 NovaSeq 16S samples: Q37 70%, Q36 11.7%, Q35 5.5%, Q25 3.4%, Q11 1.7%, and every other column under 1%.An anchor that misses the true bins lands in the smear. It still fits, but from a handful of transitions: anchor Q12 rested on 37 transitions, against 1.66% of all transitions at Q11 next to it. With
2,12,23,37on that data, 241 of 456 off-diagonal rates sat at the1e-7floor and 21 at the0.25clamp, up to 6 orders of magnitude from the correct model. R builds the identical model (1.9e-15), with no error.Changes (
src/error_models.rs,check_binned_anchors)1e-7; the whole model when every segment is affectedNA, or stops)Alternative bin schemes still fit; the warning only says why they are suspect.
--errfun externalremains the path for deliberately custom models.docs/commands/learn-errors.mdexplains the mean-quality smear and the new error and warning.Verification
Real data, NovaSeq 16S (5 samples, filtered once, and the same files given to R and to dada2-rs):
2,11,25,37(correct, unobserved Q2)sum(trans))2,12,23,3711,24,37(extremes on anchors, so R's warning stays silent)No false alarm on MiSeq i100 (bins
12,24,38, 3 samples).Unit tests:
Mutation-checked: with the empty-anchor error disabled, both error tests fail.
Full suite:
cargo test --profile test-optpasses on default and--features wfa(465 passed); clippy is clean on both;mkdocs build --strictpasses.A correction to the issue
#263 was filed from a synthetic case where the bin columns hold all the mass, and it says the model "collapses to
min_error_rate". On real data, wrong anchors corrupt the model rather than zeroing it: about half the rates at the floor and some at the clamp. The issue text and PR #265's comments are corrected to say so.🤖 Generated with Claude Code