[5/5] Perf: index the fragment rows instead of the dense slots - #438
Merged
Merged
Conversation
flatten_fragments had no unit test. Add black-box tests that pin its contract, so the refactoring that follows can be checked step by step. `_dense_library` builds dense frames with mz == 0 padding from both sources that produce it: unmodified precursors carry no modloss fragments, and some fragments fall outside the mz range. Intensities are distinct, so the top-k selection is unambiguous. `_expected_keep_mask` derives the expected mask per precursor, without flatten_fragments. The tests cover the kept fragments over five combinations of `keep_top_k_fragments` and `min_fragment_intensity`, every annotation column, the reannotated precursor pointers, `custom_df` and `custom_columns`, the path without intensities, an empty library, and a precursor long enough to need more than a uint8 fragment number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`parse_fragment`, `fill_in_indices` and `calculate_fragment_numbers` only serve `flatten_fragments`, and nothing in alphabase, alphadia or peptdeep calls them. A leading underscore states that, and frees their signatures for the refactoring that follows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tion Move two concerns out of the body of `flatten_fragments`: the per-column annotation of the charged fragment types, and the reannotation of the precursor pointers. The function now reads as its steps, and the memory refactoring that follows changes a named helper instead of a long function body. `_annotate_charged_frag_types` returns typed arrays rather than four parallel lists of Python ints. Tiling a typed array keeps its dtype, so the dense type/loss_type/charge/direction arrays no longer pass through an int64 intermediate of 8 bytes per dense slot. No functional change. Output stays bit-identical to main across 5400 generated configurations of library shape, padding, top-k, intensity threshold and custom_columns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Building the flat fragment dataframe materialised every column at the full dense length of n_fragment_rows * n_fragment_types, assembled them into a dataframe and then copied the retained subset out of it. Reannotating the precursor pointers allocated an int64 cumulative sum over all dense slots on top of that. Only mz and intensity are needed at dense length, because they give the keep mask. `_select_dense_fragments` accumulates that mask in place and returns the ascending indices of the kept slots. `_annotate_kept_fragments` then reads the type, the loss type and the charge straight off those indices, because the index of a slot gives its dense column. The dataframe is built from the kept values alone, so the filtering copy is gone. `_reannotate_precursor_pointers` finds a new pointer with a binary search on the kept indices, which is equivalent to subtracting the cumulative number of removed fragments. Output stays bit-identical to main across 5400 generated configurations of library shape, padding, top-k, intensity threshold and custom_columns: same fragment count, column order, dtypes, values and precursor pointers. At 48M dense slots, peak RSS drops from 3.49 GB to 1.77 GB and the call is ~27% faster. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_parse_fragment` held the fragment number, the position and the precursor length of every dense slot as uint32, which cost 3 dense arrays of 4 bytes per slot. All three values are shared by every charged fragment type of a fragment row, so `_fill_in_indices` now writes one value per fragment row instead, and `_parse_fragment` no longer takes the tiled directions array. uint16 holds every value, because `max_frag_per_peptide` of `_fill_in_indices` bounds the row positions and the row counts at 300. `flatten_fragments` casts the kept `number` and `position` values back to uint32, so the flat columns keep their public dtype. `_calculate_fragment_numbers` takes a direction, a row position and a row count, and runs for the kept fragments only. The dense pass over every slot is gone, and so is the discarded output array it needed. Dropping its in-out `frag_number` parameter also gives direction 0 a defined result. `_fill_in_indices` writes the row count as a scalar instead of a product with an array of ones, which removes one allocation per precursor. `flatten_fragments` passes the intensities without a copy, because nothing writes to them any more. Zeroing the intensity of padding slots was a no-op, as padding is always excluded. Output stays bit-identical to main across 5400 generated configurations, with one exception: fragment rows that no precursor covers now report number 0 and position 0. main leaves those slots unwritten, because it allocates the arrays with `np.empty`, so their values were unspecified. At 48M dense slots, peak RSS drops from 1.78 GB to 1.32 GB, and from 3.47 GB on main. The call is ~30% faster than the previous step and ~45% faster than main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GeorgWa
changed the title
mschwoer
reviewed
Sep 1, 2026
Comment on lines
+957
to
+958
| row_positions = np.zeros(n_fragment_rows, dtype=np.uint16) | ||
| row_counts = np.zeros(n_fragment_rows, dtype=np.uint16) |
Contributor
There was a problem hiding this comment.
we intantiate here, then pass to _fill_in_indices .. why not instantiate there and return?
Collaborator
Author
There was a problem hiding this comment.
implemented it, nicer!
| ---------- | ||
| frag_direction : np.int8 | ||
| directions of fragments for each peptide | ||
| direction of the fragment type. 'abc' ions count from the first amino |
Contributor
There was a problem hiding this comment.
could there be string constants for 'abc' and 'xyz' ?
Collaborator
Author
There was a problem hiding this comment.
I think this is beyiond the scope of the PR
| return row_position + 1 | ||
| if frag_direction == -1: | ||
| return row_count - row_position | ||
| return 0 |
Contributor
There was a problem hiding this comment.
please add this behaviour to the docstring
| ) | ||
| needs_row = bool({"number", "position"}.intersection(custom_columns)) | ||
| kept_frag_types = kept_indices % n_fragment_types if needs_frag_type else None | ||
| kept_rows = kept_indices // n_fragment_types if needs_row else None |
Contributor
There was a problem hiding this comment.
please double check if a +1 is required due to flooring
| ) | ||
| if "position" in custom_columns: | ||
| columns["position"] = positions.reshape(-1)[kept_indices] | ||
| # the flat column keeps its uint32 dtype |
Contributor
There was a problem hiding this comment.
what extra information (compared to the next line) does this comment carry?
_expected_keep_mask reimplemented the filtering logic, so it could agree with a broken flatten_fragments. Add a minimal_library fixture of two precursors over a forward and a reverse fragment type, with the kept slots written out per filter setting, and assert both the helper and flatten_fragments against it. Derive the long precursor row count from the uint8 boundary it tests instead of picking a number near the fill_in_indices limit, and guard the limit explicitly so exceeding it fails on an assertion rather than on a broadcast error inside numba. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"fragment number" read as ambiguous between a count and an enumeration. Spell out that it is the MS series numbering, and contrast it with the fragment position that all series share. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The library was generated from a seeded rng and the expectations came from _expected_keep_mask, which reimplemented the filtering logic and so could agree with a broken flatten_fragments. Replace both with one library fixture over hand-written mz and intensity grids, and hand-written keep masks per filter setting. Two fragment types carry the whole annotation surface: y_z1 is reverse without loss and b_modloss_z1 is forward with loss, so direction, series and loss all vary over two columns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comparing column by column left the column set, the column order and the dtypes unchecked. Assert the whole annotated frame against an expected frame instead, and fold the row count of the filter cases into a frame comparison of mz and intensity. type and charge come out as int8, so the expected frame spells that out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three tile calls differed only in column name and source array, and the comment above them explained the dtype of the previous implementation rather than the code it sat on. Loop over the columns instead; the dtypes stay visible where they are set, in _annotate_charged_frag_types. Trim the two comments carried into the extracted helpers. The direction comment claimed a third value of 0 that DIRECTION_MAPPING never produces. Column order of the flat fragment dataframe is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
They free nothing. Peak numpy allocation measured with tracemalloc is 22.00 B/slot with all three dels, with one, and with none, at both 20.8M and 62.4M dense slots. The high water mark sits inside _parse_fragment, before any del in the caller runs: mz 4 + intensity 4 + directions 1 + numbers 4 + indices 4 + max_indices 4 + excluded 1. Every del frees memory that is already past that peak. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Match the bracket notation already used for the writes in the same functions, for the four column reads this branch introduced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ragment-parsing-helpers
The merged tests named fill_in_indices, which this branch renamed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e-charged-frag-types
…ns-from-kept-indices Two conflicts, both where this branch had already deleted the code the other branch edited: the cumsum in _reannotate_precursor_pointers, which searchsorted replaced, and the dense annotation tiling in flatten_fragments, which _annotate_kept_fragments replaced. Kept this branch's version of both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gment-positions Three conflicts: - _parse_fragment docstring. This branch no longer returns fragment numbers or positions from it, so the numbering paragraph does not apply here. It survives in _calculate_fragment_numbers, which does. - the _parse_fragment call. Kept the per-row call of this branch and carried the [] column access from the other side. - del not_top_k. Dropped, it frees memory past the peak. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_parse_fragment only allocated three arrays and passed them to _fill_in_indices to be filled. Allocating them inside and returning them leaves nothing for the wrapper to do, so the two become one njit function: 9 parameters drop to 7, and the allocation moves into compiled code instead of the interpreter. Output is bit identical across 324 configurations of library shape, padding, top-k, intensity threshold and missing intensities. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Spell out the three branches next to the numbering they implement. The Returns section claimed 0 for direction 0, but the fall through gives 0 for every direction other than 1 and -1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace the remaining column comparisons. Each one now also pins the column set, the column order and the dtypes: - without_intensity subsumes its separate missing-column assertion - selects_custom_columns checked only the column names - filters_custom_df_columns now pins where cardinality lands - reannotates_precursor_pointers compares hardcoded flat pointers instead of reslicing mz per precursor, which was the last place the tests recomputed an expectation - empty_precursor_df compares both frames instead of their lengths Five tests assert against the same unfiltered library, so it moves into a flat_unfiltered fixture. test_flatten_fragments_long_precursor keeps its property assertions. It builds 257 rows to push fragment numbers past 255, so a frame of 1542 generated values would bury what it tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ragment-parsing-helpers
…e-charged-frag-types
…ns-from-kept-indices
The four scalar assertions become two equality comparisons. The dtypes need their own comparison, because .max() over columns of different dtypes upcasts to a common one and would hide a narrowed column. Also drop the tuple rebuild in the parse test, expected is already the tuple being compared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ragment-parsing-helpers
…e-charged-frag-types
…ns-from-kept-indices
…gment-positions One conflict where the reworked long precursor assertions met the two helper tests this branch appends after them. Kept both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
np.testing.assert_array_equal takes strict=True, which compares dtype and shape alongside the values. The separate dtype assertions and the length check of not_top_k fold into the comparisons they belong to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Main carries PRs #434 to #436, so the three branches below this one arrive from main and only fragment.py still differs. Six conflicts, all where this branch had rewritten what main still has: the dense annotation tiling, the cumsum in _reannotate_precursor_pointers that searchsorted replaces, and the helper signatures around them. Kept this branch's version of each, and main's tuple annotation, since main dropped Python 3.8 and the typing imports with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gment-positions Brings main and PRs #434 to #436 in behind it. Two conflicts, both where the annotation of a function this branch reworked met main's builtin generics. Kept this branch's signature and docstring with the builtin tuple, since main dropped Python 3.8 and the typing imports with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mschwoer
approved these changes
Sep 15, 2026
Base automatically changed from
perf/build-flat-columns-from-kept-indices
to
main
September 15, 2026 15:19
Main carries PR #437, so only the per-row rework still differs. Sixteen conflicts, all where this branch replaced the per-dense-slot arrays that main now has. Kept the per-row version of each. Reword the two comments mschwoer asked about: - the position cast said what the line already says. It now names the reason for the cast, that row_positions is the narrower uint16. - the copy=False comment said a copy is kept "out of memory". It now says a view is returned instead of a second dense array being allocated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance. One documented behaviour change, otherwise bit-identical to main.
parse_fragmentheld the number, the position and the precursor length of every dense slot.max_frag_per_peptidebounds.Behaviour change: fragment rows that no precursor covers now report number 0 and position 0.
mainleaves those slots unwritten (np.empty), so their values were unspecified.Peak RSS and runtime of the whole stack,
keep_top_k_fragments=12:Last of 5. Stacked on #437, and lands the same end state as #429.
🤖 Generated with Claude Code