Skip to content

streaming: re-aligned tracks disagree with the written path when with_len(L) exceeds a deletion-shrunk haplotype's natural length #398

Description

@d-laub

Real parity break, pre-existing, affects SVAR1 / VCF/BCF / PGEN identically. Found during the
#375 Track B review; not introduced by it.

With re-aligned tracks and a fixed output length, when L exceeds a deletion-shrunk haplotype's
natural length, StreamingDataset fills the track's trailing region with real signal read from
beyond the region's end, while a written Dataset zero-pads that tail. The two disagree
there, silently.

Root cause. The writer extends the stored chromEnd only as far as the last overlapping
variant's end (python/genvarloader/_dataset/_write.py:1143-1167) — never by max_del. Streaming
instead queries tracks out to t_ends_ext, the deletion-extended end
(_streaming.py:598-602 and :3635-3639, consumed at :1848-1855). So streaming has real data
where the oracle has zeros.

Minimal reproduction — no new fixture needed, uses the committed SVAR1 streaming_tracks_fixture
(20 bp regions; region 1's haplotype has natural length 18):

  • .with_len(19) → cell (1, 0): expected 0.0, got 12.0
  • .with_len(20) → 2 differing values
  • .with_len(<= 18) → matches

Record backends, same shape: with_len(78) → 2 differing values per cell; with_len(80) → 6, on
both VCF and PGEN. Forcing t_ends_ext = t_ends restores exact parity in every case.

Trigger condition is L > region_len + net_ilen. Existing fixed-length tests use small L
(test_record_mixed_fixed_length_parity at L=10, test_realign_false_with_len_matches_written
at L=12) and never reach the regime, which is why this was never caught.

Interaction worth recording. On the default ragged path the extension is unreachable: the
track_idx = out_idx - d invariant in shift_and_realign_track_core caps the last reference index
at length - d - 1, the region's last base, for any interval density. So t_ends_ext is dead code
on the path that works and live-but-wrong on the path that does not. Removing it is the leading
candidate fix
— but confirm the written path's own tail behaviour is the one we want before
matching it.

When fixing, add a boundary test at L = natural_length + 1 for both the SVAR1 and the record
backends; that is the smallest case that discriminates.

Documented as a known gap in docs/source/dataset.md and skills/genvarloader/SKILL.md by
commit 646c5d40; remove those notes when this is fixed.

Relates to #375.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions