Real parity break, pre-existing, affects SVAR1 / VCF/BCF / PGEN identically. Found during the
#375 Track B review; not introduced by it.
With re-aligned tracks and a fixed output length, when L exceeds a deletion-shrunk haplotype's
natural length, StreamingDataset fills the track's trailing region with real signal read from
beyond the region's end, while a written Dataset zero-pads that tail. The two disagree
there, silently.
Root cause. The writer extends the stored chromEnd only as far as the last overlapping
variant's end (python/genvarloader/_dataset/_write.py:1143-1167) — never by max_del. Streaming
instead queries tracks out to t_ends_ext, the deletion-extended end
(_streaming.py:598-602 and :3635-3639, consumed at :1848-1855). So streaming has real data
where the oracle has zeros.
Minimal reproduction — no new fixture needed, uses the committed SVAR1 streaming_tracks_fixture
(20 bp regions; region 1's haplotype has natural length 18):
.with_len(19) → cell (1, 0): expected 0.0, got 12.0
.with_len(20) → 2 differing values
.with_len(<= 18) → matches
Record backends, same shape: with_len(78) → 2 differing values per cell; with_len(80) → 6, on
both VCF and PGEN. Forcing t_ends_ext = t_ends restores exact parity in every case.
Trigger condition is L > region_len + net_ilen. Existing fixed-length tests use small L
(test_record_mixed_fixed_length_parity at L=10, test_realign_false_with_len_matches_written
at L=12) and never reach the regime, which is why this was never caught.
Interaction worth recording. On the default ragged path the extension is unreachable: the
track_idx = out_idx - d invariant in shift_and_realign_track_core caps the last reference index
at length - d - 1, the region's last base, for any interval density. So t_ends_ext is dead code
on the path that works and live-but-wrong on the path that does not. Removing it is the leading
candidate fix — but confirm the written path's own tail behaviour is the one we want before
matching it.
When fixing, add a boundary test at L = natural_length + 1 for both the SVAR1 and the record
backends; that is the smallest case that discriminates.
Documented as a known gap in docs/source/dataset.md and skills/genvarloader/SKILL.md by
commit 646c5d40; remove those notes when this is fixed.
Relates to #375.
Real parity break, pre-existing, affects SVAR1 / VCF/BCF / PGEN identically. Found during the
#375 Track B review; not introduced by it.
With re-aligned tracks and a fixed output length, when
Lexceeds a deletion-shrunk haplotype'snatural length,
StreamingDatasetfills the track's trailing region with real signal read frombeyond the region's end, while a written
Datasetzero-pads that tail. The two disagreethere, silently.
Root cause. The writer extends the stored
chromEndonly as far as the last overlappingvariant's end (
python/genvarloader/_dataset/_write.py:1143-1167) — never bymax_del. Streaminginstead queries tracks out to
t_ends_ext, the deletion-extended end(
_streaming.py:598-602and:3635-3639, consumed at:1848-1855). So streaming has real datawhere the oracle has zeros.
Minimal reproduction — no new fixture needed, uses the committed SVAR1
streaming_tracks_fixture(20 bp regions; region 1's haplotype has natural length 18):
.with_len(19)→ cell (1, 0): expected0.0, got12.0.with_len(20)→ 2 differing values.with_len(<= 18)→ matchesRecord backends, same shape:
with_len(78)→ 2 differing values per cell;with_len(80)→ 6, onboth VCF and PGEN. Forcing
t_ends_ext = t_endsrestores exact parity in every case.Trigger condition is
L > region_len + net_ilen. Existing fixed-length tests use smallL(
test_record_mixed_fixed_length_parityatL=10,test_realign_false_with_len_matches_writtenat
L=12) and never reach the regime, which is why this was never caught.Interaction worth recording. On the default ragged path the extension is unreachable: the
track_idx = out_idx - dinvariant inshift_and_realign_track_corecaps the last reference indexat
length - d - 1, the region's last base, for any interval density. Sot_ends_extis dead codeon the path that works and live-but-wrong on the path that does not. Removing it is the leading
candidate fix — but confirm the written path's own tail behaviour is the one we want before
matching it.
When fixing, add a boundary test at
L = natural_length + 1for both the SVAR1 and the recordbackends; that is the smallest case that discriminates.
Documented as a known gap in
docs/source/dataset.mdandskills/genvarloader/SKILL.mdbycommit
646c5d40; remove those notes when this is fixed.Relates to #375.