Skip to content

Stop the DeepLC workers filling the log with blank lines - #84

Merged
RobbinBouwmeester merged 2 commits into
mainfrom
fix/deeplc-blank-log-lines
Sep 14, 2026
Merged

Stop the DeepLC workers filling the log with blank lines#84
RobbinBouwmeester merged 2 commits into
mainfrom
fix/deeplc-blank-log-lines

Conversation

@RobbinBouwmeester

Copy link
Copy Markdown
Member

Reported from the desktop interface: predicting retention times produces a run log that is almost entirely empty lines.

Cause

DeepLC's progress writer emits a bare \r per update, which renders as nothing when stdout is not a terminal — so each update arrives as an empty line. The engine inherits a sidecar's stdout rather than capturing it (sidecar.rs::run_worker, deliberately, so hours-long progress reaches the user live), so those lines land in the terminal, in any redirected log file, and in the desktop run log, pushing the real messages out of view.

Measured before the fix:

log total lines blank
2.9M-peptide library prediction 5,743 5,697 (99%)
entrapment A/B, baseline arm 5,810 5,722 (98%)
entrapment A/B, multi-head arm 5,824 5,734 (98%)

Not new, but it now affects every run that predicts retention times — which, since multi-head calibration became the default, is every run with a DeepLC interpreter.

No upstream switch exists

deeplc.predict takes a predict_kwargs passthrough, but neither enable_progress_bar nor verbose reaches the writer — both raise TypeError. So the filtering has to happen in the workers.

The fix

Both DeepLC workers wrap their prediction in quiet_deeplc_progress(), which replaces sys.stdout with a proxy that drops only segments that are empty once carriage returns and whitespace are stripped.

  • anything DeepLC actually says still comes through, in order;
  • the workers' own per-chunk progress is unaffected;
  • stderr is not touched at all, so a traceback is never swallowed;
  • \r counts as a terminator as well as \n, so a writer that never emits a newline cannot accumulate in the buffer for a whole run;
  • MUMDIA_DEEPLC_RAW_OUTPUT=1 restores the unfiltered output for debugging the worker itself.

Verified on the real workers

before after
deeplc_worker.py, 6,000 peptides 13 lines, 12 blank 1 line, 0 blank
deeplc_finetune.py, re-prediction mode 8 lines, 0 blank, every message intact

The surviving output is exactly what you want to see:

no-finetune: predicting with the DeepLC base model (seed ignored)
predicting 3 unique standard peptidoforms with the DeepLC base model (torch threads=8)
  3/3  0.0s for this chunk (76 peptidoforms/s, ETA 0.0 min)
prediction phase: 0.0s total
wrote library with re-predicted iRT (the DeepLC base model): ...
  rows=3000 repredicted=3000 retained_imported=0 (...)

Tests

Three: both workers route prediction through the filter, the escape hatch is present, and the proxy keeps content and ordering while dropping blanks. pytest tests/python — 83 passed in the DeepLC environment, and the new static checks also pass on a bare interpreter with no ML dependencies.

The filter is duplicated in the two workers rather than shared, following the precedent _check_deeplc_version already sets for the same reason: the sidecars are standalone files invoked by path.

🤖 Generated with Claude Code

RobbinBouwmeester and others added 2 commits September 13, 2026 12:08
Reported from the desktop interface: predicting retention times produces a run
log that is almost entirely empty lines.

Cause: DeepLC's progress writer emits a bare `\r` per update, which renders as
nothing when stdout is not a terminal, so each update arrives as an empty line.
The engine INHERITS a sidecar's stdout rather than capturing it
(`sidecar.rs::run_worker`, on purpose, so hours-long progress reaches the user
live), so those lines land in the terminal, in any redirected log file, and in
the desktop application's run log, pushing the real messages out of view.

Measured before the fix: 5,697 blank lines from a single 2.9M-peptide prediction
(99% of that log), and 98% of both arms of the entrapment A/B. It is not new, but
it affects every run that predicts retention times, which since multi-head
calibration became the default is every run with a DeepLC interpreter.

There is no upstream switch: `deeplc.predict` takes a `predict_kwargs`
passthrough, but neither `enable_progress_bar` nor `verbose` reaches the writer
(both raise TypeError), so the filtering has to happen in the workers.

Both DeepLC workers now wrap their prediction in `quiet_deeplc_progress()`, which
replaces `sys.stdout` with a proxy that drops only segments that are empty once
carriage returns and whitespace are stripped. Anything DeepLC actually says still
comes through, in order; the workers' own per-chunk progress is unaffected;
stderr, where a traceback goes, is not touched at all. `\r` counts as a
terminator as well as `\n`, so a writer that never emits a newline cannot
accumulate in the buffer for a whole run. `MUMDIA_DEEPLC_RAW_OUTPUT=1` disables
the filter for debugging the worker itself.

Verified on the real workers: `deeplc_worker.py` over 6,000 peptides went from 13
lines (12 blank) to 1, and `deeplc_finetune.py` in its base-model re-prediction
mode emits 8 lines and 0 blanks, with every message intact including the
per-chunk progress. Three tests cover it: both workers route prediction through
the filter, the escape hatch is present, and the proxy keeps content and ordering
while dropping blanks.

The filter is duplicated in the two workers rather than shared, following the
precedent already set by `_check_deeplc_version`, which each carries its own copy
of for the same reason: the sidecars are standalone files invoked by path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ci/gen_config_reference.py` scans tracked scripts for environment reads, so the
new escape hatch became visible to it only once the worker change was committed:
59 variables read, 46 sidecar-side. Same step the peptdeep device variable needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@RobbinBouwmeester
RobbinBouwmeester merged commit 2389170 into main Sep 14, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant