Stop the DeepLC workers filling the log with blank lines - #84
Merged
Conversation
Reported from the desktop interface: predicting retention times produces a run log that is almost entirely empty lines. Cause: DeepLC's progress writer emits a bare `\r` per update, which renders as nothing when stdout is not a terminal, so each update arrives as an empty line. The engine INHERITS a sidecar's stdout rather than capturing it (`sidecar.rs::run_worker`, on purpose, so hours-long progress reaches the user live), so those lines land in the terminal, in any redirected log file, and in the desktop application's run log, pushing the real messages out of view. Measured before the fix: 5,697 blank lines from a single 2.9M-peptide prediction (99% of that log), and 98% of both arms of the entrapment A/B. It is not new, but it affects every run that predicts retention times, which since multi-head calibration became the default is every run with a DeepLC interpreter. There is no upstream switch: `deeplc.predict` takes a `predict_kwargs` passthrough, but neither `enable_progress_bar` nor `verbose` reaches the writer (both raise TypeError), so the filtering has to happen in the workers. Both DeepLC workers now wrap their prediction in `quiet_deeplc_progress()`, which replaces `sys.stdout` with a proxy that drops only segments that are empty once carriage returns and whitespace are stripped. Anything DeepLC actually says still comes through, in order; the workers' own per-chunk progress is unaffected; stderr, where a traceback goes, is not touched at all. `\r` counts as a terminator as well as `\n`, so a writer that never emits a newline cannot accumulate in the buffer for a whole run. `MUMDIA_DEEPLC_RAW_OUTPUT=1` disables the filter for debugging the worker itself. Verified on the real workers: `deeplc_worker.py` over 6,000 peptides went from 13 lines (12 blank) to 1, and `deeplc_finetune.py` in its base-model re-prediction mode emits 8 lines and 0 blanks, with every message intact including the per-chunk progress. Three tests cover it: both workers route prediction through the filter, the escape hatch is present, and the proxy keeps content and ordering while dropping blanks. The filter is duplicated in the two workers rather than shared, following the precedent already set by `_check_deeplc_version`, which each carries its own copy of for the same reason: the sidecars are standalone files invoked by path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ci/gen_config_reference.py` scans tracked scripts for environment reads, so the new escape hatch became visible to it only once the worker change was committed: 59 variables read, 46 sidecar-side. Same step the peptdeep device variable needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reported from the desktop interface: predicting retention times produces a run log that is almost entirely empty lines.
Cause
DeepLC's progress writer emits a bare
\rper update, which renders as nothing when stdout is not a terminal — so each update arrives as an empty line. The engine inherits a sidecar's stdout rather than capturing it (sidecar.rs::run_worker, deliberately, so hours-long progress reaches the user live), so those lines land in the terminal, in any redirected log file, and in the desktop run log, pushing the real messages out of view.Measured before the fix:
Not new, but it now affects every run that predicts retention times — which, since multi-head calibration became the default, is every run with a DeepLC interpreter.
No upstream switch exists
deeplc.predicttakes apredict_kwargspassthrough, but neitherenable_progress_barnorverbosereaches the writer — both raiseTypeError. So the filtering has to happen in the workers.The fix
Both DeepLC workers wrap their prediction in
quiet_deeplc_progress(), which replacessys.stdoutwith a proxy that drops only segments that are empty once carriage returns and whitespace are stripped.\rcounts as a terminator as well as\n, so a writer that never emits a newline cannot accumulate in the buffer for a whole run;MUMDIA_DEEPLC_RAW_OUTPUT=1restores the unfiltered output for debugging the worker itself.Verified on the real workers
deeplc_worker.py, 6,000 peptidesdeeplc_finetune.py, re-prediction modeThe surviving output is exactly what you want to see:
Tests
Three: both workers route prediction through the filter, the escape hatch is present, and the proxy keeps content and ordering while dropping blanks.
pytest tests/python— 83 passed in the DeepLC environment, and the new static checks also pass on a bare interpreter with no ML dependencies.The filter is duplicated in the two workers rather than shared, following the precedent
_check_deeplc_versionalready sets for the same reason: the sidecars are standalone files invoked by path.🤖 Generated with Claude Code