[skourta]

A failed charm run often comes from the host: an OOM kill, an AppArmor denial, an I/O error, or a snap refresh mid-run. The collect bundle had none of those events, and nothing tied a log line to the test that was running. This adds the last of the DA282 Phase 3 files.

Solution

• host-watch.sh follows dmesg -W into kernel.log and journalctl -u snapd into snapd.log. It runs as root, is skipped without passwordless sudo or with logs: false, and stop-watcher.sh stops it. Both start at start, and the kernel log is unfiltered.
• env-snapshot.sh writes env-start.json and env-end.json from snapd's REST API: kernel version, snaps and recent changes. It needs no root and gives null fields without snapd.
• New stop input test-log copies the job's tee'd output to pytest.log.
• New start input markers, default false, loads pytest/flake0_markers.py through PYTEST_PLUGINS and PYTHONPATH in $GITHUB_ENV. The plugin writes a start and a finish line per test to markers.ndjson. It is opt-in because a later step that replaces PYTHONPATH would make pytest fail at startup.
• stop.sh now redacts every *.log and markers.ndjson, and removes the __pycache__ that root's pytest leaves in the action directory.
• redact.sh applies listed values before the patterns, longest first, from a program file. Before, a pattern or a shorter value could split a listed secret, and + ? ( ) { } | in a value could mask the wrong text or stop all masking. A value list over 128 KB stopped sed from running.
• redact.sed now masks quoted JSON and Python dict values, JSON nested in a string, secret-key and URL credentials. A one-line PEM no longer deletes the rest of the file, and TestPassword::test_x and test_basic FAILED stay readable.

Testing

• pytest tests/: 94 passed. New files are test_markers.py, test_env_snapshot.py and test_start_stop.py, with more cases in test_redact.py.
• shellcheck 0.9.0 and ruff clean.
• Local run under sudo: a token written to /dev/kmsg reached kernel.log, and no follower survived stop-watcher.sh.
• Real tox with the consumer lines below: markers written with the variables set, none with them unset. Forwarding PYTHONPATH through pass_env alone makes pytest fail.
• Real pytest-rerunfailures 16.7 gives a pair per attempt. pytest-retry 1.7.0 gives one pair with failed.
• CI on 8465d3a passed: smoke-collect now covers the new files, and all 6 consumer-sim jobs passed. consumer-sim runs pytest as root inside spread and checks that the probe test's markers reach markers.ndjson, so the $(HOST: ...) forwarding works. It only checks that kernel.log exists, so dmesg -W is shown to start on the 24.04 and 26.04 runners, not to capture lines.
• Trial in OpenSearch ([DO NOT MERGE] flake0 Phase 3 trial opensearch-single-kernel-library#266, closed): the full integration suite with markers: true, the six consumer lines and redact-values fed from the job's secrets. In run 42178729841, 172 checks passed and 15 failed, none from flake0. Snap store downloads stalled at 0 bytes/s in 10 jobs (charmcraft, jhack, lxd, juju-db), and the rest were load timeouts, the 180-minute step limit, a secret-get permission error and the PR title check.
• All 162 bundles from that run had snapd.log and both env snapshots. kernel.log had lines in 156, and pytest.log (156) and markers.ndjson (153) were missing only where no test ran: 6 jobs timed out installing charmcraft and 3 failed in spread prepare. snapd.log named the stalled jhack download, and env-end.json showed charmcraft stuck at 16.7 of 81.9 MB. A pattern scan of every bundle found no unredacted secret, and the AWS, Azure and GCP export lines read ***REDACTED***.

Notes

• To get markers, a consumer sets markers: true and adds six lines. spread.yaml forwards FLAKE0_MARKERS, PYTEST_PLUGINS and PYTHONPATH in the github-ci backend. tox.ini adds FLAKE0_MARKERS and PYTEST_PLUGINS to pass_env, and appends :{env:PYTHONPATH:} to the existing set_env line, because set_env overrides pass_env. These go in together, or pytest fails at startup.
• env-*.json are not redacted, and logs: false does not turn them off. snapd's err and log fields can hold snap hook output. The README warns about this.
• $GITHUB_ENV beats a job-level env: PYTHONPATH, and a step-level env: beats $GITHUB_ENV (checked in a scratch workflow). Only a step that sets its own PYTHONPATH must keep the existing value, as the README says.

[reneradoi]

Minor comment on including the new watchers for the overhead metrics.