Repository navigation
feat(mdrun): measured CPU, memory, and GPU use per job - #188
Merged
Merged
Conversation
The job script logs its container's cgroup CPU time and peak memory, plus nvidia-smi samples on GPU jobs only, every minute and on exit. MDRun reads the last line from the pod log when it closes the usage window and keeps it in mdrun_usage_measured; make usage, the exporter, and the dashboard compare it with the allocation. The exit trap runs the measurement with || true before the completion marker, because set -e would otherwise skip the marker the s3-sync sidecar waits for. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kubernetes 36.0.3 skips decoding str responses, so the pod log came
back as the repr of its bytes ("b'...\n...'") on a single line: no
mddash-usage line parsed, and stopped or finished jobs stored no
measurement. A raw read also never returns None, which crashed Stop
for a job whose pod never started.
Refs #169
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #169. Stacked on #187: the base is
feat/usage-metrics, so retarget it tomasteronce #187 merges. It adds measured usage next to allocation, as decided in the spec's new "Measured usage (phase 2b)" section.nvidia-smiutilization and memory are sampled every 60 s. The script logs amddash-usageline every 60 s and again when it exits.mdrun_usage_measuredtable.make usagegains CPU used % and GPU busy % per user, andJOBS=1lists each job with its measurements. The exporter addsmddash_mdrun_measured_*gauges, and the dashboard table gets matching columns.Worth knowing:
pods/log, and containers can read their own cgroup v2 counters (checked on dev:cgroup2fs, kernel 6.8)._preload_content=False), as the dashboard API's job-log reader already does. kubernetes 36.0.3 returns a preloaded text/plain body as the repr of its bytes ("b'...\n...'"on one line), so nomddash-usageline would ever parse.nvidia-smi, because a CPU job on a GPU node may see other jobs' GPUs.mddash_usage_final || truenow runs before the completion marker'stouch. Underset -e, a failing trap command skips the rest of the trap, and the s3-sync sidecar would then wait for a marker that never comes (checked in bash). Every measuring command tolerates failure, so measuring can't fail a job.ownerthey are a client claim.mdrun_usage_measuredis a new table, socreate_allcreates it and no migration is needed.How to test
make push-mdrun-api, thenmake deploy.make usage JOBS=1should show CPU used % and peak memory for both jobs, plus GPU busy % and GPU memory for the GPU job. WithCSV=1,measured_to_exitshould beyes.measured_to_exitset tono.Checked on dev with CPU-only GROMACS jobs (1 rank, 4 threads):
complete= 0). The Extend run that followed finished and stored its exit line (complete= 1).kubectl top(999m).make usage JOBS=1(24.6% and 23.6% CPU used), the exporter, and the Grafana panel's CPU used query (24.4% combined, evaluated in Prometheus) agree with the stored rows.mpirunbinds jobs with 1–2 ranks to a single core, so all 4 OpenMP threads shared one. That launch problem is outside this PR.Not yet tested: the GPU path, including whether
nvidia-smiis available inside the job images, because no A40 was free. I also ran the real generated GROMACS script locally with a fakempirun: it logged the final usage line after "Simulation completed." and still reached the markertouch.Checklist
mdrun_usage_measuredis a new table thatcreate_allcreates)Prepared with Claude Code (Claude Opus 5.5).