Skip to content

feat(mdrun): measured CPU, memory, and GPU use per job - #188

Merged
andrejnespor merged 2 commits into
feat/usage-metricsfrom
feat/usage-measured
Nov 5, 2026
Merged

andrejnespor merged 2 commits into
feat/usage-metricsfrom
feat/usage-measured

Conversation

@andrejnespor

@andrejnespor andrejnespor commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Refs #169. Stacked on #187: the base is feat/usage-metrics, so retarget it to master once #187 merges. It adds measured usage next to allocation, as decided in the spec's new "Measured usage (phase 2b)" section.

  • The GROMACS and AMBER job scripts measure their own container. CPU time and peak memory come from its cgroup v2 counters. On GPU jobs, nvidia-smi utilization and memory are sampled every 60 s. The script logs a mddash-usage line every 60 s and again when it exits.
  • When MDRun closes the usage window, it reads the last such line from the pod log before deleting the Job, and stores it in the new mdrun_usage_measured table.
  • make usage gains CPU used % and GPU busy % per user, and JOBS=1 lists each job with its measurements. The exporter adds mddash_mdrun_measured_* gauges, and the dashboard table gets matching columns.

Worth knowing:

  • It needs no new permissions. The hub role already grants pods/log, and containers can read their own cgroup v2 counters (checked on dev: cgroup2fs, kernel 6.8).
  • MDRun reads the pod log raw (_preload_content=False), as the dashboard API's job-log reader already does. kubernetes 36.0.3 returns a preloaded text/plain body as the repr of its bytes ("b'...\n...'" on one line), so no mddash-usage line would ever parse.
  • CPU-only jobs never call nvidia-smi, because a CPU job on a GPU node may see other jobs' GPUs.
  • The exit trap changed. mddash_usage_final || true now runs before the completion marker's touch. Under set -e, a failing trap command skips the rest of the trap, and the s3-sync sidecar would then wait for a marker that never comes (checked in bash). Every measuring command tolerates failure, so measuring can't fail a job.
  • A stopped or deleted job keeps its last periodic reading, at most 60 s old, because the pod is gone before its exit line.
  • Efficiency covers measured jobs that ended in the period. Each is weighed against its whole window, which includes the image pull and S3 download that the allocation also covers.
  • Measurements come from the job container, so like owner they are a client claim.
  • mdrun_usage_measured is a new table, so create_all creates it and no migration is needed.

How to test

  1. Deploy feat: per-user allocated and measured usage in Prometheus and Grafana #187 and this branch to dev: make push-mdrun-api, then make deploy.
  2. Let a short CPU-only GROMACS job and a GPU job finish.
  3. make usage JOBS=1 should show CPU used % and peak memory for both jobs, plus GPU busy % and GPU memory for the GPU job. With CSV=1, measured_to_exit should be yes.
  4. Stop a running job. Its row should keep the last periodic reading, with measured_to_exit set to no.
  5. In Grafana, the usage table should show CPU used % and GPU busy % for the range.

Checked on dev with CPU-only GROMACS jobs (1 rank, 4 threads):

  • A run stopped mid-way stored its last periodic reading (complete = 0). The Extend run that followed finished and stored its exit line (complete = 1).
  • The cgroup counter matched kubectl top (999m). make usage JOBS=1 (24.6% and 23.6% CPU used), the exporter, and the Grafana panel's CPU used query (24.4% combined, evaluated in Prometheus) agree with the stored rows.
  • The low figure is real: mpirun binds jobs with 1–2 ranks to a single core, so all 4 OpenMP threads shared one. That launch problem is outside this PR.

Not yet tested: the GPU path, including whether nvidia-smi is available inside the job images, because no A40 was free. I also ran the real generated GROMACS script locally with a fake mpirun: it logged the final usage line after "Simulation completed." and still reached the marker touch.

Checklist

  • PR title and summary are descriptive.
  • Max one DB migration per PR. (none: mdrun_usage_measured is a new table that create_all creates)
  • Docs updated.
  • Tests included.
  • I have seen this code, I have run this code, and I take responsibility for this code.

Prepared with Claude Code (Claude Opus 5.5).

andrejnespor and others added 2 commits November 4, 2026 14:28
The job script logs its container's cgroup CPU time and peak memory,
plus nvidia-smi samples on GPU jobs only, every minute and on exit.
MDRun reads the last line from the pod log when it closes the usage
window and keeps it in mdrun_usage_measured; make usage, the exporter,
and the dashboard compare it with the allocation. The exit trap runs
the measurement with || true before the completion marker, because
set -e would otherwise skip the marker the s3-sync sidecar waits for.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kubernetes 36.0.3 skips decoding str responses, so the pod log came
back as the repr of its bytes ("b'...\n...'") on a single line: no
mddash-usage line parsed, and stopped or finished jobs stored no
measurement. A raw read also never returns None, which crashed Stop
for a job whose pod never started.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andrejnespor
andrejnespor merged commit 3672079 into feat/usage-metrics Nov 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant