Repository navigation
feat: per-user allocated and measured usage in Prometheus and Grafana - #187
Merged
Merged
Conversation
With METRICS_PORT set, the poller serves Prometheus series recomputed from mdrun_usage at each scrape. Cumulative usage is a gauge: a window closes at the container's exit, which can come after a scrape already counted the job as running, and Prometheus would read the step back as a counter reset. The report and the exporter share the window rules. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
monitoring.enabled (on for dev) deploys a server-only Prometheus that scrapes the MDRun ledger metrics and a stateless Grafana with the "Usage by user" dashboard, reached through make grafana. The series carry usernames, so NetworkPolicies limit the metrics port to Prometheus, Prometheus to Grafana, and Grafana to port-forward, and neither pod mounts the hub namespace's service-account token. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The job script logs its container's cgroup CPU time and peak memory, plus nvidia-smi samples on GPU jobs only, every minute and on exit. MDRun reads the last line from the pod log when it closes the usage window and keeps it in mdrun_usage_measured; make usage, the exporter, and the dashboard compare it with the allocation. The exit trap runs the measurement with || true before the completion marker, because set -e would otherwise skip the marker the s3-sync sidecar waits for. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 5 tasks
kubernetes 36.0.3 skips decoding str responses, so the pod log came
back as the repr of its bytes ("b'...\n...'") on a single line: no
mddash-usage line parsed, and stopped or finished jobs stored no
measurement. A raw read also never returns None, which crashed Stop
for a job whose pod never started.
Refs #169
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each mddash-usage line also logs the container's clock, CPU throttling, OOM kills, pressure-stall totals, disk bytes, and GPU power; the exit line adds the engine's ns/day. Queue wait and waiting jobs come from the ledger. The new mdrun_usage_measured columns ship with the table, but dev already has it and needs ALTER TABLE: create_all never alters an existing table. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prometheus scrapes the hub's /hub/metrics as a JupyterHub service with only the read:metrics scope. z2jh generates its token and keeps it in the hub Secret, and the Prometheus pod projects just that key: the Secret also holds the hub's cookie secret and crypt keys. A Platform dashboard shows active users, running servers, spawn time and outcomes, and hub API latency and errors. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The target echoed its kubectl command and passed through a "Handling connection" line for every browser request, with no link to open. It now waits for the tunnel, prints a link per provisioned dashboard, and ends cleanly on Ctrl+C. A config with monitoring off gets a one-line reason instead of a kubectl error. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments and agent docs keep only durable context, two lines at most; behavior the code or a HELP string already shows is dropped. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stop, DELETE, and status reads take the job's usage measurement from its pod log before deleting the Job. A broken or hung log stream raised a urllib3 error through them, and a count past SQLite's INTEGER (oom_kills=1e30) failed the insert on every retry, so that job could never be stopped. The read now gives up after 10 s and treats any failure as no measurement. The parser keeps only numbers in range and warns about the rest, naming the job, and only the newest usage line is read. A test runs the real job script through finish, failure, and stop. Refs #169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One test per behavior that matters: the job script, reading and parsing the measurement, storing it, the report, and the exporter. Variant and formatting tests go, including the fake-mpirun exit-path runs. Refs #169
The kubelet joins a usage line onto unterminated output before it, such as `mdrun -v` progress, so a stop stored a stale reading or none. The reader now finds the prefix anywhere in a line. An Extend's appended log still holds the previous run's Performance line, so a run that died early logged that speed. The exit line reads ns/day only after a clean exit. It also ends the sampler's sleep, which outlived the script. The log read times out after 2 s: urllib3 retries it three times, and the 10 s it had could hold a stop for 40 s, past the dashboard's 10 s. One pod lookup now feeds the window and the measurement. Tests cover the engines' ns/day extraction and that CPU jobs never sample GPUs again. Refs #169 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
GPU utilization is sampled while the simulation container runs, but busy seconds applied it to the whole window, so image pull, S3 download, and upload counted as busy. A measurement without the container's clock keeps the whole window. The report and exporter tests for queue wait, pending jobs, CPU stall, OOM kills, and GPU energy are back: nothing else asserted them as anything but zero. Refs #169 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A range that started in a scrape gap found no sample to subtract and showed all-time totals. The table now takes the last sample in the hour before the range start. Its description says that Jobs counts jobs started in the range, unlike `make usage`. The hub errors panel also counts a new series' first errors, which increase() cannot see. retentionSize (1500MB of 2Gi) is a backstop: dev grows about 2 MiB a day at 2,000 series, and time retention alone does not bound the TSDB. Refs #169 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Refs #169 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #169. Phases 2, 2b, and 2c of the usage-monitoring design, plus the hub's own metrics, (
docs/specs/2026-10-23-usage-monitoring-design.md, updated with the decisions below). Tuner and user-namespace usage are later phases, so the issue stays open. This PR now also contains the former stacked #188.Allocated usage in Prometheus and Grafana (phase 2)
METRICS_PORTset, the MDRun poller serves per-user series computed from themdrun_usageledger at each scrape: cumulative jobs, CPU core-seconds, GPU-seconds, and memory GiB-seconds, plus the cores, GPUs, and memory held right now. The labels areuser,namespace, andengine.monitoring.enableddeploys a Prometheus server and a stateless Grafana into the hub namespace. It is on inconfig.dev.yamland off for prod and edc. Grafana ships a provisioned "Usage by user" dashboard. There is no ingress:make grafanaforwards Grafana and prints a link to each dashboard.kubectl port-forward.Worth knowing:
x - x offset d, neverrate()orincrease(). Grafana's range totals matchmake usageexcept at range boundaries (off by at most MDRun's detection delay) and for ranges that start before Prometheus' first sample, which show all-time totals. The table's Jobs column counts jobs that started in the range, wheremake usagecounts every job that ran in it. The start-of-range value is the last sample in the hour before it, which bridges a scrape gap there.helm upgradewithout a restart. The chart's Kubernetes discovery jobs are disabled, because they need cluster RBAC.grafana-community, because thegrafana/helm-chartscopy is deprecated.csi-ceph-rbd-duvolume, with a 1500MB size cap as a backstop. Dev holds about 2,000 series, mostly the hub's and Prometheus' own, and grows about 2 MiB a day. On dev the stack adds 110m CPU and 416Mi memory to the hub requests, andscripts/resource_summary.pycounts them.Measured usage per job (phase 2b)
nvidia-smiutilization and memory are sampled every 60 s. The script logs amddash-usageline every 60 s and again when it exits.mdrun_usage_measuredtable.make usagegains CPU used % and GPU busy % per user, andJOBS=1lists each job with its measurements. The exporter addsmddash_mdrun_measured_*gauges, and the dashboard table gets matching columns.Worth knowing:
pods/log, and containers can read their own cgroup v2 counters (checked on dev:cgroup2fs, kernel 6.8)._preload_content=False), as the dashboard API's job-log reader already does. kubernetes 36.0.3 returns a preloaded text/plain body as the repr of its bytes ("b'...\n...'"on one line), so nomddash-usageline would ever parse.mdrun -vprogress. The parser keeps non-negative numbers that SQLite can store (an oversized count would fail the insert, and with it the stop) and logs the rest with the job's name.nvidia-smi, because a CPU job on a GPU node may see other jobs' GPUs.mddash_usage_final || truenow runs before the completion marker'stouch. Underset -e, a failing trap command skips the rest of the trap, and the s3-sync sidecar would then wait for a marker that never comes (checked in bash). Every measuring command tolerates failure, so measuring can't fail a job.ownerthey are a client claim.mdrun_usage_measuredis a new table, socreate_allcreates it and no migration is needed.Job diagnostics (phase 2c)
mddash-usageline also carries the container's clock, CPU throttling, OOM kills, pressure-stall totals for CPU, memory, and I/O, disk bytes, and, on GPU jobs, power draw. The exit line adds the engine's own ns/day.make usageadds CPU stall %, OOM kills, GPU kWh, and average wait per user.JOBS=1adds wait, ns/day, CPU stall %, and OOM, and its CSV has every diagnostic. The exporter adds pending and queue-wait gauges and measured stall, OOM, and energy totals. The dashboard table gets four columns and a "Jobs waiting to start" panel.Worth knowing:
mdrun_usage_measuredgains 11 nullable columns. The table ships with this PR, so prod creates it complete. Dev already had it, andcreate_allnever alters a table, so dev got the columns throughALTER TABLE(rows kept).JupyterHub platform metrics
/hub/metricsas a JupyterHub serviceprometheuswith only theread:metricsscope. z2jh generates its token and keeps it in the hub Secret. The Prometheus pod projects only that key, because the Secret also holds the hub's cookie secret and crypt keys.increase()alone cannot see.How to test
make push-mdrun-api, thenmake deploy.kubectl -n mddash-dev get podsshould showprometheuswith 2/2 containers ready,grafanawith 1/1, andmdrun-apirestarted.make usage JOBS=1should show CPU used %, peak memory, wait, CPU stall %, and OOM for the jobs, ns/day for the finished ones, and GPU busy % and GPU memory for the GPU job. WithCSV=1,measured_to_exitshould beyesfor the finished jobs andnofor the stopped one, next to startup, throttling, memory and I/O stall, disk GiB, and GPU power.make grafanaand open its Usage by user link. A running job should appear in the "held" panels within a minute while its dashboard page is open, and a job waiting for a GPU in "Jobs waiting to start". For a range that starts after Prometheus' first sample (00:00 UTC of a later day until now), the table should matchmake usage SINCE=<that day>, including the measured columns. Jobs can differ when a job spans the range start.kubectl -n mddash-dev port-forward svc/prometheus 9090:80. Under Status, then Targets,mdrun-apiandjupyterhubshould be up.mdrun-api:80should answer, whilemdrun-api:9100andprometheus:80should time out.Deployed to dev and checked with CPU-only GROMACS jobs (1 rank, 4 threads):
complete= 0). The Extend run that followed finished and stored its exit line (complete= 1).kubectl top(999m).make usage JOBS=1(24.6% and 23.6% CPU used), the exporter, and the Grafana panel's CPU used query (24.4% combined) agree with the stored rows.mpirunbinds jobs with 1–2 ranks to a single core, so all 4 OpenMP threads shared one. That launch problem is outside this PR.make usage), because the range started before Prometheus' first sample.make usage JOBS=1, the exporter, and the dashboard (100% CPU stall) agree. The stall is thempirunpinning: four threads queueing for one core.make grafanaprints both dashboard links, ends cleanly on Ctrl+C, and fails with one line when monitoring is off, Grafana is missing, or port 3000 is taken.jupyterhubtarget is up with the service token, and the hub Secret gained onlyhub.services.prometheus.apiToken. The Platform dashboard shows active users and running servers, and the next login appeared as one successful spawn of about 45 s.make usage JOBS=1are unchanged.mdrun -v-style progress and then fails. Its 60 s usage line arrived joined onto a progress chunk, which only the new reader finds. The exit line carried no ns/day although the log held an earlierPerformance:line, the exit code and the marker were kept, and no sampler process was left.mdrun-api:80connects, whilemdrun-api:9100,prometheus:80, andgrafana:80time out.Not yet tested: GPU power draw and the diagnostics on a GPU job. The one measured GPU job stored 42.8% GPU busy and 0.3 GiB GPU memory, so
nvidia-smiworks in the job image, but its script predates the diagnostics. A realmdrun -vrun stopped mid-way and a real failed Extend were reproduced with a stand-in engine only. Step 8 was checked from a user pod, not from another hub pod.Locally, the real generated job scripts of all four engine variants (GROMACS and AMBER, CPU and GPU) ran through finish, failure, and stop with fake engines: each logged the final usage line and wrote the marker, and a failure kept its exit code.
Checklist
mdrun_usage_measuredis a new table thatcreate_allcreates)Prepared with Claude Code (Claude Opus 5.5 and Claude Fable 5.2).