Skip to content

feat: per-user allocated and measured usage in Prometheus and Grafana - #187

Merged
andrejnespor merged 14 commits into
masterfrom
feat/usage-metrics
Nov 12, 2026
Merged

andrejnespor merged 14 commits into
masterfrom
feat/usage-metrics

Conversation

@andrejnespor

@andrejnespor andrejnespor commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Refs #169. Phases 2, 2b, and 2c of the usage-monitoring design, plus the hub's own metrics, (docs/specs/2026-10-23-usage-monitoring-design.md, updated with the decisions below). Tuner and user-namespace usage are later phases, so the issue stays open. This PR now also contains the former stacked #188.

Allocated usage in Prometheus and Grafana (phase 2)

  • With METRICS_PORT set, the MDRun poller serves per-user series computed from the mdrun_usage ledger at each scrape: cumulative jobs, CPU core-seconds, GPU-seconds, and memory GiB-seconds, plus the cores, GPUs, and memory held right now. The labels are user, namespace, and engine.
  • monitoring.enabled deploys a Prometheus server and a stateless Grafana into the hub namespace. It is on in config.dev.yaml and off for prod and edc. Grafana ships a provisioned "Usage by user" dashboard. There is no ingress: make grafana forwards Grafana and prints a link to each dashboard.
  • The series carry usernames. NetworkPolicies let only Prometheus reach the metrics port (MDRun's API port stays open), only Grafana reach Prometheus, and nothing reach Grafana except kubectl port-forward.

Worth knowing:

  • Cumulative usage is exported as gauges, not counters. A window closes at the container's exit, which MDRun can notice after a scrape already counted the job as running, so totals can step back. Prometheus treats any counter decrease as a reset and would count the whole total again. A period is therefore x - x offset d, never rate() or increase(). Grafana's range totals match make usage except at range boundaries (off by at most MDRun's detection delay) and for ranges that start before Prometheus' first sample, which show all-time totals. The table's Jobs column counts jobs that started in the range, where make usage counts every job that ran in it. The start-of-range value is the last sample in the hour before it, which bridges a scrape gap there.
  • Neither new pod mounts a ServiceAccount token, because every hub-namespace service account holds the hub's project role.
  • Prometheus keeps its config reloader, which the spec had dropped, so a scrape-config change applies on helm upgrade without a restart. The chart's Kubernetes discovery jobs are disabled, because they need cluster RBAC.
  • Grafana comes from grafana-community, because the grafana/helm-charts copy is deprecated.
  • Retention is 13 months on a 2Gi csi-ceph-rbd-du volume, with a 1500MB size cap as a backstop. Dev holds about 2,000 series, mostly the hub's and Prometheus' own, and grows about 2 MiB a day. On dev the stack adds 110m CPU and 416Mi memory to the hub requests, and scripts/resource_summary.py counts them.

Measured usage per job (phase 2b)

  • The GROMACS and AMBER job scripts measure their own container. CPU time and peak memory come from its cgroup v2 counters. On GPU jobs, nvidia-smi utilization and memory are sampled every 60 s. The script logs a mddash-usage line every 60 s and again when it exits.
  • When MDRun closes the usage window, it reads the last such line from the pod log before deleting the Job, and stores it in the new mdrun_usage_measured table.
  • make usage gains CPU used % and GPU busy % per user, and JOBS=1 lists each job with its measurements. The exporter adds mddash_mdrun_measured_* gauges, and the dashboard table gets matching columns.

Worth knowing:

  • It needs no new permissions. The hub role already grants pods/log, and containers can read their own cgroup v2 counters (checked on dev: cgroup2fs, kernel 6.8).
  • MDRun reads the pod log raw (_preload_content=False), as the dashboard API's job-log reader already does. kubernetes 36.0.3 returns a preloaded text/plain body as the repr of its bytes ("b'...\n...'" on one line), so no mddash-usage line would ever parse.
  • Measuring never blocks a stop, DELETE, or status read, the paths that read the log before the Job is deleted. Any read failure means no measurement. The log read times out after 2 s and urllib3 retries it three times, so a stop waits at most 8 s, within the dashboard's 10 s. Only the newest usage line counts, wherever it sits in a log line: the kubelet joins it onto unterminated output before it, such as mdrun -v progress. The parser keeps non-negative numbers that SQLite can store (an oversized count would fail the insert, and with it the stop) and logs the rest with the job's name.
  • CPU-only jobs never call nvidia-smi, because a CPU job on a GPU node may see other jobs' GPUs.
  • The exit trap changed. mddash_usage_final || true now runs before the completion marker's touch. Under set -e, a failing trap command skips the rest of the trap, and the s3-sync sidecar would then wait for a marker that never comes (checked in bash). Every measuring command tolerates failure, so measuring can't fail a job.
  • A stopped or deleted job keeps its last periodic reading, at most 60 s old, because the pod is gone before its exit line.
  • Efficiency covers measured jobs that ended in the period. Each is weighed against its whole window, which includes the image pull and S3 download that the allocation also covers. GPU utilization is sampled while the container runs, so the GPU counts as idle for the rest of the window.
  • Measurements come from the job container, so like owner they are a client claim.
  • mdrun_usage_measured is a new table, so create_all creates it and no migration is needed.

Job diagnostics (phase 2c)

  • Queue wait (submission to pod start) and the jobs and GPUs waiting right now come from the ledger.
  • Each mddash-usage line also carries the container's clock, CPU throttling, OOM kills, pressure-stall totals for CPU, memory, and I/O, disk bytes, and, on GPU jobs, power draw. The exit line adds the engine's own ns/day.
  • make usage adds CPU stall %, OOM kills, GPU kWh, and average wait per user. JOBS=1 adds wait, ns/day, CPU stall %, and OOM, and its CSV has every diagnostic. The exporter adds pending and queue-wait gauges and measured stall, OOM, and energy totals. The dashboard table gets four columns and a "Jobs waiting to start" panel.

Worth knowing:

  • mdrun_usage_measured gains 11 nullable columns. The table ships with this PR, so prod creates it complete. Dev already had it, and create_all never alters a table, so dev got the columns through ALTER TABLE (rows kept).
  • ns/day is the engine's end-of-run figure, so stopped runs have none: the engine reports after MDRun has read the log. Failed runs have none either, because an Extend's appended log may still hold the previous run's figure.
  • The "held" and "waiting" gauges follow MDRun's status reads. A job that has started counts as waiting until the dashboard or the poller (every 15 min) next reads it. A shorter poll interval waits for MDRun marks running jobs ERROR and deletes them on a transient Kubernetes API error #189.
  • A node without pressure accounting leaves the stall fields empty, and its jobs drop out of the stall denominator.

JupyterHub platform metrics

  • Prometheus scrapes the hub's /hub/metrics as a JupyterHub service prometheus with only the read:metrics scope. z2jh generates its token and keeps it in the hub Secret. The Prometheus pod projects only that key, because the Secret also holds the hub's cookie secret and crypt keys.
  • A "Platform" dashboard shows active users (24 h, 7 d, 30 d), running servers, spawn time and outcomes, and hub API latency and server errors. These series carry no usernames. The server-errors panel also counts a new series' first errors, which increase() alone cannot see.
  • The first deploy restarts the hub for its new config. User pods keep running.

How to test

  1. On dev, run make push-mdrun-api, then make deploy.
  2. kubectl -n mddash-dev get pods should show prometheus with 2/2 containers ready, grafana with 1/1, and mdrun-api restarted.
  3. Let a short CPU-only GROMACS job and a GPU job finish, and stop another job while it runs.
  4. make usage JOBS=1 should show CPU used %, peak memory, wait, CPU stall %, and OOM for the jobs, ns/day for the finished ones, and GPU busy % and GPU memory for the GPU job. With CSV=1, measured_to_exit should be yes for the finished jobs and no for the stopped one, next to startup, throttling, memory and I/O stall, disk GiB, and GPU power.
  5. Run make grafana and open its Usage by user link. A running job should appear in the "held" panels within a minute while its dashboard page is open, and a job waiting for a GPU in "Jobs waiting to start". For a range that starts after Prometheus' first sample (00:00 UTC of a later day until now), the table should match make usage SINCE=<that day>, including the measured columns. Jobs can differ when a job spans the range start.
  6. Open the Platform dashboard: active users and running servers show at once, and spawn time and outcomes after the next spawn.
  7. Run kubectl -n mddash-dev port-forward svc/prometheus 9090:80. Under Status, then Targets, mdrun-api and jupyterhub should be up.
  8. From another hub pod, mdrun-api:80 should answer, while mdrun-api:9100 and prometheus:80 should time out.

Deployed to dev and checked with CPU-only GROMACS jobs (1 rank, 4 threads):

  • Prometheus scrapes the exporter, and the dashboard's "held" and "Jobs started" panels track the runs.
  • A run stopped mid-way stored its last periodic reading (complete = 0). The Extend run that followed finished and stored its exit line (complete = 1).
  • The cgroup counter matched kubectl top (999m). make usage JOBS=1 (24.6% and 23.6% CPU used), the exporter, and the Grafana panel's CPU used query (24.4% combined) agree with the stored rows.
  • The low figure is real: mpirun binds jobs with 1–2 ranks to a single core, so all 4 OpenMP threads shared one. That launch problem is outside this PR.
  • "Last 2 days" showed all-time totals (8.7 core-h against 7.3 from make usage), because the range started before Prometheus' first sample.
  • With the diagnostics deployed, a GPU job waiting for an A40 shows as one pending job and GPU, and every query of both dashboards evaluates in Prometheus.
  • A CPU run stopped after 32 min stored its last periodic line with the diagnostics: 99.8% CPU stall, no throttling or OOM kills, and 8 s of startup. Its Extend stored the exit line with 6.02 ns/day. make usage JOBS=1, the exporter, and the dashboard (100% CPU stall) agree. The stall is the mpirun pinning: four threads queueing for one core.
  • make grafana prints both dashboard links, ends cleanly on Ctrl+C, and fails with one line when monitoring is off, Grafana is missing, or port 3000 is taken.
  • The jupyterhub target is up with the service token, and the hub Secret gained only hub.services.prometheus.apiToken. The Platform dashboard shows active users and running servers, and the next login appeared as one successful spawn of about 45 s.
  • Redeployed with the review fixes. The new MDRun closed a finished Extend on its first poll and stored its exit line (5.88 ns/day, 22.6% CPU used, 97.7% CPU stall). The nine earlier rows of make usage JOBS=1 are unchanged.
  • Prometheus runs with the 1500MB cap and kept its history, and all three targets are up. Grafana picked up the changed dashboards without a restart, and anonymous viewers still have Explore.
  • Dev's history has two one-minute scrape holes. For a range that starts inside one, the table shows 5 jobs and 4.71 core-h, where the earlier query showed the all-time 9 jobs and 11.70 core-h.
  • The newly generated job script ran in the GROMACS job image with a stand-in engine that prints mdrun -v-style progress and then fails. Its 60 s usage line arrived joined onto a progress chunk, which only the new reader finds. The exit line carried no ns/day although the log held an earlier Performance: line, the exit code and the marker were kept, and no sampler process was left.
  • From a user pod, mdrun-api:80 connects, while mdrun-api:9100, prometheus:80, and grafana:80 time out.

Not yet tested: GPU power draw and the diagnostics on a GPU job. The one measured GPU job stored 42.8% GPU busy and 0.3 GiB GPU memory, so nvidia-smi works in the job image, but its script predates the diagnostics. A real mdrun -v run stopped mid-way and a real failed Extend were reproduced with a stand-in engine only. Step 8 was checked from a user pod, not from another hub pod.

Locally, the real generated job scripts of all four engine variants (GROMACS and AMBER, CPU and GPU) ran through finish, failure, and stop with fake engines: each logged the final usage line and wrote the marker, and a failure kept its exit code.

Checklist

  • PR title and summary are descriptive.
  • Max one DB migration per PR. (none: mdrun_usage_measured is a new table that create_all creates)
  • Docs updated.
  • Tests included.
  • I have seen this code, I have run this code, and I take responsibility for this code.

Prepared with Claude Code (Claude Opus 5.5 and Claude Fable 5.2).

andrejnespor and others added 3 commits November 4, 2026 13:54
With METRICS_PORT set, the poller serves Prometheus series recomputed
from mdrun_usage at each scrape. Cumulative usage is a gauge: a window
closes at the container's exit, which can come after a scrape already
counted the job as running, and Prometheus would read the step back
as a counter reset. The report and the exporter share the window rules.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
monitoring.enabled (on for dev) deploys a server-only Prometheus that
scrapes the MDRun ledger metrics and a stateless Grafana with the
"Usage by user" dashboard, reached through make grafana. The series
carry usernames, so NetworkPolicies limit the metrics port to
Prometheus, Prometheus to Grafana, and Grafana to port-forward, and
neither pod mounts the hub namespace's service-account token.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The job script logs its container's cgroup CPU time and peak memory,
plus nvidia-smi samples on GPU jobs only, every minute and on exit.
MDRun reads the last line from the pod log when it closes the usage
window and keeps it in mdrun_usage_measured; make usage, the exporter,
and the dashboard compare it with the allocation. The exit trap runs
the measurement with || true before the completion marker, because
set -e would otherwise skip the marker the s3-sync sidecar waits for.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kubernetes 36.0.3 skips decoding str responses, so the pod log came
back as the repr of its bytes ("b'...\n...'") on a single line: no
mddash-usage line parsed, and stopped or finished jobs stored no
measurement. A raw read also never returns None, which crashed Stop
for a job whose pod never started.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andrejnespor andrejnespor changed the title feat: per-user usage metrics in Prometheus and Grafana feat: per-user allocated and measured usage in Prometheus and Grafana Nov 5, 2026
andrejnespor and others added 6 commits November 5, 2026 10:31
Each mddash-usage line also logs the container's clock, CPU throttling,
OOM kills, pressure-stall totals, disk bytes, and GPU power; the exit
line adds the engine's ns/day. Queue wait and waiting jobs come from
the ledger. The new mdrun_usage_measured columns ship with the table,
but dev already has it and needs ALTER TABLE: create_all never alters
an existing table.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prometheus scrapes the hub's /hub/metrics as a JupyterHub service with
only the read:metrics scope. z2jh generates its token and keeps it in
the hub Secret, and the Prometheus pod projects just that key: the
Secret also holds the hub's cookie secret and crypt keys. A Platform
dashboard shows active users, running servers, spawn time and
outcomes, and hub API latency and errors.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The target echoed its kubectl command and passed through a "Handling
connection" line for every browser request, with no link to open. It
now waits for the tunnel, prints a link per provisioned dashboard, and
ends cleanly on Ctrl+C. A config with monitoring off gets a one-line
reason instead of a kubectl error.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments and agent docs keep only durable context, two lines at most;
behavior the code or a HELP string already shows is dropped.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stop, DELETE, and status reads take the job's usage measurement from
its pod log before deleting the Job. A broken or hung log stream raised
a urllib3 error through them, and a count past SQLite's INTEGER
(oom_kills=1e30) failed the insert on every retry, so that job could
never be stopped.

The read now gives up after 10 s and treats any failure as no
measurement. The parser keeps only numbers in range and warns about the
rest, naming the job, and only the newest usage line is read. A test
runs the real job script through finish, failure, and stop.

Refs #169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One test per behavior that matters: the job script, reading and
parsing the measurement, storing it, the report, and the exporter.
Variant and formatting tests go, including the fake-mpirun exit-path
runs.

Refs #169
andrejnespor and others added 4 commits November 11, 2026 23:17
The kubelet joins a usage line onto unterminated output before it, such
as `mdrun -v` progress, so a stop stored a stale reading or none. The
reader now finds the prefix anywhere in a line.

An Extend's appended log still holds the previous run's Performance
line, so a run that died early logged that speed. The exit line reads
ns/day only after a clean exit. It also ends the sampler's sleep, which
outlived the script.

The log read times out after 2 s: urllib3 retries it three times, and
the 10 s it had could hold a stop for 40 s, past the dashboard's 10 s.
One pod lookup now feeds the window and the measurement.

Tests cover the engines' ns/day extraction and that CPU jobs never
sample GPUs again.

Refs #169

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
GPU utilization is sampled while the simulation container runs, but
busy seconds applied it to the whole window, so image pull, S3
download, and upload counted as busy. A measurement without the
container's clock keeps the whole window.

The report and exporter tests for queue wait, pending jobs, CPU stall,
OOM kills, and GPU energy are back: nothing else asserted them as
anything but zero.

Refs #169

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A range that started in a scrape gap found no sample to subtract and
showed all-time totals. The table now takes the last sample in the hour
before the range start. Its description says that Jobs counts jobs
started in the range, unlike `make usage`.

The hub errors panel also counts a new series' first errors, which
increase() cannot see.

retentionSize (1500MB of 2Gi) is a backstop: dev grows about 2 MiB a
day at 2,000 series, and time retention alone does not bound the TSDB.

Refs #169

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Refs #169

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@andrejnespor
andrejnespor merged commit 80db0a8 into master Nov 12, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant