Add a physics-plus-render mode to the render benchmark - #7797
mataylor-nvidia wants to merge 6 commits into
Conversation
The render benchmark could only ever report renderer time. Add a benchmark_mode switch so the same scene can be measured either way. In "render" mode (the default, and what the task did before) the articulations are posed by writing joint positions straight into the simulation, so the frame a renderer is timed on is the analytic sinusoid and nothing has to be actuated to produce it. In "physics_render" mode the same poses are requested as actuator targets, so the solver does an ordinary task's tracking work. Measuring the second mode needs a physics timer, so add PHYSICS_PROFILE_SCOPE around the physics step in SimulationContext.step, gated by ISAACLAB_PHYSICS_PROFILE and mirroring the existing ISAACLAB_RENDER_PROFILE timer. benchmark_renderer.py selects the mode with --mode, sums the decimated physics steps that precede each rendered frame, and grows PHYSICS and TOTAL columns when a sweep has them.
|
There was a problem hiding this comment.
Isaac Lab Review Bot
The physics-plus-render benchmark plumbing is coherent, but render mode writes the analytic joint pose before the environment’s decimated physics steps. The camera therefore observes a subsequently integrated state rather than necessarily the exact pose promised by the new documentation.
- Design and architecture: The mode switch and profiling scope are consistently connected across task configuration, environment behavior, subprocess environment variables, timing parsing, and reporting. However, the render-mode design does not fully isolate the rendered pose from physics because the direct state write occurs in
_pre_physics_step, before physics integration. - API: The new mode values, profiling scope, CLI option, and additive JSON fields are kept consistent and covered by tests. The documented contract for
benchmark_mode="render"overstates the behavior by claiming the rendered scene is exactly the analytic sinusoid even though physics steps occur after the direct write. - Implementation: Physics timings are accumulated across decimated steps and paired with each render timing appropriately. The render-mode state path needs adjustment: apply the direct pose immediately before rendering if exact analytic poses are required, or revise the documentation and benchmark claims to describe the post-integration pose accurately.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
Follows how isaac-sim#7702 handled ISAACLAB_RENDER_PROFILE: the timer is simply always on rather than gated behind a flag. Doing the same for ISAACLAB_PHYSICS_PROFILE removes the reason for everything --mode was carrying. Every log now has physics timings, so the parser no longer tracks whether a frame has them, the table no longer chooses its columns per run, and the test log writer no longer makes physics optional. The scene switch stays on BENCHMARK_MODE, read directly the way the profile env vars are. Both modes report physics, render, and total; the mode decides what the physics number means, not whether there is one.
Keep only the physics scope name, its pattern, and the env var that turns the timer on. Physics timings land in the run log next to the render ones; how they get summarized is a separate question from adding the timer, and reworking parse_log, build_record, and the results table to answer it here was scope this change did not need.
Greptile and the review bot both caught that render mode wrote the joint pose from _pre_physics_step, so decimation physics steps ran before the camera was read. The implicit drives were never retargeted and stayed active, and gravity and joint limits applied too, so the solver pulled the joints off the written pose and the renderer was timed on its output rather than the analytic one. Split the drive paths: _pre_physics_step now only advances the clock and, in physics_render mode, requests actuator targets. Render mode writes the pose in _get_observations instead, after the last physics step and just before the lazy camera read, with sim.forward() to propagate the joint write to the body transforms the renderer reads. The regression test asserts the call order across the step, so moving the write back ahead of physics fails it.
|
@greptile review |
Greptile caught that the post-physics joint write can miss the render. RenderContext.update_scene_state syncs at most once per physics step and sim.forward() does not advance that count, so with lazy_sensor_update off InteractiveScene.update has already stamped the current step and the lazy camera read reuses the transforms captured before the write. The task's default config leaves lazy_sensor_update on, so nothing had synced yet and the default path was already correct, but that is one config flag away from silently rendering the solver pose. reset_scene_state_cadence() is the documented hook for this and costs an extra sync only in the case that needs one.
|
@greptile review |
|
run-ci |
| """ | ||
|
|
||
| RENDER_SCOPE_PATTERN = re.compile(rf"{re.escape(RENDER_SCOPE)} took ([\d.]+) ms") | ||
| PHYSICS_SCOPE_PATTERN = re.compile(rf"{re.escape(PHYSICS_SCOPE)} took ([\d.]+) ms") |
There was a problem hiding this comment.
PHYSICS_SCOPE and PHYSICS_SCOPE_PATTERN seems to not be used anywhere, is that right?
There was a problem hiding this comment.
I've updated parse_log -> parse_frames to ingest these
parse_log took one hardcoded regex, so PHYSICS_SCOPE_PATTERN was compiled and never read: the timer ran and its numbers sat in the log unsummarized. Scopes are now a name-to-pattern mapping, and a caller can pass its own to summarize a timer this script does not know about. Scopes cannot be collected independently and zipped. Decimation puts several physics steps in front of one render, so a non-frame scope accumulates until the frame scope closes it out; counting the steps as frames instead would report the cost of a single step and shift the samples the warm-up padding skips past. parse_frames carries that, and a scope missing from a log reads as zero, so a run captured without ISAACLAB_PHYSICS_PROFILE still parses. Render stays flat in the results, so build_record and the existing tests did not move. Physics and total ride along as sub-dicts, and the record picks up sub-dicts by shape rather than by name, so another scope reaches the JSON without further edits. The table gained PHYSICS and TOTAL, which meant rebuilding it from column pairs: its widths were baked into noqa'd string literals that adding a column would have rewritten anyway.
|
@greptile review |
|
run-ci |
Description
Follow-up to #7702. That PR added the
Isaac-RenderBenchmark-Franka-Cabinettask, whose scene is always driven through the physics solver. This adds a mode switch so the same scene can be benchmarked with or without the solver doing that actuation work, plus a physics timer so what physics costs is visible either way.source/isaaclab/isaaclab/sim/simulation_context.py: addsPHYSICS_PROFILE_SCOPEaround the physics step inSimulationContext.step, gated by theISAACLAB_PHYSICS_PROFILEenvironment variable. This mirrors the existingISAACLAB_RENDER_PROFILEtimer added by Added Rendering Benchmark #7702 — same scope-name-per-backend approach, samewp.ScopedTimerprint format — and is off by default, so an ordinary run pays neither the print nor the device synchronization.source/isaaclab_tasks/.../render_benchmark_env_cfg.py: newbenchmark_modefield ("render"|"physics_render"), read from theBENCHMARK_MODEenvironment variable and rejecting unknown values at import, so a typo fails at launch instead of quietly benchmarking the default for a whole sweep.source/isaaclab_tasks/.../render_benchmark_env.py:_animate_jointspicks its drive path from the mode."physics_render"requests the sinusoid as actuator position targets, which is what the task did before."render"instead writes the joint positions and zero velocities straight into the simulation.scripts/benchmarks/benchmark_renderer.py: turns the physics timer on for every run, the same way Added Rendering Benchmark #7702 turns on the render timer, and declares the scope name and its log pattern. The script's parsing and reporting are untouched — physics timings land in the run log next to the render ones.Note that the physics backend still integrates in
"render"mode, because an Isaac Lab environment has no way to skip its own physics step; the mode changes how the joints are driven, not whether the step happens.Known issue
Greptile and the Isaac Lab review bot both flagged the same defect in
"render"mode, and they are right:_animate_jointsruns from_pre_physics_step, so the direct joint write is followed bydecimationphysics steps before the camera is read. The implicit actuator drives are still active with their previous targets, and gravity and joint limits also apply, so the rendered frame is the post-step state rather than the analytic sinusoid the docstrings currently claim. The fix is to either apply the pose after physics and before the render, or soften the documented contract. Not yet addressed — do not merge as-is.Type of change
Release backport
developChecklist
pre-commitchecks with./isaaclab.sh --formatsource/<pkg>/changelog.d/for every touched packageCONTRIBUTORS.mdor my name already exists there