Skip to content

Superseded by focused CI fixes - #7865

Closed
ooctipus wants to merge 4 commits into
isaac-sim:developfrom
ooctipus:fix/shared-franka-ci
Closed

ooctipus wants to merge 4 commits into
isaac-sim:developfrom
ooctipus:fix/shared-franka-ci

Conversation

@ooctipus

@ooctipus ooctipus commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Superseded by independently reviewable fixes:

This combined PR is closed in favor of those focused changes. Each description records the confirmed introducing commit or external asset change, affected CI failures, validation, and remaining uncertainty.

CI on this branch passed the controller, RL export, Pour, and standalone jobs. Its renderer change fixed the 12 Franka image comparisons but introduced four heterogeneous Kuka failures by visiting intentionally unused clone-source paths; #7871 corrects that renderer-boundary mistake and validates all 16 cases against unchanged reference images.

Two additional failures investigated alongside these fixes have separate PRs: #7872 selects the CUDA runtime matching PyTorch, fixing Newton graph capture with the CUDA 13 wheels used by #7674; MuJoCo-Warp #1675 removes duplicate CCD compilation during cold startup. The latter reduces measured startup time locally, but its effect on Isaac Lab's cloud startup timeouts still needs CI confirmation.

Use link-origin velocity feedback consistently with the pose and Jacobian, normalize task-frame orientations, and damp large nullspace pose steps to avoid joint-limit stalls.

Validated all 18 operational-space integration tests, both affected tests on a second GPU, and all pre-commit checks. Existing convergence tolerances remain unchanged.
@ooctipus
ooctipus requested a review from a team September 17, 2026 08:25
@github-actions github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Sep 17, 2026
@greptile-apps

greptile-apps Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

The PR appears safe to merge, with a non-blocking hardening issue around non-finite standalone soak configuration.

Findings

  1. P2 Non-finite soak can hang

Summary

This PR addresses shared Franka asset, legacy rendering, operational-space test, and standalone smoke-supervisor regressions.

  • Selects convex-hull colliders for Franka Pour, Lift, and Reorient configurations.
  • Expands nested USD instances only in the legacy OVRTX export session.
  • Aligns operational-space feedback with link-origin conventions and stabilizes nullspace convergence.
  • Allows a ready standalone process to complete its configured soak interval after the startup deadline.
  • Adds focused regression coverage for the changed configurations and execution paths.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Launch standalone process] --> B{Process ready?}
  B -- No --> C{Startup timeout reached?}
  C -- Yes --> D[Terminate process group]
  C -- No --> B
  B -- Yes --> E[Begin soak interval]
  E --> F{Soak interval elapsed?}
  F -- No --> E
  F -- Yes --> D
Loading

Reviews (1) · Last reviewed commit: "Allow standalone demos to finish soaking..."

returncode = process.poll()
break
if now - start_time >= startup_timeout:
if ready_at is None and now - start_time >= startup_timeout:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Non-finite soak can hang

Once readiness is observed, the startup deadline is disabled and soak_time becomes the only remaining bound. Because ISAACLAB_STANDALONE_SOAK_TIME is parsed with float(...) without a finiteness check, values such as inf or nan make the soak comparison permanently false. A healthy but still-running demo would then remain supervised indefinitely, so the configured soak interval should be validated as finite.

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Reviewed the four targeted regression fixes: task-local Franka collider selection, legacy OVRTX export-session uninstancing, operational-space test feedback corrections, and post-readiness smoke-test soaking. The changed paths are scoped to their stated consumers and include focused regression coverage.

  • Design and architecture: Nested USD instances are expanded only on the anonymous legacy OVRTX export stage when keep_env_roots is enabled, preserving instancing on the simulation stage, single-environment exports, and the ovstage path. Franka collider selection remains localized to Pour and rigid Lift/Reorient configurations rather than changing the shared asset default.
  • API: No public interface is removed or renamed. export_stage_to_string retains its signature and default behavior boundary, while the smoke supervisor change refines the existing startup-timeout semantics so readiness transitions control to the soak interval. Package-specific changelog fragments are present.
  • Implementation: The OVRTX regression test verifies exported geometry, path-specific instance preservation, and unchanged input layers. Collider tests verify the selected USD variant for Pour, Lift, and Reorient, while the supervisor test covers readiness immediately before the startup deadline. Operational-space feedback now consistently uses link-origin velocity data alongside the link-origin Jacobian and pose. A non-blocking tradeoff is that each task assigns the complete spawn.variants mapping, so future presets that replace that mapping wholesale must preserve the collider selection.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

FRANKA_PANDA_LIFT_CFG = FRANKA_PANDA_CFG.copy()
FRANKA_PANDA_LIFT_CFG.spawn.usd_path = f"{ISAACLAB_NUCLEUS_DIR}/Robots/FrankaEmika/franka_panda.usda"
# Reset clearance was calibrated for these arm meshes; the asset's primitive colliders intersect the ground.
FRANKA_PANDA_LIFT_CFG.spawn.variants = {"Colliders": "convex_hulls"}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you check that this doesnt create regressions in other tasks?

@maxkra15 maxkra15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, was working on this as well

@kellyguo11

Copy link
Copy Markdown
Contributor

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 17, 2026
@ooctipus ooctipus changed the title Fix shared Franka and GPU CI regressions Superseded by focused CI fixes Sep 17, 2026
@ooctipus ooctipus closed this Sep 17, 2026
kellyguo11 pushed a commit that referenced this pull request Sep 17, 2026
)

Franka Lift can stall while filling its reset-state bank: the updated
`franka_panda.usda` defaults to `Colliders=primitives`, whose first-link
capsule extends 6 cm below the ground. The existing ground-clearance
criterion rejects every candidate, leaving the RSL-RL export batch at
`0/2048` until its 600-second timeout. This PR selects
`Colliders=convex_hulls` on the robot configuration shared by rigid
Franka Lift and Reorient, restoring the previous collision geometry and
successful reset sampling.

The affected asset and reset setup originated in
[#6457](#6457)
([564a008](564a008)),
then moved into the current Lift tasks in
[#6732](#6732)
([73106da](73106da)).
The triggering change is external: the production Franka asset's top
layer now selects primitive colliders and reports `Last-Modified:
2026-09-15 22:49:27 GMT`. Controlled comparisons found 0 valid states in
128 draws with the new default; explicit convex hulls and the preserved
old asset both produced 63 valid states from the same draws.

The same reset hang appeared in CI for #7838, #7608, #7832, #7834, and
#7866. [The #7866 RL
log](https://github.com/isaac-sim/IsaacLab/actions/runs/35202342707/job/105139978507)
shows the batch blocked on Franka Lift after environment setup. #7866
fixes the independently configured Franka Pour task; it does not change
Lift's robot configuration. This change covers Lift/Reorient only.

Validation:

- Fresh regression check on this branch: both Lift/Reorient cases failed
without the production fix; all 9 tests in `test_lift_env_cfg.py` passed
with it.
- `uv run isaaclab -f` passed.
- Earlier GPU validation of the identical Lift change completed the
original 2,048-state harvest and reset in 22.169 seconds; the existing
four-task RSL-RL export batch passed in 107.92 seconds.
- Local GPU validation used PyTorch 2.11+cu128, Newton 1.6rc1, and
RSL-RL 5.4.1. The identical Lift fix also passed [the full
locked-environment RL CI job in
#7865](https://github.com/isaac-sim/IsaacLab/actions/runs/35207648168/job/105158711330).

Reset-bank sizes, clearance criteria, export behavior, the global Franka
defaults, and Pour remain unchanged.
kellyguo11 pushed a commit to kellyguo11/IsaacLab-public that referenced this pull request Sep 17, 2026
…aac-sim#7870)

Franka Lift can stall while filling its reset-state bank: the updated
`franka_panda.usda` defaults to `Colliders=primitives`, whose first-link
capsule extends 6 cm below the ground. The existing ground-clearance
criterion rejects every candidate, leaving the RSL-RL export batch at
`0/2048` until its 600-second timeout. This PR selects
`Colliders=convex_hulls` on the robot configuration shared by rigid
Franka Lift and Reorient, restoring the previous collision geometry and
successful reset sampling.

The affected asset and reset setup originated in
[isaac-sim#6457](isaac-sim#6457)
([564a008](isaac-sim@564a008)),
then moved into the current Lift tasks in
[isaac-sim#6732](isaac-sim#6732)
([73106da](isaac-sim@73106da)).
The triggering change is external: the production Franka asset's top
layer now selects primitive colliders and reports `Last-Modified:
2026-09-15 22:49:27 GMT`. Controlled comparisons found 0 valid states in
128 draws with the new default; explicit convex hulls and the preserved
old asset both produced 63 valid states from the same draws.

The same reset hang appeared in CI for isaac-sim#7838, isaac-sim#7608, isaac-sim#7832, isaac-sim#7834, and
isaac-sim#7866. [The isaac-sim#7866 RL
log](https://github.com/isaac-sim/IsaacLab/actions/runs/35202342707/job/105139978507)
shows the batch blocked on Franka Lift after environment setup. isaac-sim#7866
fixes the independently configured Franka Pour task; it does not change
Lift's robot configuration. This change covers Lift/Reorient only.

Validation:

- Fresh regression check on this branch: both Lift/Reorient cases failed
without the production fix; all 9 tests in `test_lift_env_cfg.py` passed
with it.
- `uv run isaaclab -f` passed.
- Earlier GPU validation of the identical Lift change completed the
original 2,048-state harvest and reset in 22.169 seconds; the existing
four-task RSL-RL export batch passed in 107.92 seconds.
- Local GPU validation used PyTorch 2.11+cu128, Newton 1.6rc1, and
RSL-RL 5.4.1. The identical Lift fix also passed [the full
locked-environment RL CI job in
isaac-sim#7865](https://github.com/isaac-sim/IsaacLab/actions/runs/35207648168/job/105158711330).

Reset-bank sizes, clearance criteria, export behavior, the global Franka
defaults, and Pour remain unchanged.

(cherry picked from commit c5d33ef)
kellyguo11 pushed a commit to kellyguo11/IsaacLab-public that referenced this pull request Sep 17, 2026
…aac-sim#7870)

Franka Lift can stall while filling its reset-state bank: the updated
`franka_panda.usda` defaults to `Colliders=primitives`, whose first-link
capsule extends 6 cm below the ground. The existing ground-clearance
criterion rejects every candidate, leaving the RSL-RL export batch at
`0/2048` until its 600-second timeout. This PR selects
`Colliders=convex_hulls` on the robot configuration shared by rigid
Franka Lift and Reorient, restoring the previous collision geometry and
successful reset sampling.

The affected asset and reset setup originated in
[isaac-sim#6457](isaac-sim#6457)
([564a008](isaac-sim@564a008)),
then moved into the current Lift tasks in
[isaac-sim#6732](isaac-sim#6732)
([73106da](isaac-sim@73106da)).
The triggering change is external: the production Franka asset's top
layer now selects primitive colliders and reports `Last-Modified:
2026-09-15 22:49:27 GMT`. Controlled comparisons found 0 valid states in
128 draws with the new default; explicit convex hulls and the preserved
old asset both produced 63 valid states from the same draws.

The same reset hang appeared in CI for isaac-sim#7838, isaac-sim#7608, isaac-sim#7832, isaac-sim#7834, and
isaac-sim#7866. [The isaac-sim#7866 RL
log](https://github.com/isaac-sim/IsaacLab/actions/runs/35202342707/job/105139978507)
shows the batch blocked on Franka Lift after environment setup. isaac-sim#7866
fixes the independently configured Franka Pour task; it does not change
Lift's robot configuration. This change covers Lift/Reorient only.

Validation:

- Fresh regression check on this branch: both Lift/Reorient cases failed
without the production fix; all 9 tests in `test_lift_env_cfg.py` passed
with it.
- `uv run isaaclab -f` passed.
- Earlier GPU validation of the identical Lift change completed the
original 2,048-state harvest and reset in 22.169 seconds; the existing
four-task RSL-RL export batch passed in 107.92 seconds.
- Local GPU validation used PyTorch 2.11+cu128, Newton 1.6rc1, and
RSL-RL 5.4.1. The identical Lift fix also passed [the full
locked-environment RL CI job in
isaac-sim#7865](https://github.com/isaac-sim/IsaacLab/actions/runs/35207648168/job/105158711330).

Reset-bank sizes, clearance criteria, export behavior, the global Franka
defaults, and Pour remain unchanged.

(cherry picked from commit c5d33ef)
kellyguo11 pushed a commit to kellyguo11/IsaacLab-public that referenced this pull request Sep 17, 2026
…aac-sim#7870)

Franka Lift can stall while filling its reset-state bank: the updated
`franka_panda.usda` defaults to `Colliders=primitives`, whose first-link
capsule extends 6 cm below the ground. The existing ground-clearance
criterion rejects every candidate, leaving the RSL-RL export batch at
`0/2048` until its 600-second timeout. This PR selects
`Colliders=convex_hulls` on the robot configuration shared by rigid
Franka Lift and Reorient, restoring the previous collision geometry and
successful reset sampling.

The affected asset and reset setup originated in
[isaac-sim#6457](isaac-sim#6457)
([564a008](isaac-sim@564a008)),
then moved into the current Lift tasks in
[isaac-sim#6732](isaac-sim#6732)
([73106da](isaac-sim@73106da)).
The triggering change is external: the production Franka asset's top
layer now selects primitive colliders and reports `Last-Modified:
2026-09-15 22:49:27 GMT`. Controlled comparisons found 0 valid states in
128 draws with the new default; explicit convex hulls and the preserved
old asset both produced 63 valid states from the same draws.

The same reset hang appeared in CI for isaac-sim#7838, isaac-sim#7608, isaac-sim#7832, isaac-sim#7834, and
isaac-sim#7866. [The isaac-sim#7866 RL
log](https://github.com/isaac-sim/IsaacLab/actions/runs/35202342707/job/105139978507)
shows the batch blocked on Franka Lift after environment setup. isaac-sim#7866
fixes the independently configured Franka Pour task; it does not change
Lift's robot configuration. This change covers Lift/Reorient only.

Validation:

- Fresh regression check on this branch: both Lift/Reorient cases failed
without the production fix; all 9 tests in `test_lift_env_cfg.py` passed
with it.
- `uv run isaaclab -f` passed.
- Earlier GPU validation of the identical Lift change completed the
original 2,048-state harvest and reset in 22.169 seconds; the existing
four-task RSL-RL export batch passed in 107.92 seconds.
- Local GPU validation used PyTorch 2.11+cu128, Newton 1.6rc1, and
RSL-RL 5.4.1. The identical Lift fix also passed [the full
locked-environment RL CI job in
isaac-sim#7865](https://github.com/isaac-sim/IsaacLab/actions/runs/35207648168/job/105158711330).

Reset-bank sizes, clearance criteria, export behavior, the global Franka
defaults, and Pour remain unchanged.

(cherry picked from commit c5d33ef)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants