add settings to make training results deterministic#5449
add settings to make training results deterministic#5449redfler wants to merge 6 commits intoisaac-sim:developfrom
Conversation
redfler
requested review from
Mayankm96,
hhansen-bdai,
kellyguo11 and
ooctipus
as code owners
github-actions
Bot
added
isaac-sim
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
🤖 Isaac Lab Review Bot
Summary
This PR adds determinism support for vision-based RL training by: (1) calling configure_seed() in the rl_games training script to enable PyTorch deterministic algorithms, and (2) providing a new .kit experience file with RTX renderer settings that force synchronous rendering. The changes are minimal and focused, addressing a known reproducibility issue (GitHub #3505) for camera-based tasks.
Architecture Impact
- train.py: The
configure_seed()call affects only this specific training script. Other RL library wrappers (RSL-RL, SB3, SKRL) are NOT updated — users of those frameworks will still have non-deterministic vision training. - .kit file: New experience file extends
isaaclab.python.headlesswith RTX determinism settings. No breaking changes to existing.kitfiles.
Implementation Verdict
Minor fixes needed — The core approach is sound, but there are gaps in test coverage, documentation, and consistency across training scripts.
Test Coverage
❌ No tests added. The PR description acknowledges this (checklist unchecked). For a bug fix claiming to resolve reproducibility issues, a regression test demonstrating determinism would be valuable — e.g., running 2 training iterations with the same seed and asserting identical loss/reward values. Without tests, the claim in the screenshots cannot be automatically verified.
CI Status
No CI checks available yet — cannot verify the changes don't break existing functionality.
Findings
🟡 Warning: scripts/reinforcement_learning/rl_games/train.py:207 — configure_seed hardcodes True for deterministic mode
configure_seed(env_cfg.seed, True)The second argument enables torch.use_deterministic_algorithms(True), which can cause runtime errors for operations without deterministic implementations (e.g., certain scatter/gather ops on CUDA). This should either:
- Be gated behind a CLI flag (e.g.,
--deterministic) so users can opt-in - At minimum, document which operations may fail
Looking at isaaclab.utils.seed.configure_seed, if the user hasn't explicitly requested determinism, forcing it could break existing workflows that previously worked.
🟡 Warning: Inconsistency across training scripts — only rl_games is updated
The RSL-RL (scripts/reinforcement_learning/rsl_rl/train.py), Stable-Baselines3, and SKRL training scripts are not updated. Users expecting determinism across all frameworks will be surprised. Either:
- Update all training scripts consistently
- Document that determinism is only supported for rl_games
🔵 Improvement: apps/isaaclab.python.headless.determinism.kit:168 — Missing newline at end of file
rtx.rtpt.lightcache.cached.enabled = false
File doesn't end with a newline. While minor, this violates POSIX standards and can cause issues with some tools.
🔵 Improvement: apps/isaaclab.python.headless.determinism.kit:12 — Package title doesn't reflect determinism purpose
title = "Isaac Lab Python Headless Camera"The title says "Headless Camera" but this file is specifically for deterministic rendering. Should be "Isaac Lab Python Headless Deterministic Rendering" or similar for clarity.
🔵 Improvement: apps/isaaclab.python.headless.determinism.kit — Missing documentation comments explaining the key determinism settings
The critical settings for determinism are:
- Line 79:
omni.replicator.asyncRendering = false - Line 82-84:
app.renderer.waitIdle=true,app.hydraEngine.waitIdle=true - Line 165-167:
rtx.rtpt.cached.enabled = false,rtx.rtpt.lightcache.cached.enabled = false
These should have comments explaining WHY they enable determinism, not just performance implications. Users looking at this file need to understand what makes it different from the non-deterministic version.
🔵 Improvement: PR description shows command example but no documentation update
The command example in the PR description:
./isaaclab.sh -p ... --experience isaaclab.python.headless.determinism.kit
This should be documented in the official docs (the checklist shows documentation is unchecked). Users won't discover this without documentation.
🟡 Warning: scripts/reinforcement_learning/rl_games/train.py:207 — configure_seed called after environment creation
The configure_seed call happens after gym.make() and wrapper creation (lines 161-189). If environment initialization involves any non-deterministic operations (random initialization, etc.), those won't be affected by this late seeding. The seed should be configured before environment creation, not after. Looking at the code flow:
- Line 111:
env_cfg.seed = agent_cfg["params"]["seed"]— seed is set in config - Line 161:
env = gym.make(...)— environment created - Line 207:
configure_seed(...)— deterministic mode enabled too late
This ordering means the environment creation itself may still be non-deterministic even with this fix.
Greptile SummaryThis PR adds two changes to make Isaac Lab RL training results deterministic for vision-based tasks: a new
Confidence Score: 3/5Not safe to merge as-is; hardcoded A P1 finding (unconditional
Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant train.py
participant gym as gym.make (env)
participant configure_seed as configure_seed()
participant runner as rl_games Runner
User->>train.py: run with --seed 42 --experience determinism.kit
train.py->>train.py: set env_cfg.seed from agent_cfg["params"]["seed"]
train.py->>gym: gym.make(task, cfg=env_cfg) → env internally calls configure_seed(seed)
gym-->>train.py: env created (seed already applied)
train.py->>configure_seed: configure_seed(env_cfg.seed, True)
Note over configure_seed: Sets torch.use_deterministic_algorithms(True)<br/>Sets CUBLAS_WORKSPACE_CONFIG<br/>Re-seeds torch/numpy/random/warp
configure_seed-->>train.py: seed applied (deterministic mode ON)
train.py->>runner: runner.load(agent_cfg)
train.py->>runner: runner.run(train=True)
Note over runner: Any non-deterministic torch op<br/>raises RuntimeError here
Reviews (1): Last reviewed commit: "added name to the CONTRIBUTORS.md" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
github-actions
Bot
added
the
documentation
There was a problem hiding this comment.
🤖 Isaac Lab Review Bot
Summary
This follow-up addresses the previous review findings. The author has made several improvements: added --deterministic CLI flag (addressing the hardcoded True concern), renamed the kit file title to reflect determinism, added documentation to reproducibility.rst, and added explanatory comments for determinism settings in the kit file.
Previous Concerns Addressed
✅ configure_seed now uses args_cli.deterministic instead of hardcoded True
✅ Kit file title updated to "Isaac Lab Python Headless Deterministic Rendering"
✅ Documentation added explaining the --experience flag and --deterministic option
✅ Comments added explaining RTPT cache settings for determinism
Remaining/New Findings
🟡 Warning: scripts/reinforcement_learning/rl_games/train.py:215 — configure_seed still called after environment creation
The previous concern about ordering remains unaddressed. configure_seed() is called at line 215, but gym.make() happens at line 161. Any randomness during environment initialization won't be affected by the deterministic settings. The call should be moved before env = gym.make(...).
🟡 Warning: Test coverage still missing
The PR checklist still shows tests are not added. For a bug fix PR claiming to resolve reproducibility issues (#3505), a regression test demonstrating determinism would strengthen confidence in the fix.
🔵 Improvement: apps/isaaclab.python.headless.determinism.kit — File still missing trailing newline
Minor POSIX compliance issue remains.
Implementation Verdict
Minor fixes needed — The seed ordering issue is the main remaining concern that could affect correctness.
|
|
||
| [dependencies] | ||
| # Isaac Lab minimal app | ||
| "isaaclab.python.headless" = {} |
There was a problem hiding this comment.
probably better to depend on the headless rendering app? and then we can avoid the additional dependencies below
| ) | ||
| parser.add_argument("--seed", type=int, default=None, help="Seed used for the environment") | ||
| parser.add_argument( | ||
| "--deterministic", |
There was a problem hiding this comment.
can this flag be added to AppLauncher instead so that we can switch to using the reproducible .kit file automatically?
| ``scripts/reinforcement_learning/`` still honor ``--seed`` / agent configuration for the Isaac Lab | ||
| environment and learning stack, but they do **not** expose an equivalent opt-in for strict PyTorch-wide | ||
| deterministic algorithms. If you need that behavior with another framework, call | ||
| :meth:`~isaaclab.utils.seed.configure_seed` with ``torch_deterministic=True`` from your own training |
There was a problem hiding this comment.
what's the reason this can't be added to the other training scripts?
2 participants
Description
This PR adds a deterministic training path and documentation for Isaac Lab RL workflows.
and clarify that strict PyTorch determinism is currently exposed only for RL-Games.
Test command example:
./isaaclab.sh -p scripts/reinforcement_learning/rl_games/train.py --task Isaac-Cartpole-RGB-v0 --enable_cameras --headless --seed 42 --max_iteration 20 --deterministic --experience isaaclab.python.headless.determinism.kit
Fixes # (issue)
#3505 Non-reproducible training results in vision-based tasks with identical seeds
Type of change
Screenshots
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there