Skip to content

persist cudagraphs #4364

persist cudagraphs

persist cudagraphs #4364

Triggered via push December 8, 2025 17:59
Status Success
Total duration 46m 29s
Artifacts 17

cicd-main.yml

on: push
is-not-external-contributor
8s
is-not-external-contributor
pre-flight  /  pre-flight
15s
pre-flight / pre-flight
cicd-wait-in-queue
4s
cicd-wait-in-queue
cicd-container-build
46s
cicd-container-build
cicd-parse-unit-tests
7s
cicd-parse-unit-tests
Matrix: cicd-unit-tests-latest
cicd-parse-integration-tests
12s
cicd-parse-integration-tests
Matrix: cicd-integration-tests-latest
Nemo_CICD_Test
8s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
ko3n1g
approved Dec 8, 2025
test

Automatically approved by queue manager

Annotations

33 errors and 13 warnings
cicd-container-build
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_model_configs.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/data/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/data/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/pipeline_parallel/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/pipeline_parallel/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_fp8_param.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_fp8_param.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/distributed/fsdp/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/distributed/fsdp/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/moe/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/moe/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/data/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/pipeline_parallel/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/test_fp8_param.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/distributed/fsdp/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/moe/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.

Artifacts

Produced during runtime
Name Size Digest
coverage-unit-test-20037761308-07d484ff-fb18-4112-a711-fb0a874dcf1f
119 KB
sha256:f4f17340108f4c771c2f9e106cba135fcaf299729ee7c098045c642ba68786c1
coverage-unit-test-20037761308-0a359964-7c20-4afe-aadb-957d5235f4ce
118 KB
sha256:8ac41079f41fa4ee2fb4d70299672f9898d13271986284d64b54a4d4ec5993dc
coverage-unit-test-20037761308-4d2d48f3-d407-4c92-9a77-01f2ddc51242
118 KB
sha256:e90e8d4d30a1565dd4d3f0a16de5d9121e49f29a16e267b5cb278f6668850ef5
coverage-unit-test-20037761308-5640a7c7-9b94-4f03-b46c-9f265e7cf85d
117 KB
sha256:cd021d2a8845a8e12332297515a55f5cc8f3fa875a5c71b79e1787c2bee39929
coverage-unit-test-20037761308-5da393b1-c97b-4e46-8533-842a7eac0e9d
117 KB
sha256:d1bae116759979ce5d16822a6ae0da852167e0d0351b9d5d5a82f25c8cbda322
coverage-unit-test-20037761308-5e920957-d3d6-4933-88b1-665e4118d26a
117 KB
sha256:4b0d4831a9397765d745b542283aaa61eb53c724c759cab5eac737796fe3bfbd
coverage-unit-test-20037761308-a604fb2d-d3cc-4376-bcd4-4b21f2368d6d
126 KB
sha256:09e4548e613779471c7aa2245e69f280196bd5b699a423299bcff9592ff4714e
coverage-unit-test-20037761308-cb5f9eae-d024-4a35-bf53-0822a5e0da10
119 KB
sha256:90f7a73b46f01fc260e985a608d1d4d87d56886cdb320150bb8c738d63bb140c
coverage-unit-test-20037761308-cf4665c7-185a-4d15-8d0b-cdbf30d6c831
120 KB
sha256:eb1f0036a84d6bffcfbddaba9ab3f56b87117c7c89e2f2757d004ab2da32221e
coverage-unit-test-20037761308-daf95e78-f1ad-4548-a58a-6e18c17a7138
114 KB
sha256:5c892e13f56257e88631eb0fda525e28dc102c65bb827a57fe7a94743bddd605
coverage-unit-test-20037761308-e02716c1-0fe2-49c4-92f7-65049d64dfa8
119 KB
sha256:d912c0c1c0629c55be7a2a7ef24d70dedd1005f551589abfbc90b96be4c93def
coverage-unit-test-20037761308-e0c43358-08d2-4577-9acf-34c2453c6f50
121 KB
sha256:3008eabf2bac5e86265ece742ce60d4767847711fe67244cf8ef4fad686c64f4
coverage-unit-test-20037761308-f082763a-50c6-42b8-b842-688005a7529d
116 KB
sha256:36ba2414f119c94227a76db39d8a4fd3134e6677f7c9f350df2ae5d443904d16
coverage-unit-test-aggregated
27.4 KB
sha256:6bd496ae879638b8bb210d3e24df88b3154d8b818e928d97392ca023a2ac0874
logs-gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer-20037761308-0ece6dc0-6f2c-4adf-81e8-84de7f695c9a
54.3 KB
sha256:ba8b177f1b99ddaeb9268a57d272c46042aabd129c4afde8de1c2d8d1000a0ff
logs-gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather-20037761308-d1f6f7f2-f84a-4d8a-94c0-c026543e4305
124 KB
sha256:994f51f47e32ad8becac277e6033b720bdbffae8c62090242b03eaf8accb7239
logs-gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer-20037761308-74a0a248-2120-4e06-b57e-347570d98b35
127 KB
sha256:6c7946f3e2178d5be9af7b2f8dd0555b8cb6e677ce24cf9418e8ab65a2f7a9b0