16.1.X: build root in debug mode by smuzaffar · Pull Request #10212 · cms-sw/cmsdist

smuzaffar · 2025-11-19T15:05:42Z

No description provided.

Removed conditional CUDA generation and adjusted build type.

smuzaffar · 2025-11-19T15:05:55Z

please test

cmsbuild · 2025-11-19T15:06:05Z

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_16_0_X/master.

@akritkbehera, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2025-11-19T15:06:05Z

cms-bot internal usage

smuzaffar · 2025-11-19T15:07:30Z

please test using full cmssw

cmsbuild · 2025-11-20T03:02:18Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/49555/summary.html
COMMIT: f585951
CMSSW: CMSSW_16_0_X_2025-11-19-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10212/49555/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/49555/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/49555/git-merge-result

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testDiMuonBiasesPlotting had ERRORS

Comparison Summary

Summary:

You potentially removed 162 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3896448
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3896422
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

makortel · 2025-12-10T14:27:55Z

@cmsbuild, please build

To help investigating cms-sw/cmssw#49568

cmsbuild · 2025-12-10T19:36:15Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/49882/summary.html
COMMIT: f585951
CMSSW: CMSSW_16_0_X_2025-12-10-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10212/49882/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/49882/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/49882/git-merge-result

makortel · 2026-01-22T16:03:30Z

@cmsbuild, please build

To aid investigating cms-sw/cmssw#49640 (comment)

cmsbuild · 2026-01-22T20:56:10Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/50814/summary.html
COMMIT: f585951
CMSSW: CMSSW_16_1_X_2026-01-22-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10212/50814/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@fwyzard Run the ONNX test on GPU only if CUDA is available cmssw#49883
@Dr15Jones Add additional forwarding files for DataFormats cmssw#49889
@slava77 LST standalone fix after interface change in Portable*MultiCollection cmssw#49895
@bsunanda Phase2-hgx364Y2a Update several test scripts in Validation/HGCalValidation/script in view of depreciation of several Phase2 scenarios cmssw#49788
@mmusich complete unit tests for testTrackingDQMAnalyzers cmssw#49890

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/50814/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/50814/git-merge-result

makortel · 2026-02-02T17:06:37Z

@cmsbuild, please build

To help with cms-sw/cmssw#49966

cmsbuild · 2026-02-02T20:49:37Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/51046/summary.html
COMMIT: f585951
CMSSW: CMSSW_16_1_X_2026-02-02-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10212/51046/install.sh to create a dev area with all the needed externals and cmssw changes.

fabiocos · 2026-02-09T15:36:53Z

@makortel sorry, how are you using this for debugging? If I follow the straightforward access to the PR area, even based on the build in #49969 , I do not manage to access to the ROOT code when in gdb from cmsdev45:

Begin processing the 1st record. Run 1, Event 1, LumiSection 1 on stream 0 at 09-Feb-2026 16:33:54.972 CET
++++ starting: processing event : stream = 0 run = 1 lumi = 1 event = 1 time = 5000001
++++++ starting: processing path 'p1' : stream = 0
++++++++ starting: prefetching before processing event for module: stream = 0 label = 'simHitCaloHitDumper' id = 3
++++++++++ starting: event delayed read from source: stream = 0 label = 'simHitCaloHitDumper' id = 3

Thread 1 "cmsRun" received signal SIGSEGV, Segmentation fault.
0x00007fffc598d620 in ROOT::new_Basic3DVectorlEfloatgR(void*) ()
   from /build/fabiocos/161X/psimhit_bot/CMSSW_16_1_X_2026-02-05-2300/lib/el8_amd64_gcc13/libDataFormatsGeometryVector.so
(gdb) up
#1  0x00007ffff78325a2 in TClass::NewObject (this=0x7fffc77d8900, arena=0x7fffbf500018, defConstructor=TClass::kClassNew)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-7e7bd6224bf7cc5dd5ace47373ed9aba/root-6.36.07/core/meta/src/TClass.cxx:5257
warning: 5257   /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-7e7bd6224bf7cc5dd5ace47373ed9aba/root-6.36.07/core/meta/src/TClass.cxx: No such file or directory

So do I need to rerun the external build myself?

makortel · 2026-02-09T17:18:06Z

The ROOT source code indeed is not available via these builds. I use other means (e.g. editor in a local clone of ROOT or GitHub directly) to view the ROOT code that corresponds the build (well enough).

The debug build gives the possibility to access ~any function argument, member data, or local variable.

makortel · 2026-02-16T20:49:02Z

@cmsbuild, please build

cmsbuild · 2026-02-17T00:58:46Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/51353/summary.html
COMMIT: f585951
CMSSW: CMSSW_16_1_X_2026-02-16-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10212/51353/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw use dumpfullversion so that for system gcc we also get Major.Minor.Patch #10346

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/51353/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7435c9/51353/git-merge-result

16.0.X: build root in debug mode

f585951

Removed conditional CUDA generation and adjusted build type.

cmsbuild added externals-pending pending-signatures tests-started orp-pending labels Nov 19, 2025

smuzaffar mentioned this pull request Nov 19, 2025

ROOT 6.36 results 510-660 MB higher memory usage in some workflows cms-sw/cmssw#49399

Closed

cmsbuild added tests-rejected and removed tests-started labels Nov 20, 2025

makortel mentioned this pull request Dec 10, 2025

"validateJR" object comparisons broken for step2 (HLT) in PR tests cms-sw/cmssw#49568

Closed

smuzaffar changed the base branch from IB/CMSSW_16_0_X/master to IB/CMSSW_16_1_X/master December 18, 2025 13:46

makortel mentioned this pull request Jan 27, 2026

Storing trigger::TriggerEvent and reco::GlobalHaloData in RNTuple take ~500 MB extra memory cms-sw/cmssw#49949

Open

smuzaffar changed the title ~~16.0.X: build root in debug mode~~ 16.1.X: build root in debug mode Jan 27, 2026

smuzaffar mentioned this pull request Jan 27, 2026

enable dqm comparison job for rntuple cms-sw/cms-bot#2651

Merged

makortel mentioned this pull request Feb 5, 2026

Revision of the PSimHit type storage, revert trackId to its basic meaning cms-sw/cmssw#49969

Open

Conversation

smuzaffar commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Nov 19, 2025

Uh oh!

cmsbuild commented Nov 19, 2025

Uh oh!

cmsbuild commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Nov 19, 2025

Uh oh!

cmsbuild commented Nov 20, 2025

Failed Unit Tests

Comparison Summary

Uh oh!

makortel commented Dec 10, 2025

Uh oh!

cmsbuild commented Dec 10, 2025

Uh oh!

makortel commented Jan 22, 2026

Uh oh!

cmsbuild commented Jan 22, 2026

Uh oh!

makortel commented Feb 2, 2026

Uh oh!

cmsbuild commented Feb 2, 2026

Uh oh!

fabiocos commented Feb 9, 2026

Uh oh!

makortel commented Feb 9, 2026

Uh oh!

makortel commented Feb 16, 2026

Uh oh!

cmsbuild commented Feb 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

smuzaffar commented Nov 19, 2025 •

edited

Loading

cmsbuild commented Nov 19, 2025 •

edited

Loading