SRE-3534 ci: adapt CI to use el9.7 for Build and UT by grom72 · Pull Request #17632 · daos-stack/daos

grom72 · 2026-03-03T06:51:18Z

Requires:

SRE-3534 ci: enable el9 functional tests pipeline-lib#495

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Allow-unstable-test: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

Priority: 2 Skip-unit-tests:true Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Priority: 2 Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Priority: 2 Skip-unit-tests:true Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: false Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Allow-unstable-test: true Skip-func-vm: true Skip-func-hw-test: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-9.6-rpms: false Skip-test-leap-15-rpms: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: false Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-9.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: false Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true Skip-func-test-vm-valgrind: false Skip-func-test-el8: false Skip-func-test-el9: false Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false

Skip-func-test: true Skip-func-test-vm: true Skip-func-test-hw: true Skip-nlt: true Skip-coverity-test: true Skip-fault-injection-test: true Skip-el8: true Signed-off-by: Oksana Salyk <[email protected]>

Signed-off-by: Oksana Salyk <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-test-vm-valgrind: false Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-8.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-func-hw-test-medium: true Skip-func-hw-test-medium-md-on-ssd: true Skip-func-hw-test-medium-vmd: true Skip-func-hw-test-medium-verbs-provider: true Skip-func-hw-test-medium-verbs-provider-md-on-ssd: true Skip-func-hw-test-medium-ucx-provider: true Skip-func-hw-test-large: true Skip-func-hw-test-large-md-on-ssd: true

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-test-vm-valgrind: false Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-8.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-9.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test-medium: true Skip-func-hw-test-medium-md-on-ssd: true Skip-func-hw-test-medium-vmd: true Skip-func-hw-test-medium-verbs-provider: true Skip-func-hw-test-medium-verbs-provider-md-on-ssd: true Skip-func-hw-test-medium-ucx-provider: true Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false

Signed-off-by: Oksana Salyk <[email protected]>

Use image_version parameter to provide OS version Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true

This reverts commit d4871a1. Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true

Signed-off-by: Oksana Salyk <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true

Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Signed-off-by: Oksana Salyk <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true

To test that default EL9 FT works. Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Test-tag: test_self_test Skip-python-bandit: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-fault-injection-test: true Skip-test-el-8.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true

daosbuild3 · 2026-03-03T18:23:49Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17632/1/execution/node/1353/log

This reverts commit f106467. Signed-off-by: Tomasz Gromadzki <[email protected]>

CI_MORE_FUNCTIONAL_PR_TESTS is no longer needed to trigger FT on el8,leap15. Only CI_FUNCTIONAL_*_TEST parameters are used to controll which tests should be run. Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Skip-func-hw-test: true

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Skip-python-bandit: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-test-el9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true

grom72 · 2026-03-04T12:58:45Z

Final validation results:
https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17632/8

New warnings in Fault Injection stage are triggered by fault injection operation.
6 of them are because of misinterpretation of Logging allocation failure text in fault injection log line.
Similar situation in NLT tests.

New "warning" in NLT -> srv_container.c: has already reported by: https://daosio.atlassian.net/browse/DAOS-18631

src/container/srv_container.c:2628 cont_open() a3e7e074/98b8d535/018a3529: failed to open the container with flags 82, capas 1/0: DER_NO_PERM(-1001): 'Operation not permitted'

daltonbohning · 2026-03-04T18:54:21Z

Why have the point release in the fault injection stage but not Functional stage?

Fault injection testing on EL 9.7
Functional on EL 9

Also, the build stages no longer specify the point release, but the functional stages do? E.g.

Build on Leap 15
Functional on Leap 15.6

The reason I'm being strict here is because changing the merge requirements to reflect the new stage names is a pain because it requires everyone to rebase and/or force landings each time we change required stage names.

Move the point release to the image_version parameter. Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true Skip-func-test-el8: false Skip-func-test-leap15: false Skip-test-el-9-rpms: false Skip-test-leap-15-rpms: false Skip-func-hw-test: true

grom72 · 2026-03-04T23:28:11Z

Why have the point release in the fault injection stage but not Functional stage?

Fault injection testing on EL 9.7

Functional on EL 9

Also, the build stages no longer specify the point release, but the functional stages do? E.g.

Build on Leap 15

Functional on Leap 15.6

The reason I'm being strict here is because changing the merge requirements to reflect the new stage names is a pain because it requires everyone to rebase and/or force landings each time we change required stage names.

@daltonbohning I have removed all point release from stage names in dd00593.
Please confirm that they are proper now.

daosbuild3 · 2026-03-05T03:30:49Z

Test stage Functional on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17632/10/execution/node/1200/log

grom72 · 2026-03-05T13:13:16Z

Validation results:
https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17632/10

Functional Leap 15 in separate run:
https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17632/11
due to a temporary error on KVM host: https://daosio.atlassian.net/browse/SRE-3635

phender · 2026-03-06T15:49:55Z

Jenkinsfile

+                stage('Test RPMs on EL 9.6') {
                    when {
                        beforeAgent true
-                        expression { params.CI_TEST_EL8_RPMs && !skipStage() }


Why are we removing params.CI_TEST_EL8_RPMs? We are keeping params.CI_TEST_LEAP15_RPMs on line 1082. Shouldn't this be the following?
expression { params.CI_TEST_EL_RPMs && !skipStage() }

phender · 2026-03-06T15:50:55Z

Jenkinsfile

                     defaultValue: true,
-                     description: 'Run the Test RPMs on EL 8 test stage')
+                     description: 'Run the Test RPMs on EL stage')
        booleanParam(name: 'CI_TEST_LEAP15_RPMs',


Based upon the above change, should this be?

Suggested change

booleanParam(name: 'CI_TEST_LEAP15_RPMs',

booleanParam(name: 'CI_TEST_LEAP_RPMs',

grom72 and others added 30 commits January 14, 2026 12:56

Trigger FT skip NLT

8590393

Priority: 2 Skip-unit-tests:true Signed-off-by: Tomasz Gromadzki <[email protected]>

Force 9.7 via Jenkins file

1750cc8

Cancel-prev-build: false Priority: 2 Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Switch to EL9

0cabf7b

Signed-off-by: Tomasz Gromadzki <[email protected]>

Use DISTRO provided by pipeline-lib

eded0e1

Cancel-prev-build: false Priority: 2 Skip-unit-tests:true Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Merge remote-tracking branch 'origin/master' into grom72/SRE-3522

afd3f01

Signed-off-by: Tomasz Gromadzki <[email protected]>

Trigger UT and FT on el9

f02ea77

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Allow-unstable-test: true Skip-func-vm: true Skip-func-hw-test: true Signed-off-by: Tomasz Gromadzki <[email protected]>

SRE_3459 New explicit parameter: 'image_version'

a644496

Skip-func-test: true Skip-func-test-vm: true Skip-func-test-hw: true Skip-nlt: true Skip-coverity-test: true Skip-fault-injection-test: true Skip-el8: true Signed-off-by: Oksana Salyk <[email protected]>

update files after code review

c4ca941

Signed-off-by: Oksana Salyk <[email protected]>

fix Jenkinsfile

17f8d94

Signed-off-by: Oksana Salyk <[email protected]>

update Jenkinsfile

77a96eb

Signed-off-by: Oksana Salyk <[email protected]>

try new branch from pipeline-lib

8855d3a

Signed-off-by: Oksana Salyk <[email protected]>

improve NLT test

3ffdc55

Signed-off-by: Oksana Salyk <[email protected]>

add changes to Jenkinsfile

d796541

Signed-off-by: Oksana Salyk <[email protected]>

add some changes to Jenkinsfile

50c4c0f

Signed-off-by: Oksana Salyk <[email protected]>

Remove OS versions from stage names

d4871a1

Use image_version parameter to provide OS version Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true

Revert "Remove OS versions from stage names"

bb4c509

This reverts commit d4871a1. Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2

Address review comments

b5725a0

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true

updating stage names

987c12c

Signed-off-by: Oksana Salyk <[email protected]>

Normalize distro name by removing minor version number

bd05432

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true

SRE-3466 test: use image_version in Functional Hardware Medium Verbs

c44a30b

Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Signed-off-by: Oksana Salyk <[email protected]>

Simplify stages names

7ec12f3

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true

grom72 added 2 commits March 3, 2026 21:39

Revert "For pipeline-lib test purpose only"

e06e3b1

This reverts commit f106467. Signed-off-by: Tomasz Gromadzki <[email protected]>

janekmi previously approved these changes Mar 4, 2026

View reviewed changes

grom72 dismissed janekmi’s stale review via fea838d March 4, 2026 12:28

osalyk approved these changes Mar 4, 2026

View reviewed changes

grom72 requested a review from janekmi March 4, 2026 12:38

grom72 marked this pull request as ready for review March 4, 2026 12:43

grom72 requested a review from a team as a code owner March 4, 2026 12:43

grom72 mentioned this pull request Mar 4, 2026

SRE-3534 ci: adapt CI to use el9.7 for Build and Unit tests #17376

Closed

25 tasks

osalyk previously approved these changes Mar 4, 2026

View reviewed changes

janekmi previously approved these changes Mar 4, 2026

View reviewed changes

grom72 requested a review from a team March 4, 2026 17:11

grom72 dismissed stale reviews from janekmi and osalyk via dd00593 March 4, 2026 23:22

grom72 requested review from janekmi and osalyk March 4, 2026 23:22

grom72 removed the request for review from a team March 5, 2026 11:27

janekmi approved these changes Mar 5, 2026

View reviewed changes

osalyk approved these changes Mar 5, 2026

View reviewed changes

grom72 requested a review from a team March 5, 2026 13:41

phender requested changes Mar 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SRE-3534 ci: adapt CI to use el9.7 for Build and UT#17632

SRE-3534 ci: adapt CI to use el9.7 for Build and UT#17632
grom72 wants to merge 62 commits intomasterfrom
grom72/SRE-3534-final-tuning

grom72 commented Mar 3, 2026 •

edited

Loading

Uh oh!

daosbuild3 commented Mar 3, 2026

Uh oh!

grom72 commented Mar 4, 2026

Uh oh!

daltonbohning commented Mar 4, 2026

Uh oh!

grom72 commented Mar 4, 2026 •

edited

Loading

Uh oh!

daosbuild3 commented Mar 5, 2026

Uh oh!

grom72 commented Mar 5, 2026

Uh oh!

phender Mar 6, 2026

Uh oh!

phender Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

	booleanParam(name: 'CI_TEST_LEAP15_RPMs',
	booleanParam(name: 'CI_TEST_LEAP_RPMs',

Conversation

grom72 commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

daosbuild3 commented Mar 3, 2026

Uh oh!

grom72 commented Mar 4, 2026

Uh oh!

daltonbohning commented Mar 4, 2026

Uh oh!

grom72 commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Mar 5, 2026

Uh oh!

grom72 commented Mar 5, 2026

Uh oh!

phender Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

phender Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

grom72 commented Mar 3, 2026 •

edited

Loading

grom72 commented Mar 4, 2026 •

edited

Loading