DAOS-18368 rebuild: fix bug of ec_agg_boundary and agg peer update by liuxuezhao · Pull Request #17324 · daos-stack/daos

liuxuezhao · 2025-12-29T09:18:41Z

1. fix a bug of using ec_agg_boundary before checking its valid
2. add some more logs for rebuild fetch getting zero iod_size,
   to provide some hints for layout information.
3. fix a bug of EC agg peer update, some failed update need to be retried
    to avoid data corruption.
4. refine some detailed process of dtx_resync wating for rebuild scan.

 Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-12-29T09:18:57Z

Ticket title is 'Data corruption observed with master branch under MDonSSD environment.'
Status is 'In Progress'
Labels: 'md_on_ssd,scrubbed_2.8'
https://daosio.atlassian.net/browse/DAOS-18368

daosbuild3 · 2025-12-29T10:40:43Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17324/2/display/redirect

daosbuild3 · 2025-12-31T06:58:07Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17324/4/testReport/

daosbuild3 · 2025-12-31T12:45:24Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/4/execution/node/1313/log

daosbuild3 · 2025-12-31T16:00:59Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/4/execution/node/1323/log

liuxuezhao · 2026-01-04T10:47:06Z

just refresh to change a few logs.

daosbuild3 · 2026-01-04T12:26:50Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17324/7/testReport/

kccain

rebuild/ source file changes LGTM.

daosbuild3 · 2026-01-08T23:05:05Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/12/execution/node/1277/log

daosbuild3 · 2026-01-09T01:30:43Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/12/execution/node/1318/log

1. fix a bug of using ec_agg_boundary before checking its valid 2. add some more logs for rebuild fetch getting zero iod_size, to provide some hints for layout information. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

Some failures need to be retried. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

For reint ranks is excluded from rebuild/reclaim if the co_in_ver exceed rebuild ver. Should set its completion in rebuild leader to avoid possible stuck. Refine dtx_resync wait handling, need not wait anymore if previously already resynced. Add some log. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

daosbuild3 · 2026-01-13T04:08:52Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17324/14/testReport/

daosbuild3 · 2026-01-13T06:34:28Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/14/execution/node/1075/log

daosbuild3 · 2026-01-13T08:12:00Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17324/15/testReport/

Nasf-Fan · 2026-01-13T11:06:44Z

src/object/srv_obj_migrate.c

 	int			 iod_cnt = 0;
 	int			 start;
-	char		 iov_buf[OBJ_ENUM_UNPACK_MAX_IODS][MAX_BUF_SIZE];
+	char                     iov_buf[OBJ_ENUM_UNPACK_MAX_IODS][MAX_BUF_SIZE];


Honestly, it is not good idea to put 32KB on stack that may potentially cause ULT overflow.

right, it is original code this PR only change the indent.

Nasf-Fan · 2026-01-13T11:07:19Z

src/rebuild/srv.c

+	if (status->dtx_resync_version != resync_ver)
+		D_INFO(DF_RB " rank %d, update dtx_resync_version from %d to %d", DP_RB_RGT(rgt),
+		       rank, status->dtx_resync_version, resync_ver);
 	status->dtx_resync_version = resync_ver;


if (status->dtx_resync_version != resync_ver) {
D_INFO(DF_RB " rank %d, update dtx_resync_version from %d to %d", DP_RB_RGT(rgt),
rank, status->dtx_resync_version, resync_ver);
status->dtx_resync_version = resync_ver;
}

daosbuild3 · 2026-01-13T18:07:19Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/15/execution/node/1350/log

liuxuezhao · 2026-01-15T07:54:02Z

there is one known NLT err, another only test failure is interactive-rebuild https://daosio.atlassian.net/browse/DAOS-18466

liuxuezhao · 2026-01-15T07:54:27Z

there is one known NLT err, another only test failure is interactive-rebuild https://daosio.atlassian.net/browse/DAOS-18466

gnailzenh · 2026-01-16T04:29:37Z

src/object/srv_obj_migrate.c

+				mrone->mo_epoch);
+			if (log_nr <= 128) {
+				mrone_dump_info(mrone, oh, &mrone->mo_iods[i]);
+				log_nr++;


we might want to reset this counter at somehow.

liuxuezhao requested review from a team as code owners December 29, 2025 09:18

liuxuezhao mentioned this pull request Dec 29, 2025

DAOS-17843 rebuild: add some logs for rebuild enumerate #17309

Merged

6 tasks

liuxuezhao requested review from NiuYawei, gnailzenh and wangshilong December 29, 2025 09:25

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch 2 times, most recently from 122e9e8 to 10fa58e Compare December 29, 2025 10:39

NiuYawei previously approved these changes Dec 30, 2025

View reviewed changes

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from 10fa58e to 66f44a9 Compare December 31, 2025 06:06

liuxuezhao dismissed NiuYawei’s stale review via 30a8331 January 4, 2026 10:46

liuxuezhao requested a review from NiuYawei January 4, 2026 10:47

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch 2 times, most recently from 32db84f to e1c08ea Compare January 4, 2026 11:00

NiuYawei previously approved these changes Jan 5, 2026

View reviewed changes

wangshilong previously approved these changes Jan 5, 2026

View reviewed changes

liuxuezhao dismissed stale reviews from wangshilong and NiuYawei via c474fe4 January 7, 2026 07:31

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from e1c08ea to c474fe4 Compare January 7, 2026 07:31

liuxuezhao requested review from NiuYawei and wangshilong January 7, 2026 07:33

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from 56751dc to f4bc272 Compare January 8, 2026 13:00

NiuYawei previously approved these changes Jan 8, 2026

View reviewed changes

wangshilong previously approved these changes Jan 8, 2026

View reviewed changes

kccain previously approved these changes Jan 8, 2026

View reviewed changes

liuxuezhao dismissed stale reviews from kccain, wangshilong, and NiuYawei via 17ef124 January 9, 2026 10:51

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from f4bc272 to 17ef124 Compare January 9, 2026 10:51

liuxuezhao requested review from Nasf-Fan, NiuYawei and wangshilong January 9, 2026 10:56

liuxuezhao added 5 commits January 13, 2026 03:02

DAOS-18368 rebuild: fix use before check of ec_agg_boundary

a287678

1. fix a bug of using ec_agg_boundary before checking its valid 2. add some more logs for rebuild fetch getting zero iod_size, to provide some hints for layout information. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

DAOS-18368 rebuild: a few log change

da0699c

Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

DAOS-18368 rebuild: refine rebuild_leader_status_check

aac91d7

Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

DAOS-18368 object: refine EC agg peer update

d1200f2

Some failures need to be retried. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from 17ef124 to 90de18f Compare January 13, 2026 03:03

NiuYawei approved these changes Jan 13, 2026

View reviewed changes

Nasf-Fan approved these changes Jan 13, 2026

View reviewed changes

gnailzenh approved these changes Jan 16, 2026

View reviewed changes

gnailzenh merged commit e717e1f into master Jan 16, 2026
38 of 41 checks passed

gnailzenh deleted the lxz/rb_fix_ec_agg_eph branch January 16, 2026 08:36

Conversation

liuxuezhao commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

liuxuezhao commented Jan 4, 2026

Uh oh!

daosbuild3 commented Jan 4, 2026

Uh oh!

kccain left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 8, 2026

Uh oh!

daosbuild3 commented Jan 9, 2026

Uh oh!

daosbuild3 commented Jan 13, 2026

Uh oh!

daosbuild3 commented Jan 13, 2026

Uh oh!

daosbuild3 commented Jan 13, 2026

Uh oh!

Nasf-Fan Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

liuxuezhao Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

liuxuezhao Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 13, 2026

Uh oh!

liuxuezhao commented Jan 15, 2026

Uh oh!

liuxuezhao commented Jan 15, 2026

Uh oh!

gnailzenh Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

liuxuezhao commented Dec 29, 2025 •

edited

Loading