`Restart()` returns early on non-reader ranks and can skip required post-read redistribution

[WeiqunZhang]

• Type: Correctness / Parallel synchronization
• Severity: High
• Component: Particle restart path (level-loss handling)
• Location:
  • Src/Particle/AMReX_ParticleIO.H:855
  • Src/Particle/AMReX_ParticleIO.H:939

Problem

In the lev > finestLevel() branch, ranks with rank >= NReaders execute:

if (rank >= NReaders) { return; }

This exits Restart() immediately from inside the per-level loop. Those ranks skip the rest of the restart flow, including final Redistribute() and consistency checks.

Impact

• Non-reader ranks can exit restart before the collective redistribution phase.
• In large runs with particles.nreaders < NProcs, this can cause inconsistent post-restart state and potential hangs when reader ranks enter MPI-heavy redistribution.

Suggested patch

Skip file reads for non-reader ranks, but do not leave Restart() early.

--- a/Src/Particle/AMReX_ParticleIO.H
+++ b/Src/Particle/AMReX_ParticleIO.H
@@
-            if (rank >= NReaders) { return; }
+            if (rank >= NReaders) { continue; }

Prepared by Codex

[BenWibking]

This potentially affects us.

Actually, I'm not sure how our restarts on Frontier work at present, since we always use particles_nfiles equal to the number of nodes.

[BenWibking]

This may be related: quokka-astro/quokka#1544

[atmyers]

Fixed here: #5074

I believe this only would get triggered if you are restarting with fewer levels than exist in the Checkpoint file. Is that usually the case for you?

[BenWibking]

That is not usually the case for us. I think @chongchonghe encountered this, so I'm not sure what the precise set-up was.

[chongchonghe]

I tried this with Quokka:

Checkpoint: 8 nodes, 64 GPUs, plt_nfiles=64
Restart with: 64 nodes, 512 GPUs, plt_nfiles=512

This causes a hang during writing a plotfile, and it only write a single plotfile.

[BenWibking]

@chongchonghe I thought that was a different issue from quokka-astro/quokka#1544?

[chongchonghe]

@chongchonghe I thought that was a different issue from quokka-astro/quokka#1544?

You are right. That's a different issue.