Optimization of computational efficiency of 3km North American hybrid fv3jedi variational analysis

[TingLei-NOAA]

The 3km hybrid fv3jedi variational analysis is a core component in the targeted fv3jedi based RRFS data assimilation system.

Improving its computational efficiency is a major fucus of ongoing RRFS RRFS development at EMC.
Several primary major bottlenecks have been identified and their optimization have been being explored , including the ensemble localization steps, the scalability of the observation related steps issue #489 , IO and other steps

The expert D. Kokron at GDIT had helped finish a parallelized IO fv3jed interface , which has caused substantial improvement in the IO performance as in rdasapp PR #528

This issue is intended to facilitate collaborations among experts at EMC, GDIT and other partner groups to achieve significant speedups for the 3km NA fv3jedi hybrid runs.

[dkokron]

The attached screen shot shows a small portion of time (x-axis) and processes (y-axis). The meaning of 'small' is depicted as the white rectangle on the grey background in the lower right panel. The white cross near the middle of the image (~242s on the timeline) selects what is shown in the stack-trace in the upper-right panel. The stack trace shows that this rank+thread combo (rank=62, thread=4; this experiment use 8 threads/rank) is not doing anything useful. In fact, all the peach and neighboring yellow (mustard?) colors is omp_barrier(). What does this mean? It means this application would be better off running without threads. This conclusion is supported by timings from four runs using 484 MPI ranks spread evenly across 31 nodes. I varied the number of OpenMP threads from 1,2,4,8. Here are the overall walltimes from OOPS_STATS

1 thread/rank Runtime:   1461.40 sec
2 thread/rank Runtime:   1352.52 sec.  ~7.5% speedup
4 thread/rank Runtime:   1302.37 sec
8 thread/rank Runtime:   1303.54 sec

A 7.5% speedup from doubling the number of cores is not a good use of resources.

The next set of runs explored MPI scaling. I ran three experiments, all using a single thread/rank. I adjusted the mpiexec "depth" option to ensure all ranks were spread out on the available sockets.
Scenarios:
484 ranks on 31 nodes (16/node on 30 nodes and 4 ranks on the last node)
484 ranks on 61 nodes (8/node on 60 nodes and 4 ranks on the last nodes)
1936 ranks on 31 nodes (64 ranks on 30 nodes and 16 ranks on the last node)
1936 ranks on 61 nodes (32 ranks/node on 60 nodes and 16 ranks on the last nodes)

484 ranks on 31 nodes Runtime:   1395.74 sec
484 ranks on 61 nodes Runtime:   1302.39 sec (note enough ranks to saturate available memory bandwidth)
1936 ranks on 31 nodes Runtime:   1113.15 sec
1936 ranks on 61 nodes Runtime:    809.73 sec

The nice bump from spreading 1936 ranks across more nodes tells me this application is memory bound.

Does the "DRPCGMinimizer" in the stack trace above mean that JEDI is using a Preconditioned Conjugate Gradient (PCG) algorithm?

[TingLei-NOAA]

@dkorron. Great! I need some time do digest info in your screenshot. Did you notice how many iteration steps were used in your runs? (or a line like "OPS_STATS saber::Localization::multiply : 481482.87 2040 236.0210" in the log files.
I think DRPCG = Derber–Rosati Preconditioned Conjugate Gradient.

[dkokron]

31x16 OOPS_STATS saber::Localization::multiply : 934330.56 2970 314.5894
61x8 OOPS_STATS saber::Localization::multiply : 856113.15 2970 288.2536
31x64 OOPS_STATS saber::Localization::multiply : 552210.24 2610 211.5748
61x32 OOPS_STATS saber::Localization::multiply : 434009.29 2610 166.2871

Conjugate Gradient turns super computers into regular computers. Are there any plans to implement other solvers like multi-grid? Are there plans to port JEDI to GPU where there is a lot more memory bandwidth?

[dkokron]

See attached performance trace highlighting different phases of execution of a 484 rank job run on 31 nodes. I am curious why there is so much time spent "changing resolution" especially before the second outer loop.

[TingLei-NOAA]

@dkokron , in the "dual resolution" run, the inner loops are run on 6km resolutions. Hence, there are a chgres between the low res increment (on inner loop grids) and the model grids (original resolutions). but I need to understand why there was an large part of changres between the first outer loop.

[TingLei-NOAA]

@dkokron Can we have a off-line meeting and you can show us your current findings first?

[dkokron]

Disabling the second update via
no outer loop update: true
Only got rid of the second read/scatter and a dark-red line (representing LocalizationFactory) just to the right of the second read/scatter. The code is still changing resolution between the first and second outer loops. Any idea what this represents? Can we get rid of it?