Use a supplied communicator group and not MPI_COMM_WORLD

[ss421]

While testing malak with XIOS in server mode, I encounter an error towards the end of the application that appear to be related to the VarObs and CX Writer. The error is not very helpful so I tested without the writer filters and the application runs. I suspect this is to do with the communicator group because when running in server mode, the jedi application does not use the MPI_COMM_WORLD comm, instead it uses a split communicator where some PEs are used by the server.

I did a search for MPI_COMM_WORLD and found a few references, the following in particular:

/// \note This filter must only be used with ObsSpaces using the \c MPI_COMM_WORLD communicator,
/// otherwise a deadlock will occur while writing the VarObs file. This is due to a limitation of
/// the \c Ops_WriteVarobs function, which could be removed by replacing \c mpl_comm_world in the
/// call to \c Ops_Mpl_Gatherv by \c mpi_group (for consistency with all other MPI calls in \c
/// Ops_WriteVarobs).

see: https://github.com/MetOffice/opsinputs/blob/2720b0b5d3ec2a129e27475d0fc6911547b4de17/src/opsinputs/VarObsWriter.h#L46C1-L50C22

suggest that there would be some additional work that is required if we want XIOS server + varWriter. I don't imagine that this is too much work. Adding a few people that may know:

Wojciech Śmigaj (@wsmigaj) David Simonin (@DavidSimonin) Chris Thomas (@ctgh) Michael Cooke (@mikecooke77) Adam Maycock (@adammaycock) David Davies (@DJDavies2)

if others have an ideas then please comment below.

Thanks, Steve.

[mikecooke77]

Given that XIOS is only going to be with LFRic this will not be a problem operationally. However we will still want this for testing VAR vs JADA and therefore I support this work.

Do you know if this is a problem with both the VarObs writer and CXWriter? Or is it just the varobs?

What you suggest makes perfect sense. It would be good to get someone with some experience of working to debug mpi issues to work on this (or perhaps someone who wants to learn a bit more about MPI).

[ss421]

Given that XIOS is only going to be with LFRic this will not be a problem operationally. However we will still want this for testing VAR vs JADA and therefore I support this work.

Do you know if this is a problem with both the VarObs writer and CXWriter? Or is it just the varobs?

What you suggest makes perfect sense. It would be good to get someone with some experience of working to debug mpi issues to work on this (or perhaps someone who wants to learn a bit more about MPI).

This is only an issue if the server is used. I am led to understand that we will need that when we get to C896 but I don't know any details at the moment.

I didn't dig too far so I'm not sure but I suspect both. The initial error appeared to be related to CX (I used a diff from a working test case) but the comment above is in the varobs. I could do a test to see if each is working.

I don't think it should be too difficult, my guess is the MPI_WORLD_COMM is hard coded somewhere likely in a module, it would just need to be passed the actual group.

[ss421]

Doing a diff on a working Verizon (no-server) and the version with the server enabled. I can see that the error appears during the varobs processing. For context, in the working version, the following is printed:

 VarobPGEs % RadialVelocity Allocate OK
PGE structure after Ops_VarobPGEs
 ************** START PRINT OF ModelOB header ***************
 ModelOB: Validity Time =-32768-32768-32768-32768     0     0-32768
 ModelOB: ObsGroup = 38
         ModelOB columns           1         248         495         742         989        1236        1483
              BG % Tskin -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000
 BG % RadialVelocity   1 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000
 ************** END PRINT OF ModelOB header ***************
BG % Tskin Deallocate OK
BG % RadialVelocity Deallocate OK
1672 observations output to varobs file
247000 observations not output to varobs file
LatLon array resized from 3344 to 4096
Times array resized from 1672 to 2048
<a href="http://www-nwp/~opsrc/OPS/view/dev/doc/ops_get_env_character.html">ops_get_env_character</a>
Environment variable OPS_CREATE_VAROBS_DONE_FILE hasn't been set
Using default value:
Ob % Latitude Deallocate OK
Ob % Longitude Deallocate OK
Ob % Time Deallocate OK

while with the server enabled, the following (omitting the lines above 247000 observations not...) is printed:

247000 observations not output to varobs file
Application 327001397 exit codes: 139
Application 327001397 exit signals: Killed
Application 327001397 resources: utime ~156s, stime ~149s, Rss ~1883948, inblocks ~12753617, outblocks ~2758561
[WARN] lfricjedi_hofx.x returned an error 137
[INFO] Checking that output has been produced, otherwise task fails
Check JEDI output ............ FAIL

Removing the write for both varobs and CX (see: https://github.com/MetOffice/jjdocs/commit/5082277dc246ad817762804377ba4360d6d6f482) and the application runs.

When adding back the CX writer (https://github.com/MetOffice/jjdocs/commit/726e53a6ad3eb8eac7026e0b1b9c20473db35ebd) I get a time-out failure rather than a hard failure like this. The failure in that case is likely due to miss-match in communicator group - I think it is likely waiting for synchronization across the "world comm" which is clearly not going to happen.

switching to just VARobs writer (https://github.com/MetOffice/jjdocs/commit/be21419c8060767ae1b82efe7b2434006a4fe284) and I get the same failure as documented above.

[ctgh]

Steven Sandbach (@ss421) Is this still a problem for malak?

[mikecooke77]

I think this is an issue should we want to run VAR with LFRic in server mode for XIOS. I guess the question is will we use server mode with JOPA? or just JADA?

[ss421]

My understanding is that we will want to use a server to read the C896 file but we have not done any testing related to that so Im not 100% sure. I think that would be an optimization rather than not being able to run it at all.

[mikecooke77]

Looking at the timings for the high resolution benchmark this will need to be done as a lot of the JOPA tasks are taking 50 mins rather than less than 10 mins. They need to be competitve with the UM model interface ones.