Add ability to do recover with MFEM

.gitignore

@@ -229,6 +229,7 @@ framework/contrib/asio/
 *.cpr
 *.cpa.gz-*
 *.cpr-*
+*_cp/
 
 # Ignore petsc arch
 petsc/arch-*/*

framework/doc/content/source/mfem/auxkernels/MFEMAuxKernel.md

@@ -8,12 +8,12 @@ Base class for MFEM auxkernels used to evaluate real auxiliary variables to the
 
 ## Overview
 
-MFEM auxkernels are responsible for updating real auxiliary variables in the system, during pre- or
-post-processing steps.
+MFEM auxkernels are responsible for updating real auxiliary variables in the system during MFEM
+execution passes.
 
-An `MFEMAuxKernel` is derived from `MFEMGeneralUserObject`, and thus the order of their execution
-can be controlled similar to other MOOSE UserObjects using the `execution_order_group` input
-parameter.
+An `MFEMAuxKernel` is derived from `MFEMExecutedObject`. Its ordering relative to other MFEM
+executed objects is determined automatically from detected data dependencies rather than by manual
+execution groups.
 
 `MFEMAuxKernel` is a base class. Derived classes should override the `execute`
  method to update the `_result_var` during execution.

framework/doc/content/source/mfem/auxkernels/MFEMComplexAuxKernel.md

@@ -8,12 +8,12 @@ Base class for MFEM auxkernels used to evaluate complex auxiliary variables to t
 
 ## Overview
 
-Complex MFEM auxkernels are responsible for updating complex auxiliary variables in the system, during pre- or
-post-processing steps.
+Complex MFEM auxkernels are responsible for updating complex auxiliary variables in the system
+during MFEM execution passes.
 
-An `MFEMComplexAuxKernel` is derived from `MFEMGeneralUserObject`, and thus the order of their execution
-can be controlled similar to other MOOSE UserObjects using the `execution_order_group` input
-parameter.
+An `MFEMComplexAuxKernel` is derived from `MFEMExecutedObject`. Its ordering relative to other MFEM
+executed objects is determined automatically from detected data dependencies rather than by manual
+execution groups.
 
 `MFEMComplexAuxKernel` is a base class. Derived classes should override the `execute`
  method to update the `_result_var` during execution.

framework/doc/content/source/mfem/postprocessors/MFEMPostprocessor.md

@@ -8,17 +8,12 @@ Base class for MFEM postprocessors used to evaluate a single scalar.
 
 ## Overview
 
-MFEM postprocessors calculate scalar quantities from the
-(aux)variables, typically after each timestep.
+MFEM postprocessors calculate scalar quantities from the (aux)variables, typically after each
+timestep.
 
-An `MFEMPostprocessor` is derived from `MFEMGeneralUserObject`.
-Therefore, the order of their execution can be controlled similar to other
-MOOSE UserObjects using the `execution_order_group` input parameter, e.g.,
-to require the execution of a postprocessor computing on an [AuxVariable.md]
-strictly after the execution of the [MFEMAuxKernel.md] computing the variable
-field itself. For example:
-
-!listing mfem/kernels/irrotational.i block=AuxKernels Postprocessors/velocity_error
+An `MFEMPostprocessor` is derived from `MFEMExecutedObject`. Its ordering relative to MFEM initial
+conditions, aux kernels, transfers, and other MFEM postprocessors is determined automatically from
+detected data dependencies instead of manual execution groups.
 
 `MFEMPostprocessor` is a purely virtual base class. Derived classes
 should override the `execute` and `getValue` methods.

framework/doc/content/source/mfem/vectorpostprocessors/MFEMComplexPointValueSampler.md

@@ -0,0 +1,25 @@
+# MFEMComplexPointValueSampler
+
+!if! function=hasCapability('mfem')
+
+## Overview
+
+This class performs accurate interpolation of complex scalar and vector quantities at a
+set of specified points using MFEM's
+[`FindPointsGSLIB`](https://mfem.org/howto/findpts/) `gslib` interpolation.
+The real and imaginary parts of each sampled component are output as separate columns.
+
+## Example Input File Syntax
+
+!listing mfem/kernels/nl_heatconduction_complex_aux.i block=VectorPostprocessors
+
+!syntax parameters /VectorPostprocessors/MFEMComplexPointValueSampler
+
+!syntax inputs /VectorPostprocessors/MFEMComplexPointValueSampler
+
+!syntax children /VectorPostprocessors/MFEMComplexPointValueSampler
+
+!if-end!
+
+!else
+!include mfem/mfem_warning.md

framework/doc/content/source/mfem/vectorpostprocessors/MFEMVectorPostprocessor.md

@@ -8,15 +8,12 @@ Base class for MFEM vectorpostprocessors used to evaluate an array of values.
 
 ## Overview
 
-MFEM vectorpostprocessors calculate an array of values from the (aux)variables,
-typically after each timestep.
-
-An `MFEMVectorPostprocessor` is derived from `MFEMGeneralUserObject`.
-Therefore, the order of their execution can be controlled similar to other
-MOOSE UserObjects using the `execution_order_group` input parameter, e.g., to
-require the execution of a vectorpostprocessor computing on an [AuxVariable.md]
-strictly after the execution of the [MFEMAuxKernel.md] computing the variable
-field itself.
+MFEM vectorpostprocessors calculate an array of values from the (aux)variables, typically after
+each timestep.
+
+An `MFEMVectorPostprocessor` is derived from `MFEMExecutedObject`. Its ordering relative to other
+MFEM executed objects is determined automatically from detected data dependencies instead of manual
+execution groups.
 
 `MFEMVectorPostprocessor` is a virtual base class. Derived classes should use
 the `VectorPostprocessor::declareVector` method to get a reference to a vector

framework/include/interfaces/DependencyResolverInterface.h

@@ -65,6 +65,11 @@ class DependencyResolverInterface
   /**
    * A helper method for cyclic errors.
    */
+  template <typename T, typename T2, typename NameFunc>
+  static void cyclicDependencyError(CyclicDependencyException<T2> & e,
+                                    const std::string & header,
+                                    NameFunc && name_func);
+
   template <typename T, typename T2>
   static void cyclicDependencyError(CyclicDependencyException<T2> & e, const std::string & header);
 };
@@ -124,18 +129,27 @@ DependencyResolverInterface::sortDFS(typename std::vector<T> & vector)
   vector = sorted;
 }
 
-template <typename T, typename T2>
+template <typename T, typename T2, typename NameFunc>
 void
 DependencyResolverInterface::cyclicDependencyError(CyclicDependencyException<T2> & e,
-                                                   const std::string & header)
+                                                   const std::string & header,
+                                                   NameFunc && name_func)
 {
   std::ostringstream oss;
 
   oss << header << ":\n";
   const auto cycle = e.getCyclicDependencies();
   std::vector<std::string> names(cycle.size());
   for (const auto i : index_range(cycle))
-    names[i] = static_cast<T>(cycle[i])->name();
+    names[i] = name_func(cycle[i]);
   oss << MooseUtils::join(names, " <- ");
   mooseError(oss.str());
 }
+
+template <typename T, typename T2>
+void
+DependencyResolverInterface::cyclicDependencyError(CyclicDependencyException<T2> & e,
+                                                   const std::string & header)
+{
+  cyclicDependencyError<T>(e, header, [](const auto & obj) { return static_cast<T>(obj)->name(); });
+}

framework/include/mesh/MooseMesh.h

@@ -25,6 +25,7 @@
 #include "ElemInfo.h"
 
 #include <memory> //std::unique_ptr
+#include <filesystem>
 #include <unordered_map>
 #include <unordered_set>
 
@@ -180,6 +181,15 @@ class MooseMesh : public MooseObject, public Restartable, public PerfGraphInterf
    */
   virtual void buildMesh() = 0;
 
+  /**
+   * Write the mesh files needed for recovery/checkpointing.
+   *
+   * The base implementation writes the libMesh checkpoint mesh.
+   * Derived classes may extend this to write additional backend-specific files.
+   */
+  virtual std::vector<std::filesystem::path>
+  writeRecoveryFiles(const std::filesystem::path & file_base) const;
+
   /**
    * Returns MeshBase::mesh_dimension(), (not
    * MeshBase::spatial_dimension()!) of the underlying libMesh mesh
@@ -1557,6 +1567,8 @@ class MooseMesh : public MooseObject, public Restartable, public PerfGraphInterf
   bool getDisplaceNodeListBySideList() { return _displace_node_list_by_side_list; }
 
 protected:
+  bool recoveryAllowed() const { return _allow_recovery; }
+
   /// Deprecated (DO NOT USE)
   std::vector<std::unique_ptr<libMesh::GhostingFunctor>> _ghosting_functors;
 

framework/include/mfem/auxkernels/MFEMAuxKernel.h

@@ -11,13 +11,13 @@
 
 #pragma once
 
-#include "MFEMGeneralUserObject.h"
+#include "MFEMExecutedObject.h"
 #include "MFEMContainers.h"
 
 /**
  * Class to construct an auxiliary solver used to update a real auxvariable.
  */
-class MFEMAuxKernel : public MFEMGeneralUserObject
+class MFEMAuxKernel : public MFEMExecutedObject
 {
 public:
   static InputParameters validParams();
@@ -26,7 +26,9 @@ class MFEMAuxKernel : public MFEMGeneralUserObject
   virtual ~MFEMAuxKernel() = default;
 
   /// Method called to update any owned objects upon an FE space update
-  virtual void update() {};
+  virtual void update() {}
+
+  virtual std::set<std::string> producedVariableNames() const override;
 
 protected:
   /// Name of auxvariable to store the result of the auxkernel in.

framework/include/mfem/auxkernels/MFEMComplexAuxKernel.h

@@ -11,13 +11,13 @@
 
 #pragma once
 
-#include "MFEMGeneralUserObject.h"
+#include "MFEMExecutedObject.h"
 #include "MFEMContainers.h"
 
 /**
  * Class to construct an auxiliary solver used to update a complex auxvariable.
  */
-class MFEMComplexAuxKernel : public MFEMGeneralUserObject
+class MFEMComplexAuxKernel : public MFEMExecutedObject
 {
 public:
   static InputParameters validParams();
@@ -28,6 +28,8 @@ class MFEMComplexAuxKernel : public MFEMGeneralUserObject
   /// Method called to update any owned objects upon an FE space update
   virtual void update() {};
 
+  virtual std::set<std::string> producedVariableNames() const override;
+
   /// Method to add a scaled complex variable to another complex variable.
   void complexAdd(mfem::ParComplexGridFunction & a,
                   const mfem::ParComplexGridFunction & b,