AMR segmentation fault

[suniverse]

Hi,

I am trying to run a hydro simulation with AMR. It seems that if I would like to use more than one refinement level, I ran into the memory issue. A segmentation fault from InterpolateBoundaryFromParent. There is no issue is I only use one refinement level. The error message is below.

[tri0004:3048316:0:3048316] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
[tri0004:3048302:0:3048302] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace (tid:3048433) ====
 0 0x0000000000038790 __sigaction()  ???:0
 1 0x0000000000690d29 grid::InterpolateBoundaryFromParent()  /home/xinyuli/enzo-dev/src/enzo/Grid_InterpolateBoundaryFromParent.C:443
 2 0x00000000007b9950 SetBoundaryConditions()  /home/xinyuli/enzo-dev/src/enzo/SetBoundaryConditions.C:134
 3 0x00000000005c9a34 SetAccelerationBoundary()  /home/xinyuli/enzo-dev/src/enzo/Grid_AccelerationBoundaryRoutines.C:152
 4 0x00000000004d7da5 EvolveLevel()  /home/xinyuli/enzo-dev/src/enzo/EvolveLevel.C:547
 5 0x00000000004d8a8d EvolveLevel()  /home/xinyuli/enzo-dev/src/enzo/EvolveLevel.C:819
 6 0x00000000004d8a8d EvolveLevel()  /home/xinyuli/enzo-dev/src/enzo/EvolveLevel.C:819
 7 0x00000000004c698d EvolveHierarchy()  /home/xinyuli/enzo-dev/src/enzo/EvolveHierarchy.C:506
 8 0x0000000000407de2 main()  /home/xinyuli/enzo-dev/src/enzo/enzo.C:793
 9 0x000000000002394a __libc_init_first()  ???:0
10 0x0000000000023a05 __libc_start_main()  ???:0
11 0x0000000000407051 _start()  ???:0

[bwoshea]

@suniverse can you please give us more information about what you're doing? It's impossible to diagnose the problem with what you've provided thus far. Please attach:

1. The Enzo parameter file you're using
2. The entire command line argument you're using for Enzo, including any MPI arguments
3. Information about the system on which you're running, including node information (number of CPU cores, memory, etc.) and compiler type/version/etc. The output of "make show-config" and "make show-flags" would be quite helpful as well.
4. Any additional output from Enzo prior to the seg fault. If you run enzo with the debug flag turned on (enzo -d <parameter file name>), it will likely produce some useful information. Do this with the same MPI arguments you used for the run that caused the seg fault!

[suniverse]

@bwoshea Hi Brian, below are a few files for my output (est.txt) and parameter file (SCol.txt). I found the problem always occurs when the AMR levels become larger than one, independent of what criteria I am using.

job.sh

est.txt

SCol.txt

My configurations below

MACHINE: Trillium
   MACHINE-NAME: scinet-trillium
   PARAMETER_MAX_SUBGRIDS  [max-subgrids-###]                : 100000
   PARAMETER_MAX_BARYONS  [max-baryons-###]                  : 30
   PARAMETER_MAX_TASKS_PER_NODE  [max-tasks-per-node-###]    : 8
   PARAMETER_MEMORY_POOL_SIZE  [memory-pool-###]             : 100000
   CONFIG_PRECISION  [precision-{32,64}]                     : 64
   CONFIG_PARTICLES  [particles-{32,64,128}]                 : 64
   CONFIG_INTEGERS  [integers-{32,64}]                       : 64
   CONFIG_PARTICLE_IDS  [particle-id-{32,64}]                : 64
   CONFIG_INITS  [inits-{32,64}]                             : 64
   CONFIG_IO  [io-{32,64}]                                   : 32
   CONFIG_USE_MPI  [use-mpi-{yes,no}]                        : yes
   CONFIG_TASKMAP  [taskmap-{yes,no}]                        : no
   CONFIG_PACKED_AMR  [packed-amr-{yes,no}]                  : yes
   CONFIG_PACKED_MEM  [packed-mem-{yes,no}]                  : no
   CONFIG_LCAPERF  [lcaperf-{yes,no}]                        : no
   CONFIG_PAPI  [papi-{yes,no}]                              : no
   CONFIG_PYTHON  [python-{yes,no}]                          : no
   CONFIG_NEW_PROBLEM_TYPES  [new-problem-types-{yes,no}]    : no
   CONFIG_ECUDA  [cuda-{yes,no}]                             : no
   CONFIG_OOC_BOUNDARY  [ooc-boundary-{yes,no}]              : no
   CONFIG_ACCELERATION_BOUNDARY  [acceleration-boundary-{yes,no}] : yes
   CONFIG_OPT  [opt-{warn,debug,cudadebug,high,aggressive}]  : debug
   CONFIG_TESTING  [testing-{yes,no}]                        : no
   CONFIG_PHOTON  [photon-{yes,no}]                          : yes
   CONFIG_HYPRE  [hypre-{yes,no}]                            : no
   CONFIG_EMISSIVITY  [emissivity-{yes,no}]                  : no
   CONFIG_USE_HDF4  [use-hdf4-{yes,no}]                      : no
   CONFIG_NEW_GRID_IO  [newgridio-{yes,no}]                  : yes
   CONFIG_BITWISE_IDENTICALITY  [bitwise-{yes,no}]           : no
   CONFIG_FAST_SIB  [fastsib-{yes,no}]                       : yes
   CONFIG_GRAVITY_4S  [gravity-4s-{yes,no}]                  : no
   CONFIG_ENZO_PERFORMANCE  [enzo-performance-{yes,no}]      : yes
   CONFIG_GRACKLE  [grackle-{yes,no}]                        : no
   CONFIG_LOG2ALLOC  [log2alloc-{yes,no}]                    : no
   CONFIG_UUID  [uuid-{yes,no}]                              : yes

and flags

CPP = /cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcc12/openmpi/4.1.5/bin/mpicc
CC  = /cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcc12/openmpi/4.1.5/bin/mpicc
CXX = /cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcc12/openmpi/4.1.5/bin/mpicxx
FC  = /cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcc12/openmpi/4.1.5/bin/mpif77
F90 = /cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcc12/openmpi/4.1.5/bin/mpif90
LD  = /cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcc12/openmpi/4.1.5/bin/mpicxx
DEFINES = -DLINUX -DH5_USE_16_API -DWINDS -DIO_LOG -DOMPI_SKIP_MPICXX  -D__max_subgrids=100000 -D__max_baryons=30 -D__max_cpu_per_node=8 -D__me
mory_pool_size=100000 -DINITS64 -DLARGE_INTS -DCONFIG_PINT_8 -DIO_32    -DUSE_MPI   -DCONFIG_PFLOAT_8 -DCONFIG_BFLOAT_8  -DUSE_HDF5_GROUPS   -D
TRANSFER   -DNEW_GRID_IO -DFAST_SIB      -DENZO_PERFORMANCE    -DUSE_UUID -DSAB
INCLUDES =  -I/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcc12/openmpi/4.1.5/include          -I.
CPPFLAGS = -P -traditional
CFLAGS   =  -O0 -g
CXXFLAGS =  -O0 -g
FFLAGS   = -fno-second-underscore -ffixed-line-length-132 -std=legacy -O0 -g
F90FLAGS = -fno-second-underscore -std=legacy -O0 -g
LDFLAGS  =  -O0 -g
LIBS     = -L/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcc12/hdf5/1.14.2/lib -lhdf5 -lz  -lstdc++ -lgfortran