Memory problem for PL step on compute cluster

[leomarconato]

Hi,
and thank you for developing such a nice and robust package for PSDS processing.

I am trying to run dolphin on a large stack of Sentinel-1 SLCs (300 acquisition dates), in ISCE format (.slc + .slc.rsc), cropped to a size of 1600 × 6400 pixels.
I am able to run the phase linking step on this stack using fairly large subsampling values, i.e. stride_x = 16 and stride_y = 4.
I run it on 32 cores and it only takes about 10 minutes.
However, when I try to run it at full resolution (both strides set to 1), the jobs are systematically killed due to memory issues, usually after several hours of computation.
I have tried several things, such as reducing the chunk size down to 256 × 256 and even 128 × 128. This makes the process run longer, but it still eventually saturates the RAM.
I also tried doubling the number of reserved cores with respect to the number of processes I allow, and modifying the JAX memory preallocation environment variables:

export XLA_PYTHON_CLIENT_PREALLOCATE=false
export XLA_PYTHON_CLIENT_MEM_FRACTION=0.5

Unfortunately, I still cannot get a run that completes the phase linking step successfully.
Do you have any ideas or tips to reduce the memory usage of the phase linking step without reducing the computing performance too much?
Thanks in advance!

Here are a few relevant parameters from my config file, in case they are helpful:

worker_settings:
  gpu_enabled: false
  threads_per_worker: 1
  n_parallel_bursts: 32
  block_shape:
    - 128
    - 128
output_options:
  strides:
    y: 1
    x: 1
ps_options:
  amp_dispersion_threshold: 0.25
amplitude_dispersion_files: []
amplitude_mean_files: []
layover_shadow_mask_files: []
phase_linking:
  ministack_size: 15
  max_num_compressed: 10
  output_reference_idx:
  half_window:
    y: 5
    x: 20
  use_evd: false
  beta: 0.0
  zero_correlation_threshold: 0.0
  shp_method: glrt
  shp_alpha: 0.001
  mask_input_ps: false
  baseline_lag:
  compressed_slc_plan: always_first
  write_crlb: true
  write_closure_phase: false

[scottstanie]

thanks for the detailed report!

It looks lie you already shrunk the block size a little, which is good (and usually necessary when trying without any strides.)

What i might do is

• Turn n_parallel_bursts down to 16, or even ~10. Each "burst" (a block really for ISCE input format) uses more like 2 CPU (one for phase linking, one for background reading/writing). So if you want to use 32 CPUs, you only need 10-15 parallel blocks.
• Try setting max_num_compressed: 5. There won't be huge benefit beyond the most recent 5 compressed slcs, and if it's dying after a few hours, that means it's probably once the ministack size reaches 15+10 real+compressed SLCs. The 25x25 covariance matrices will take more than double the memory of the initial 15x15 covariance matrices.

Let me know if those changes don't work!

[leomarconato]

Thank you for your quick answer!

I have tried several things, including n_parallel_bursts down to 1/4th of the reserved cores and max_num_compressed down to 5, and even booking the entire node, which should have ~250 Go of memory, I get this error :

[INFO|single|L108] 2026-02-01T15:19:16+0100: Processing 15 SLCs + 6 compressed SLCs. 
[INFO|single|L113] 2026-02-01T15:19:16+0100: Total stack size (in pixels): (21, 1600, 6400)
Phase linking (/scratch/marconal/dolphin_fullres/phase_linking/linked_phase):  91%|█████████ | 592/650 [39:08<02:13,  2.30s/it]E0201 15:58:25.597714 2959144 execution_engine.cc:54] LLVM compilation error: Cannot allocate memory
E0201 15:58:25.598001 2959140 execution_engine.cc:54] LLVM compilation error: JITDylib <xla_jit_dylib_13> is defunct
E0201 15:58:25.598855 2959118 execution_engine.cc:54] LLVM compilation error: Resource tracker 0x7fa5c0d7e1f0 became defunct
/home/marconal/.conda/envs/dolphin-env-gpu/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
./run_dolphin_fullres.sh: line 30: 2958869 Segmentation fault      dolphin run dolphin_fullres_config.yaml

Maybe there is a problem of configuration on my workstation, but I checked and there should not be any memory restriction.

I will see if I have the possibility to configure a GPU, do you think it may help ?

[scottstanie]

oh! You are running this on a large CPU cluster? Sorry I didn't pick that up the first time.

Whenever @mirzaees and I have run on a large sever (~200 CPUs), we have had to use the taskset command to pin it to a smaller number of CPUs.
It's been a long-running issue with JAX to reliably pin the CPU/thread count when running on a big machine, and so far, the only way I've found to avoid errors like yours is limiting it outside of python at the linux level.

So if you are doing 4 bursts at a time, i might do something like taskset -c 0-31 dolphin run dolphin_config.yaml to ensure only 32 cpus

[scottstanie]

if this works for you, we should certainly add it to the to-be-made FAQ #512

[leomarconato]

My apologies, the setup I am using was not very clear in my first message.
Thanks, I tried to limit the job to 32 CPUs using taskset, but I am still encountering the following error: LLVM compilation error: Cannot allocate memory.
What puzzles me is that it always occurs at the same stage of the processing (at the end of the phase linking of the 7th mini-stack).
I also tried running it on a GPU, and the same error occurs at exactly the same point.
However, it seems that restarting dolphin from that step allows the computation to continue (until now, I was restarting from scratch), so resetting the memory and compilation cache probably helps.
I will keep you posted on whether I manage to complete the entire process over several runs.

[leomarconato]

Actually this was with max_num_compressed set to 5.
I'm not so sure to understand what this parameter actually sets, since whatever its value, dolphin goes up to 15 SLCs + 19 compressed SLCs for my stack of 300 SLCs (20 ministacks).

[scottstanie]

Oh interesting, if that is happening then that is a bug that we should fix

[leomarconato]

I ran a test with strong strides, modifying the max_num_compressed parameter from 5 to 10 to 20, and it seems that the steps, running time, and memory use are not affected at all.
I don't know if this is what you expect or not, but I can run other tests if you think it can be useful.

===> grep max_num_compressed dolphin_test_config.yaml
  max_num_compressed: 5
===> dolphin run dolphin_test_config.yaml
[INFO|single|L108] 2026-02-05T15:01:07+0100: Processing 15 SLCs + 0 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:01:26+0100: Processing 15 SLCs + 1 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:01:41+0100: Processing 15 SLCs + 2 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:01:57+0100: Processing 15 SLCs + 3 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:02:15+0100: Processing 15 SLCs + 4 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:02:33+0100: Processing 15 SLCs + 5 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:02:53+0100: Processing 15 SLCs + 6 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:03:14+0100: Processing 15 SLCs + 7 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:03:36+0100: Processing 15 SLCs + 8 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:03:59+0100: Processing 15 SLCs + 9 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:04:23+0100: Processing 15 SLCs + 10 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:04:48+0100: Processing 15 SLCs + 11 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:05:13+0100: Processing 15 SLCs + 12 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:05:40+0100: Processing 15 SLCs + 13 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:06:07+0100: Processing 15 SLCs + 14 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:06:38+0100: Processing 15 SLCs + 15 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:07:08+0100: Processing 15 SLCs + 16 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:07:40+0100: Processing 15 SLCs + 17 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:08:02+0100: Processing 15 SLCs + 18 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:08:20+0100: Processing 15 SLCs + 19 compressed SLCs. 
[INFO|displacement|L333] 2026-02-05T15:08:42+0100: Maximum memory usage: 10.29 GB
[INFO|_log|L102] 2026-02-05T15:08:42+0100: Total elapsed time for dolphin.workflows.displacement.run : 8.36 minutes (501.83 seconds)
===> rm -rf dolphin_test
===> sed -i -e 's/max_num_compressed: 5/max_num_compressed: 10/g' dolphin_test_config.yaml
===> grep max_num_compressed dolphin_test_config.yaml
  max_num_compressed: 10
===> dolphin run dolphin_test_config.yaml
[INFO|single|L108] 2026-02-05T15:09:38+0100: Processing 15 SLCs + 0 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:09:55+0100: Processing 15 SLCs + 1 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:10:11+0100: Processing 15 SLCs + 2 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:10:27+0100: Processing 15 SLCs + 3 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:10:44+0100: Processing 15 SLCs + 4 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:11:02+0100: Processing 15 SLCs + 5 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:11:20+0100: Processing 15 SLCs + 6 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:11:41+0100: Processing 15 SLCs + 7 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:12:03+0100: Processing 15 SLCs + 8 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:12:26+0100: Processing 15 SLCs + 9 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:12:47+0100: Processing 15 SLCs + 10 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:13:11+0100: Processing 15 SLCs + 11 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:13:36+0100: Processing 15 SLCs + 12 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:14:02+0100: Processing 15 SLCs + 13 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:14:30+0100: Processing 15 SLCs + 14 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:15:00+0100: Processing 15 SLCs + 15 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:15:31+0100: Processing 15 SLCs + 16 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:16:01+0100: Processing 15 SLCs + 17 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:16:23+0100: Processing 15 SLCs + 18 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:16:43+0100: Processing 15 SLCs + 19 compressed SLCs. 
[INFO|displacement|L333] 2026-02-05T15:17:05+0100: Maximum memory usage: 10.31 GB
[INFO|_log|L102] 2026-02-05T15:17:05+0100: Total elapsed time for dolphin.workflows.displacement.run : 8.25 minutes (494.99 seconds)
===> rm -rf dolphin_test
===> sed -i -e 's/max_num_compressed: 10/max_num_compressed: 20/g' dolphin_test_config.yaml
===> grep max_num_compressed dolphin_test_config.yaml
  max_num_compressed: 20
===> dolphin run dolphin_test_config.yaml
[INFO|single|L108] 2026-02-05T15:17:59+0100: Processing 15 SLCs + 0 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:18:16+0100: Processing 15 SLCs + 1 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:18:32+0100: Processing 15 SLCs + 2 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:18:48+0100: Processing 15 SLCs + 3 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:19:06+0100: Processing 15 SLCs + 4 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:19:24+0100: Processing 15 SLCs + 5 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:19:43+0100: Processing 15 SLCs + 6 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:20:04+0100: Processing 15 SLCs + 7 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:20:26+0100: Processing 15 SLCs + 8 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:20:49+0100: Processing 15 SLCs + 9 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:21:13+0100: Processing 15 SLCs + 10 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:21:37+0100: Processing 15 SLCs + 11 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:22:03+0100: Processing 15 SLCs + 12 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:22:29+0100: Processing 15 SLCs + 13 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:22:58+0100: Processing 15 SLCs + 14 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:23:27+0100: Processing 15 SLCs + 15 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:23:59+0100: Processing 15 SLCs + 16 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:24:31+0100: Processing 15 SLCs + 17 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:24:53+0100: Processing 15 SLCs + 18 compressed SLCs. 
[INFO|single|L108] 2026-02-05T15:25:12+0100: Processing 15 SLCs + 19 compressed SLCs. 
[INFO|displacement|L333] 2026-02-05T15:25:34+0100: Maximum memory usage: 10.35 GB
[INFO|_log|L102] 2026-02-05T15:25:34+0100: Total elapsed time for dolphin.workflows.displacement.run : 8.37 minutes (502.39 seconds)

[scottstanie]

thanks persisting and providing great detail. #683 should fix this issue and allow you to run it successfully.

[leomarconato]

Indeed, your fix seems to reduce memory use by a factor ~2 and processing time by a factor ~3 when I set max_num_compressed to 5, which is much valuable!
Thank you!