RuntimeError: Some background workers are no longer alive

[angelw55]

Hi developers,

Thanks for developing this useful package. I have run this package successfully for quite a few T1w MRI images, but I encountered this RuntimeError: Some background workers are no longer alive recently. I have no idea how to solve this issue, please kindly advise.

I post the full error messages here:
Predicting case_ 014:
perform_everything_on_device: False
Traceback (most recent call last):
File "/home/shared/Software/niproject_env/bin/DLMUSE", line 8, in
sys.exit(main())
^^^^^^
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/DLMUSE/main.py", line 335, in main
predictor.predict_from_files(
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 258, in predict_from_files
return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 370, in predict_from_data_iterator
proceed = not check_workers_alive_and_busy(export_pool, worker_list, r, allowed_num_queued=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nnunetv2/utilities/file_path_utilities.py", line 103, in check_workers_alive_and_busy
raise RuntimeError('Some background workers are no longer alive')
RuntimeError: Some background workers are no longer alive
Process SpawnProcess-19:
Traceback (most recent call last):
File "/home/shared/Software/fsl/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/shared/Software/fsl/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nnunetv2/inference/data_iterators.py", line 58, in preprocess_fromfiles_save_to_queue
raise e
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nnunetv2/inference/data_iterators.py", line 50, in preprocess_fromfiles_save_to_queue
target_queue.put(item, timeout=0.01)
File "", line 2, in put
File "/home/shared/Software/fsl/lib/python3.11/multiprocessing/managers.py", line 837, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:

Traceback (most recent call last):
File "/home/shared/Software/fsl/lib/python3.11/multiprocessing/managers.py", line 260, in serve_client
self.id_to_local_proxy_obj[ident]
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: '7f280b9f4f90'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/shared/Software/fsl/lib/python3.11/multiprocessing/managers.py", line 262, in serve_client
raise ke
File "/home/shared/Software/fsl/lib/python3.11/multiprocessing/managers.py", line 256, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
~~~~~~~~~^^^^^^^
KeyError: '7f280b9f4f90'

Traceback (most recent call last):
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nibabel/loadsave.py", line 101, in load
stat_result = os.stat(filename)
^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/hkcend_hpc/derivatives/nichart_dlmuse-1.0.7/preproc_outputs_Lijun/QE_ses-001/temp_working_dir/s4_dlmuse/sub-QAD140_ses-001_desc-preproc_T1w_DLMUSE.nii.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/shared/Software/niproject_env/bin/NiChart_DLMUSE", line 8, in
sys.exit(main())
^^^^^^
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/NiChart_DLMUSE/main.py", line 250, in main
run_pipeline(
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/NiChart_DLMUSE/dlmuse_pipeline.py", line 178, in run_pipeline
apply_relabel_rois(
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/NiChart_DLMUSE/RelabelROI.py", line 94, in apply_relabel_rois
relabel_rois(in_img, roi_map, label_from, label_to, out_img)
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/NiChart_DLMUSE/RelabelROI.py", line 32, in relabel_rois
in_nii = nib.load(in_img)
^^^^^^^^^^^^^^^^
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nibabel/loadsave.py", line 103, in load
raise FileNotFoundError(f"No such file or no access: '{filename}'")
FileNotFoundError: No such file or no access: '/home/hkcend_hpc/derivatives/nichart_dlmuse-1.0.7/preproc_outputs_Lijun/QE_ses-001/temp_working_dir/s4_dlmuse/sub-QAD140_ses-001_desc-preproc_T1w_DLMUSE.nii.gz'

Thanks for your support!

Best,
Angel

[AlexanderGetka-cbica]

Hi Angel,

Thanks for your interest in NiChart_DLMUSE.

The "some background workers are no longer alive" message usually means that a process was killed due to running out of available memory. Then since that process produces no output, the program can't find the file it needs later.

At the cost of having fewer workers to process the images, and hence it taking a little longer, we can cut down the number of workers, using the --dlicv_args and --dlmuse_args arguments in the below example command:

NiChart_DLMUSE -i {input t1 dir} -o {output dir} -d cuda --dlicv_args "-npp 1 nps 1" --dlmuse_args "-npp 1 -nps 1"

This will make sure that there's only one worker, with access to your entire memory pool, so you shouldn't see this error any longer. Of course, it'll also help to close down external memory-intensive processes.

Let me know if this helps.

Thanks,
Alex Getka

[angelw55]

Hi Alex,

Thanks for your prompt reply and support!

I haven't set up the "cuda" in my system, so I used "cpu" instead. The command likes below:

NiChart_DLMUSE \
-i {input_dir} \
-o {output_dir} \
-d cpu \
--dlicv_args "-npp 1 nps 1" \
--dlmuse_args "-npp 1 -nps 1"

But this comes up this error:
Arguments:
Namespace(in_dir='/home/hkcend_hpc/derivatives/nichart_dlmuse-1.0.7/preproc_inputs_Lijun/QE_ses-001', out_dir='/home/hkcend_hpc/derivatives/nichart_dlmuse-1.0.7/preproc_outputs_Lijun/QE_ses-001', device='cpu', cores=1, bids=False, clear_cache=False, dlmuse_args='-npp 1 -nps 1', dlicv_args='-npp 1 nps 1', refaced_data=False, help=False)

Emptying output folder: /home/hkcend_hpc/derivatives/nichart_dlmuse-1.0.7/preproc_outputs_Lijun/QE_ses-001...
usage:
DLICV v1.0
ICV calculation for structural MRI data.

    Required arguments:
        [-i, --in_dir]   The filepath of the input directory
        [-o, --out_dir]  The filepath of the output directory
    Optional arguments:
        [-device]        cpu|cuda|mps - Depending on your system configuration (default: cuda)
        [-h, --help]    Show this help message and exit.
        [-V, --version] Show program's version number and exit.
    EXAMPLE USAGE:
        DLICV  -i           /path/to/input                        -o           /path/to/output                       -device      cpu|cuda|mps

DLICV: error: unrecognized arguments: nps 1
Traceback (most recent call last):
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nibabel/loadsave.py", line 101, in load
stat_result = os.stat(filename)
^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/hkcend_hpc/derivatives/nichart_dlmuse-1.0.7/preproc_outputs_Lijun/QE_ses-001/temp_working_dir/s2_dlicv/sub-QAD061_ses-001_run-001_desc-preproc_T1w_DLICV.nii.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/shared/Software/niproject_env/bin/NiChart_DLMUSE", line 8, in
sys.exit(main())
^^^^^^
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/NiChart_DLMUSE/main.py", line 250, in main
run_pipeline(
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/NiChart_DLMUSE/dlmuse_pipeline.py", line 148, in run_pipeline
apply_mask_img(df_img, in_dir, in_suff, mask_dir, mask_suff, out_dir, out_suff)
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/NiChart_DLMUSE/MaskImage.py", line 187, in apply_mask_img
mask_img(in_img, in_mask, out_img)
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/NiChart_DLMUSE/MaskImage.py", line 83, in mask_img
nii_mask = nib.load(mask_img)
^^^^^^^^^^^^^^^^^^
File "/home/shared/Software/niproject_env/lib/python3.11/site-packages/nibabel/loadsave.py", line 103, in load
raise FileNotFoundError(f"No such file or no access: '{filename}'")
FileNotFoundError: No such file or no access: '/home/hkcend_hpc/derivatives/nichart_dlmuse-1.0.7/preproc_outputs_Lijun/QE_ses-001/temp_working_dir/s2_dlicv/sub-QAD061_ses-001_run-001_desc-preproc_T1w_DLICV.nii.gz'

Can the --dlicv_args and --dlmuse_args arguments be applied to "cpu" device?

Thanks,
Angel

[angelw55]

Hi Alex,

I noticed that I missed the "-" before nps 1. The correct comment should be:

NiChart_DLMUSE \
-i {input_dir} \
-o {output_dir} \
-d cpu \
--dlicv_args "-npp 1 -nps 1" \
--dlmuse_args "-npp 1 -nps 1"

It keeps running!

Thanks,
Angel