Non-deterministic segfault when using dask threads

[mx-moth]

I am the author of the emsarray library. Back in July 2024 the CI for emsarray started segfaulting non-deterministically but very reliably. I recorded my investigation in csiro-coasts/emsarray#139. I was never able to make a minimal example that demonstrated this issue, however this issue was very repeatable when running the entire test suite.

I narrowed the issue down to the combination of this netCDF4~=1.7.0 library from PyPI, plus the dask library using the thread scheduler. This issue did not occur when installing netCDF 1.7.x from conda-forge, and it did not occur when using other schedulers with dask. I added a work around in CI of changing the dask scheduler and hoped this was a quirk of that netCDF4 1.7.0 PyPI release and it would disappear on the next release. I then promptly forgot about the issue.

I have occasionally recalled the issue over the past year and tested it again, and it continues to fail, including just now using the latest netCDF4 1.7.3 from PyPI, running on Python 3.14, using the latest dependencies for everything. The issue continues to resolve itself using netCDF4 from conda-forge, or by changing the dask scheduler.

I apologise for the size of this minimal example - an entire test suite for a library is larger than I'd hoped - but at least the steps to reproduce it are straight forward! I am running on Ubuntu Linux x64. I do not know whether this happens on Windows or Mac.

$ uname -a
Linux gonzo-hf 6.14.0-36-generic #36~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 15 15:45:17 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

To make a broken environment using netcdf4 from pypi:

$ git clone https://github.com/csiro-coasts/emsarray.git
$ cd emsarray
$ conda env create --name emsarray-tests-pypi --no-default-packages --file ./continuous-integration/environment.yaml
$ conda activate emsarray-tests-pypi
$ conda install -c conda-forge python=3.14 pip
$ pip install -e .[testing]
$ pytest -vv --dask-scheduler threads -m 'not memory_usage'

Example segfault output will be included in comments below (issues have a maximum character limit), although which exact test fails is non-deterministic so your results may vary anyway.

To make a working environment using netcdf4 from conda-forge:

$ git clone https://github.com/csiro-coasts/emsarray.git
$ cd emsarray
$ conda env create --name emsarray-tests-conda-forge --no-default-packages --file ./continuous-integration/environment.yaml
$ conda activate emsarray-tests-conda-forge
$ conda install -c conda-forge python=3.14 pip netcdf4==1.7.3
$ pip install -e .[testing]
$ pytest -vv --dask-scheduler threads -m 'not memory_usage'

For completeness sake, the output of conda list and pip freeze from the above broken pypi environment are attached below, however this has been happening reliably with all netCDF 1.7.x releases on PyPI and all combinations of other dependencies over time

pip freeze

bokeh==3.8.1
Bottleneck==1.6.0
Cartopy==0.25.0
certifi==2025.11.12
cftime==1.6.5
cfunits==3.3.7
charset-normalizer==3.4.4
click==8.3.1
cloudpickle==3.1.2
contourpy==1.3.3
coverage==7.13.0
cycler==0.12.1
dask==2025.11.0
distributed==2025.11.0
-e git+https://github.com/csiro-coasts/emsarray.git@52b899d5ad43701adf0e3cfaba3936a15df22a5d#egg=emsarray
flake8==7.3.0
fonttools==4.61.0
freezegun==1.5.5
fsspec==2025.12.0
geojson==3.2.0
idna==3.11
iniconfig==2.3.0
isort==7.0.0
Jinja2==3.1.6
kiwisolver==1.4.9
librt==0.7.3
locket==1.0.0
lz4==4.4.5
MarkupSafe==3.0.3
matplotlib==3.10.8
mccabe==0.7.0
msgpack==1.1.2
mypy==1.19.0
mypy_extensions==1.1.0
narwhals==2.13.0
netCDF4==1.7.3
numpy==2.3.5
packaging==25.0
pandas==2.3.3
pandas-stubs==2.3.3.251201
partd==1.4.2
pathspec==0.12.1
pillow==12.0.0
platformdirs==4.5.1
pluggy==1.6.0
pooch==1.8.2
psutil==7.1.3
pyarrow==22.0.0
pycodestyle==2.14.0
pyflakes==3.4.0
Pygments==2.19.2
pykdtree==1.4.3
pyparsing==3.2.5
pyproj==3.7.2
pyshp==3.0.3
pytest==9.0.2
pytest-cov==7.0.0
pytest-mpl==0.18.0
python-dateutil==2.9.0.post0
pytz==2025.2
PyYAML==6.0.3
requests==2.32.5
shapely==2.1.2
six==1.17.0
sortedcontainers==2.4.0
tblib==3.2.2
toolz==1.1.0
tornado==6.5.2
types-pytz==2025.2.0.20251108
typing_extensions==4.15.0
tzdata==2025.2
urllib3==2.6.1
xarray==2025.12.0
xyzservices==2025.11.0
zict==3.0.0

conda list

# packages in environment at /home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bokeh                     3.8.1                    pypi_0    pypi
bottleneck                1.6.0                    pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6  
ca-certificates           2025.12.2            h06a4308_0  
cartopy                   0.25.0                   pypi_0    pypi
certifi                   2025.11.12               pypi_0    pypi
cftime                    1.6.5                    pypi_0    pypi
cfunits                   3.3.7                    pypi_0    pypi
charset-normalizer        3.4.4                    pypi_0    pypi
click                     8.3.1                    pypi_0    pypi
cloudpickle               3.1.2                    pypi_0    pypi
contourpy                 1.3.3                    pypi_0    pypi
coverage                  7.13.0                   pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
dask                      2025.11.0                pypi_0    pypi
distributed               2025.11.0                pypi_0    pypi
emsarray                  0.10.0                   pypi_0    pypi
flake8                    7.3.0                    pypi_0    pypi
fonttools                 4.61.0                   pypi_0    pypi
freezegun                 1.5.5                    pypi_0    pypi
fsspec                    2025.12.0                pypi_0    pypi
geojson                   3.2.0                    pypi_0    pypi
geos                      3.12.3               h06de43c_0    conda-forge
idna                      3.11                     pypi_0    pypi
iniconfig                 2.3.0                    pypi_0    pypi
isort                     7.0.0                    pypi_0    pypi
jinja2                    3.1.6                    pypi_0    pypi
kiwisolver                1.4.9                    pypi_0    pypi
ld_impl_linux-64          2.44                 h153f514_2  
libexpat                  2.7.3                hecca717_0    conda-forge
libffi                    3.4.4                h6a678d5_1  
libgcc                    15.2.0              he0feb66_16    conda-forge
libgcc-ng                 15.2.0              h69a702a_16    conda-forge
libgomp                   15.2.0              he0feb66_16    conda-forge
libmpdec                  4.0.0                h5eee18b_0  
librt                     0.7.3                    pypi_0    pypi
libstdcxx                 15.2.0              h934c35e_16    conda-forge
libstdcxx-ng              15.2.0              hdf11a46_16    conda-forge
libudunits2               2.2.28               h40f5838_3    conda-forge
libuuid                   1.41.5               h5eee18b_0  
libxcb                    1.17.0               h9b100fa_0  
libzlib                   1.3.1                hb25bd0a_0  
locket                    1.0.0                    pypi_0    pypi
lz4                       4.4.5                    pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_1  
markupsafe                3.0.3                    pypi_0    pypi
matplotlib                3.10.8                   pypi_0    pypi
mccabe                    0.7.0                    pypi_0    pypi
msgpack                   1.1.2                    pypi_0    pypi
mypy                      1.19.0                   pypi_0    pypi
mypy-extensions           1.1.0                    pypi_0    pypi
narwhals                  2.13.0                   pypi_0    pypi
ncurses                   6.5                  h7934f7d_0  
netcdf4                   1.7.3                    pypi_0    pypi
numpy                     2.3.5                    pypi_0    pypi
openssl                   3.0.18               hd6dcaed_0  
packaging                 25.0                     pypi_0    pypi
pandas                    2.3.3                    pypi_0    pypi
pandas-stubs              2.3.3.251201             pypi_0    pypi
partd                     1.4.2                    pypi_0    pypi
pathspec                  0.12.1                   pypi_0    pypi
pillow                    12.0.0                   pypi_0    pypi
pip                       25.3               pyh0d26453_0  
platformdirs              4.5.1                    pypi_0    pypi
pluggy                    1.6.0                    pypi_0    pypi
pooch                     1.8.2                    pypi_0    pypi
psutil                    7.1.3                    pypi_0    pypi
pthread-stubs             0.3                  h0ce48e5_1  
pyarrow                   22.0.0                   pypi_0    pypi
pycodestyle               2.14.0                   pypi_0    pypi
pyflakes                  3.4.0                    pypi_0    pypi
pygments                  2.19.2                   pypi_0    pypi
pykdtree                  1.4.3                    pypi_0    pypi
pyparsing                 3.2.5                    pypi_0    pypi
pyproj                    3.7.2                    pypi_0    pypi
pyshp                     3.0.3                    pypi_0    pypi
pytest                    9.0.2                    pypi_0    pypi
pytest-cov                7.0.0                    pypi_0    pypi
pytest-mpl                0.18.0                   pypi_0    pypi
python                    3.14.1          h90eec9f_101_cp314  
python-dateutil           2.9.0.post0              pypi_0    pypi
python_abi                3.14                    2_cp314  
pytz                      2025.2                   pypi_0    pypi
pyyaml                    6.0.3                    pypi_0    pypi
readline                  8.3                  hc2a1206_0  
requests                  2.32.5                   pypi_0    pypi
shapely                   2.1.2                    pypi_0    pypi
six                       1.17.0                   pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
sqlite                    3.51.0               h2a70700_0  
tblib                     3.2.2                    pypi_0    pypi
tk                        8.6.15               h54e0aa7_0  
toolz                     1.1.0                    pypi_0    pypi
tornado                   6.5.2                    pypi_0    pypi
types-pytz                2025.2.0.20251108          pypi_0    pypi
typing-extensions         4.15.0                   pypi_0    pypi
tzdata                    2025.2                   pypi_0    pypi
udunits2                  2.2.28               h40f5838_3    conda-forge
urllib3                   2.6.1                    pypi_0    pypi
xarray                    2025.12.0                pypi_0    pypi
xorg-libx11               1.8.12               h9b100fa_1  
xorg-libxau               1.0.12               h9b100fa_0  
xorg-libxdmcp             1.1.5                h9b100fa_0  
xorg-xorgproto            2024.1               h5eee18b_1  
xyzservices               2025.11.0                pypi_0    pypi
xz                        5.6.4                h5eee18b_1  
zict                      3.0.0                    pypi_0    pypi
zlib                      1.3.1                hb25bd0a_0  
zstd                      1.5.7                h11fc155_0  

[mx-moth]

Output from segfault

$ pytest -vv --dask-scheduler threads -m 'not memory_usage'
=================================================================================================================================== test session starts ====================================================================================================================================
platform linux -- Python 3.14.1, pytest-9.0.2, pluggy-1.6.0 -- /home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14
cachedir: .pytest_cache
Matplotlib: 3.10.8
Freetype: 2.6.1
rootdir: /home/hea211/emsarray-netcdf-tests/emsarray
configfile: pyproject.toml
testpaths: tests
plugins: mpl-0.18.0, cov-7.0.0
collected 404 items / 4 deselected / 400 selected

tests/cli/commands/test_clip.py::test_clip Fatal Python error: Segmentation fault

Thread 0x00007ebbb0b626c0 [ThreadPoolExecu] (most recent call first):
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/concurrent/futures/_base.py", line 544 in set_result
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/concurrent/futures/thread.py", line 86 in run
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/concurrent/futures/thread.py", line 116 in _worker
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1024 in run
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1044 in _bootstrap

Thread 0x00007ebbb13636c0 [ThreadPoolExecu] (most recent call first):
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/concurrent/futures/thread.py", line 116 in _worker
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1024 in run
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1044 in _bootstrap

Thread 0x00007ebbb1b646c0 [ThreadPoolExecu] (most recent call first):
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/concurrent/futures/thread.py", line 116 in _worker
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1024 in run
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1044 in _bootstrap

Current thread 0x00007ebbb23656c0 [ThreadPoolExecu] (most recent call first):
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 96 in __setitem__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/array/core.py", line 4680 in load_store_chunk
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 758 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 1078 in execute_graph
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 199 in _execute_subgraph
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 759 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 1078 in execute_graph
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 199 in _execute_subgraph
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 759 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 1078 in execute_graph
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 199 in _execute_subgraph
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/_task_spec.py", line 759 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/local.py", line 258 in execute_task
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/local.py", line 272 in batch_execute_tasks
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/concurrent/futures/thread.py", line 73 in run
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/concurrent/futures/thread.py", line 86 in run
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/concurrent/futures/thread.py", line 119 in _worker
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1024 in run
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/threading.py", line 1044 in _bootstrap

Thread 0x00007ebbef23c740 [pytest] (most recent call first):
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/re/_parser.py", line 861 in _parse
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/re/_parser.py", line 460 in _parse_sub
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/re/_parser.py", line 856 in _parse
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/re/_parser.py", line 460 in _parse_sub
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/re/_parser.py", line 973 in parse
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/re/_compiler.py", line 762 in compile
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/re/__init__.py", line 350 in _compile
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/re/__init__.py", line 289 in compile
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/utils.py", line 2278 in shorten
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/utils.py", line 2270 in __exit__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/base.py", line 661 in compute
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/dask/array/core.py", line 1221 in store
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/xarray/namedarray/daskmanager.py", line 247 in store
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/xarray/backends/common.py", line 416 in sync
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/xarray/backends/writers.py", line 450 in to_netcdf
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/xarray/core/dataset.py", line 2110 in to_netcdf
  File "/home/hea211/emsarray-netcdf-tests/emsarray/src/emsarray/utils.py", line 145 in to_netcdf_with_fixes
  File "/home/hea211/emsarray-netcdf-tests/emsarray/src/emsarray/conventions/_base.py", line 1663 in to_netcdf
  File "/home/hea211/emsarray-netcdf-tests/emsarray/src/emsarray/cli/commands/clip.py", line 53 in handle
  File "/home/hea211/emsarray-netcdf-tests/emsarray/src/emsarray/cli/__init__.py", line 40 in main
  File "/home/hea211/emsarray-netcdf-tests/emsarray/src/emsarray/cli/utils.py", line 135 in wrapper
  File "/home/hea211/emsarray-netcdf-tests/emsarray/tests/cli/commands/test_clip.py", line 22 in test_clip
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/python.py", line 1720 in runtest
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/runner.py", line 179 in pytest_runtest_call
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/runner.py", line 245 in <lambda>
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/runner.py", line 353 in from_call
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/runner.py", line 244 in call_and_report
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/main.py", line 396 in pytest_runtestloop
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/main.py", line 372 in _main
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/config/__init__.py", line 199 in main
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/_pytest/config/__init__.py", line 223 in console_main
  File "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/pytest", line 7 in <module>

Current thread's C stack trace (most recent call first):
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at _Py_DumpStack+0x4a [0x5c7497ebf8a1]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x195d0f [0x5c7497ebfd0f]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x45330 [0x7ebbef045330]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libhdf5-49234e76.so.310.5.1", at H5CX_pop+0x19 [0x7ebbb6ea70d9]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libhdf5-49234e76.so.310.5.1", at H5Screate_simple+0x92 [0x7ebbb70214d2]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libnetcdf-306d3f1b.so.22", at +0xa7f95 [0x7ebbb74b8f95]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libnetcdf-306d3f1b.so.22", at nc4_rec_write_metadata+0xae4 [0x7ebbb74ba614]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libnetcdf-306d3f1b.so.22", at nc4_enddef_netcdf4_file+0x5b [0x7ebbb74bbfcb]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libnetcdf-306d3f1b.so.22", at NC4_put_vars+0xc6 [0x7ebbb74c4306]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libnetcdf-306d3f1b.so.22", at NC4_put_vara+0x11 [0x7ebbb74c4d81]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libnetcdf-306d3f1b.so.22", at +0x78e5e [0x7ebbb7489e5e]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/../netcdf4.libs/libnetcdf-306d3f1b.so.22", at nc_put_vara+0x61 [0x7ebbb748b251]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/_netCDF4.abi3.so", at +0x106a9e [0x7ebbb770da9e]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/_netCDF4.abi3.so", at +0x33739 [0x7ebbb763a739]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at _PyObject_MakeTpCall+0x434 [0x5c7497efe6d4]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x259655 [0x5c7497f83655]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/_netCDF4.abi3.so", at +0x2fb8e [0x7ebbb7636b8e]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/lib/python3.14/site-packages/netCDF4/_netCDF4.abi3.so", at +0xcec61 [0x7ebbb76d5c61]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at _PyEval_EvalFrameDefault+0x54f1 [0x5c7497f1a671]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x2715b7 [0x5c7497f9b5b7]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x275f1b [0x5c7497f9ff1b]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at _PyEval_EvalFrameDefault+0x54f1 [0x5c7497f1a671]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x1e7968 [0x5c7497f11968]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x2c4c44 [0x5c7497feec44]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x31f69f [0x5c749804969f]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at _PyObject_MakeTpCall+0x434 [0x5c7497efe6d4]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at _PyEval_EvalFrameDefault+0x135f [0x5c7497f164df]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x1e7968 [0x5c7497f11968]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x2c4c44 [0x5c7497feec44]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at +0x31f69f [0x5c749804969f]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at _PyObject_MakeTpCall+0x434 [0x5c7497efe6d4]
  Binary file "/home/hea211/.local/share/miniconda3/envs/emsarray-tests-pypi/bin/python3.14", at _PyEval_EvalFrameDefault+0x135f [0x5c7497f164df]
  <truncated rest of calls>

Extension modules: markupsafe._speedups, yaml._yaml, psutil._psutil_linux, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pyarrow.lib, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, bottleneck.move, bottleneck.nonreduce, bottleneck.nonreduce_axis, bottleneck.reduce, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, cftime._cftime, netCDF4._netCDF4, shapely.lib, shapely._geos, shapely._geometry_helpers, pyproj._compat, pyproj._context, pyproj._network, pyproj._version, pyproj._geod, pyproj.list, pyproj._crs, pyproj.database, pyproj._transformer, pyproj._sync, cartopy.trace, PIL._imaging, kiwisolver._cext, tornado.speedups, msgpack._cmsgpack, lz4._version, lz4.block._block (total: 82)
Segmentation fault (core dumped)

[mx-moth]

Looking at various logs over time the test "tests/conventions/test_ugrid.py::test_make_and_apply_clip_mask" seems to fail most often. However running just this test individually is much less likely to trigger the segfault. It happens maybe 50% of the time, compared to almost 100% of the time for the entire test suite:

$ pytest -vv \
    --dask-scheduler threads \
    -- \
    tests/conventions/test_ugrid.py::test_make_and_apply_clip_mask

This test is doing some absolutely heinous indexing of an unstructured grid. Combining this with dask lazily loading dataset chunks in multiple threads and perhaps the huge number of small queries of the dataset in a multithreaded environment expose some rare multithreaded issue in the netCDF library?

[jswhit]

The netcdf-c library is not thread safe. There is some work underway to fix this.

[mx-moth]

OK, that is good to know.

The bit I am most confused about is why this used to work at all then. Are you aware of any locks on the Python side of this module to enforce single threaded access? The test case above has never crashed when using version 1.6.x, and has never crashed when using netcdf installed from conda, even with the rest of the environment identical with an environment that does crash.

xarray + dask will use threads by default when working with chunked netcdf files. I see in their codebases that there are locks around reading and writing to enforce single threaded access. It is possible there is an issue over there causing their locks to malfunction, but the specific nature of this problem only occurring with netCDF4 1.7.x from PyPI makes this feel unlikely to me.

[jswhit]

I suspect it worked 'by accident' before with earlier wheels, due to differences in the netcdf-c library shipped with the wheels.