Open files on Windows and feasibility of in-process upgrades

[jaimergp]

Windows famously disallows two processes from handling the same open file. That means that if a process has opened a file, it might block its manipulation til its closed and released.

This can present an opportunity to create errors in upgrade processes. We need to ensure that:

• If a certain package has been already imported, we can safely upgrade it to a new version without errors and that the module has been reloaded in-process without breaking anything. This might be tricky to get right so the best choice might be to defer the update to the next napari execution, or to report the user that a restart is needed.
• There might be a block on write to the opened files. What kind of message do we report here? Can we remember which package failed to install and try again in the next napari run before anything else runs? How can we do that? Should we register an atexit task with the function to be run?
• and maybe other issues we are not aware of.

The action items involve:

• Auditing the described scenarios on Windows and making sure the observed issues can be reproduced
• Come up with a robust workaround strategy
• Implement the solution

[dalthviz]

So I did some checks on Windows and here are the findings:

Package manager\Case	Change imported module (i.e numpy to version 1.26 from version 2.0.2) while napari is running
pip	Warning message is displayed related with a temporary directory being unable to be removed automatically. Seems like while napari is open those directories can't be removed. Closing napari unlocks them: Also, there is an inconsistency on the version displayed over napari itself. Checking the version over the console I see the old one (2.0.2)but from the napari info dialog the actually current installed one is displayed
conda	The installation command is executed without showing any warningStill the same behavior observed with pip happens related with displayed package version discrepancy between console and napari info dialog

Package manager\Case	Change napari version while napari is running (i.e napari to version 0.5.1 from version 0.5.2 )
pip	An error is shown related with permissions for a file (in this case loading.gif)Trying again the install command shows not only the previous error but a warning related with an invalid distribution (probably from the previous failed installation attempt)The invalid distribution directory can be removed while napari still is running. Closing napari enables you to proceed with the installation command but you still see the warning related with an invalid distribution being ignored in case you haven't removed it yet
conda	Initially a message related with a frozen solve failing was the only thing shown but after more than +30 min conda did proceed with the operation although showing a warning message related with a file being unable to be removed or changed (seems like it refers to the same loading.gif) Checking, over the still open napari application instance, the version shown is 0.5.2. Closing and relaunching napari shows the correct/changed version (0.5.1)

Also, some elements that came to mind that maybe could be worthy to explore related with reloading modules (different from napari itself) over the console/napari-console:

• Related with the napari-console:
  • autoreload magic: https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html

Edit: Tried the autoreload magic but that caused several errors:

Details

``` [autoreload of numpy.core failed: Traceback (most recent call last): File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 276, in check superreload(m, reload, self.old_objects) File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 455, in superreload if not append_obj(module, old_objects, name, obj): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 423, in append_obj in_module = hasattr(obj, "__module__") and obj.__module__ == module.__name__ ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\core\defchararray.py", line 2, in __getattr__ from numpy._core import defchararray File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_core\defchararray.py", line 25, in from numpy.strings import * File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\strings\__init__.py", line 1, in from numpy._core.strings import __all__, __doc__ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_core\strings.py", line 63, in MAX = np.iinfo(np.int64).max ^^^^^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\core\getlimits.py", line 687, in __init__ NameError: name 'numeric' is not defined ] [autoreload of numpy._typing._scalars failed: Traceback (most recent call last): File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 276, in check superreload(m, reload, self.old_objects) File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 475, in superreload module = reload(module) ^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\importlib\__init__.py", line 169, in reload _bootstrap._exec(spec, module) File "", line 621, in _exec File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_typing\_scalars.py", line 12, in _BoolLike_co = Union[bool, np.bool] ^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\__init__.py", line 338, in __getattr__ # Warn for expired attributes ^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'numpy' has no attribute 'bool'. `np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations ] [autoreload of numpy._typing._dtype_like failed: Traceback (most recent call last): File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 276, in check superreload(m, reload, self.old_objects) File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 475, in superreload module = reload(module) ^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\importlib\__init__.py", line 169, in reload _bootstrap._exec(spec, module) File "", line 621, in _exec File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_typing\_dtype_like.py", line 142, in type[np.bool], ^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\__init__.py", line 338, in __getattr__ # Warn for expired attributes ^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'numpy' has no attribute 'bool'. `np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations ] [autoreload of numpy._typing._array_like failed: Traceback (most recent call last): File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 276, in check superreload(m, reload, self.old_objects) File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 475, in superreload module = reload(module) ^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\importlib\__init__.py", line 169, in reload _bootstrap._exec(spec, module) File "", line 621, in _exec File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_typing\_array_like.py", line 97, in dtype[np.bool], ^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\__init__.py", line 338, in __getattr__ # Warn for expired attributes ^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'numpy' has no attribute 'bool'. `np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations ] C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_typing\_scalars.py:12: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.! C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_typing\_dtype_like.py:142: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.! C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_typing\_array_like.py:97: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.! C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\ma\core.py:79: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.! [autoreload of numpy.ma.core failed: Traceback (most recent call last): File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 276, in check superreload(m, reload, self.old_objects) File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 475, in superreload module = reload(module) ^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\importlib\__init__.py", line 169, in reload _bootstrap._exec(spec, module) File "", line 621, in _exec File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\ma\core.py", line 79, in MaskType = np.bool ^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\__init__.py", line 338, in __getattr__ # Warn for expired attributes ^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'numpy' has no attribute 'bool'. `np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations ] [autoreload of numpy failed: Traceback (most recent call last): File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 276, in check superreload(m, reload, self.old_objects) File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 455, in superreload if not append_obj(module, old_objects, name, obj): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\IPython\extensions\autoreload.py", line 423, in append_obj in_module = hasattr(obj, "__module__") and obj.__module__ == module.__name__ ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\core\defchararray.py", line 2, in __getattr__ from numpy._core import defchararray File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_core\defchararray.py", line 25, in from numpy.strings import * File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\strings\__init__.py", line 1, in from numpy._core.strings import __all__, __doc__ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\_core\strings.py", line 63, in MAX = np.iinfo(np.int64).max ^^^^^^^^^^^^^^^^^^ File "C:\Users\dalth\anaconda3\envs\napari-pip\Lib\site-packages\numpy\core\getlimits.py", line 687, in __init__ NameError: name 'numeric' is not defined ] ```

Maybe a kernel restart is what is needed for the napari-console? 🤔

Notes:

• The testing was done by creating two conda environments with Python 3.11 from conda-forge (napari-pip and napari-conda). Over one (napari-pip) packages and operations where done using only pip while over the other one (napari-conda) only conda was used and packages were installed from conda-forge.
• I would say that for napari to be updated regardless of using pip or conda you need to first close the application itself. Maybe a way to handle that case is to check if napari is running before a command related with an update or version change is going to be executed, ask for permission to close napari, run the update/version change command and then launch again napari? (I guess all this logic should be handled by the installation manager, right? 🤔 )

[jaimergp]

I see! This confirms my suspicions, but in a surprising way. I was expecting errors with dynamic libraries and other code resources, but not static files. Makes sense, though.

I think we will have to do something like this:

• Run the installation with a --dry-run and see which packages would be updated. That is: which are already installed and will suffer a modification (potentially affecting static files).
• If there are no conflicts, we proceed as normal.
• If we find potential conflicts, we generate a lockfile and save it to disk. We will have to add some pre-startup logic (or post-exit one) so this lockfile is processed once napari has quit fully and the file handles have been freed. An alternative would be to write a command that the user can copy-paste (or a script! this might be easier) in a terminal once they have quit napari manually.

[dalthviz]

Just in case, checked again and, at least with pip, I see that under the directories that the warnings mention (~umpy and ~umpy.libs for example when updating a package like numpy) you find some .pyd files (which I think are kind of like .dll?):

[jaimergp]

I wonder were that ~umpy stuff comes from, is it a temporary directory while trying to install the package?

Seems to come from this logic, which is used to generate adjacent temporary directories during uninstall. Maybe this is because Windows allows renaming but not removing open handles? 🤔 (Not sure about that part).