[Feature]: Opening nwb files with different extension versions

[samuelbray32]

What would you like to see added to PyNWB?

Nwb files created at different time points may use different versions of an extension (e.g. ndx_optogenetics=0.2.0 and =0.3.0). When analyzing data, it would be great to be able to open multiple files with different extension versions within the same python process.

Is your feature request related to a problem?

• __TYPE_MAP will cache the spec for the version imported within python
• If these versions have breaking changes (e.g. renaming or refactoring an nwb object type), opening additional files of an incompatible version wil raise a ConstructError

What solution would you like?

Below is an example of the issue and strategy that worked for me using pynwb=3.1.3, though I haven't extensively tested it. I understand if this would create fragility that you don't want in the main package, but sharing just in case.

import pickle
from pathlib import Path
from typing import Union

import ndx_optogenetics  # import ndx-optogenetics v0.3.0 to register it in the global type map
from hdmf.backends.hdf5 import HDF5IO as _HDF5IO
from hdmf.build import BuildManager
from pynwb import NWBHDF5IO, __resources, get_type_map

PathLike = Union[str, Path]

path = "/stelmo/nwb/raw/Jacob20250619_.nwb"  # file with ndx-optogenetics v0.2.0

try:
    with NWBHDF5IO(path, "r", load_namespaces=True) as io:
        nwbfile = (
            io.read()
        )  # this raises an error due to ndx-optogenetics version conflict
except Exception as e:
    print(f"Error reading NWB file with default IO: {e} \n\n\n")


def backcompatible_io(path: PathLike, **kwargs) -> NWBHDF5IO:
    """Create an io object capable of reading files incompatible with environment

    Creates an isolated BuildManager that loads namespaces cached inside an NWB file.
    These are used to generate an NWBHDF5IO that can read the file.

    This avoids conflicts when a namespace (e.g., 'ndx-optogenetics') is already loaded
    in the global PyNWB type map with a different version.

    Parameters
    ----------
    path
        Path to an NWB file.

    Returns
    -------
    io
        An NWBHDF5IO object configured to read the file.
    """
    # load fresh type map
    with open(__resources["cached_typemap_path"], "rb") as f:
        tm = pickle.load(f)

    tm.copy_mappers(
        get_type_map()
    )  # core mappings; does not force extension versions

    print(tm.namespace_catalog.namespaces)

    # load namespaces from the file into this type map
    _HDF5IO.load_namespaces(
        tm,
        path,
    )

    print(tm.namespace_catalog.namespaces)
    manager = BuildManager(tm)
    return NWBHDF5IO(
        str(path), mode="r", manager=manager, load_namespaces=False
    )


"""Read an NWB file using the namespaces cached in that file."""
with backcompatible_io(path) as io:
    nwbfile = io.read()  # this works now

Do you have any interest in helping implement the feature?

Yes.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this change was not already requested?

[rly]

Hi @samuelbray32 , For reading NWB files with an extension, unless you want to use convenience functions from the extension python package on PyPI, you do not need to import the extension. ndx-optogenetics does not provide convenience functions for reading data (it does provide functionality to make it easier to write the OptogeneticEpochsTable and OptogeneticPulsesTable - which may not be necessary with the latest pynwb).

If you do not import the extension and simply do with NWBHDF5IO(filename, 'r') as io (load_namespaces=True is the default setting), then the extension is loaded into a copy of the type map that is scoped to the io object. Opening a file with a different extension version with a new IO object will not create a conflict.

If you cannot avoid importing the extension, then doing something like what you suggested would work. It may be easier to add a reset_type_map or unload_namespace function for more general usage though. #1009

Would it work for you to not import the extension (which modifies the global type map)?

[samuelbray32]

Ah, good to know that load_namespaces doesn't affect the global, thanks

In general I don't think our case can rely on users not having the module imported. The package they load the nwb files with (spyglass) imports the extensions modules in order to do isinstance checks of the objects within the file in order to map the relevant data to the correct pipelines.

We may be able to refactor to avoid the type checks, but something like unload_namespace would be helpful

[samuelbray32]

Looking more, it appears that the isinstance check only works as we intend when the module is imported before the nwb file is read in (I presume because of use of the global namespace). This may argue for us refactoring the object type search anyways.

[rly]

OK, good to know. Thanks! I recommend using your workaround for now while we work on a fix.

[samuelbray32]

Will do, thanks ryan