controller incompatible with pydantic 2.1

[denisri]

Tests on appveyor and gihub (capsul) install the latest version of pydantic. For some time (more than 1 month I guess) tests are failing. They were passing 2 months ago. Controller seems not compatible with the current version of pydantic (2.1.1). If I install locally the latest pydantic, tests fail. If I get back to pydantic 1.10, they pass.
Errors happen when instantiating fields in controllers using add_field() for instance:

======================================================================
ERROR: test_add_field (soma.controller.tests.test_controller.TestController)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\soma_base-6.0.0-py3.10.egg\soma\controller\tests\test_controller.py", line 956, in test_add_field
    c.add_field('toto', field(type_=str))
  File "C:\Python310\lib\site-packages\soma_base-6.0.0-py3.10.egg\soma\controller\controller.py", line 334, in add_field
    super().__setattr__(name, default_value)
  File "C:\Python310\lib\site-packages\pydantic\_internal\_dataclasses.py", line 172, in validated_setattr
    validator.validate_assignment(instance, __field, __value)
pydantic_core._pydantic_core.ValidationError: 1 validation error for EmptyController_dataclass
toto
  Object has no attribute 'toto' [type=no_such_attribute, input_value=<undefined>, input_type=UndefinedClass]
    For further information visit https://errors.pydantic.dev/2.1/v/no_such_attribute

or:

__________________________ ERROR collecting capsul/process/test/test_nipype_wrap.py __________________________
process/test/test_nipype_wrap.py:99: in <module>
    class TestNipypeWrap(unittest.TestCase):
process/test/test_nipype_wrap.py:139: in TestNipypeWrap
    @unittest.skipIf(not init_spm_config(),
process/test/test_nipype_wrap.py:69: in init_spm_config
    capsul_app = Capsul()
application.py:103: in __init__
    c = ApplicationConfiguration(app_name=app_name, user_file=user_file, site_file=site_file)
config/configuration.py:428: in __init__
    super().__init__()
/casa/host/build/python/soma/controller/controller.py:269: in __init__
    super().__init__()
config/configuration.py:232: in __init__
    self.add_builtin_fields()
config/configuration.py:237: in add_builtin_fields
    self.add_field('databases', dict[str, dict],
config/configuration.py:291: in add_field
    super().add_field(name, *args, **kwargs)
/casa/host/build/python/soma/controller/controller.py:334: in add_field
    super().__setattr__(name, default_value)
/casa/home/.local/lib/python3.10/site-packages/pydantic/_internal/_dataclasses.py:172: in validated_setattr
    validator.validate_assignment(instance, __field, __value)
E   pydantic_core._pydantic_core.ValidationError: 1 validation error for ConfigurationLayer_dataclass
E   databases
E     Object has no attribute 'databases' [type=no_such_attribute, input_value={'builtin': {'type': 'red...l_engine_database.rdb'}}, input_type=dict]
E       For further information visit https://errors.pydantic.dev/2.1/v/no_such_attribute

[denisri]

The attribute __pydantic_model__ in Controller (and other classes inheriting pydantic) doesn't exist anymore.
__pydantic_model__.__fields__ may be replaced with __pydantic_fields__. But that's not enough to make it work.

[denisri]

I am trying to use pydantic 2 (2.3 actually) but fail to make it work.
in Controller.__setattr__, the call to super().__setattr__() (which calls the controller dataclass) results in the entire __dict__ of the Controller to be wiped out, and only the current attribute being set is left in the __dict__.

[denisri]

I can't find a solution. I'm giving up, but we must find something otherwise we are stuck with an obsolete version of pydantic, which is not good for the future.

[DimitriPapadopoulos]

Perhaps ray-project/ray#37019 can help.