Dependency link problem with ruamel.yaml #2695
Description
https://pypi.org/project/ruamel.yaml/ has a few issues with PyPI/packaging/versioning. Their info on PyPI (also for future reference copied here):
Copied text
breaking changes, that may make future uploads to PyPI impossible
If you are interested in future upgrades of ruamel.yaml please check the documentation on installing, since at some point I might not be able to upload a new version to PyPI with updated information.
ruamel.yaml was intentionally named as yaml in a namespace ruamel. The namespace allows the installation name to correspond unchanged to how the package is imported, reduces the number of links I have to create in site-packages of a Python install during development, as well as providing a recognisable set of packages my company releases to the public.
However, after uploading version 0.18.7, I got an email from PyPI, about having to change the project name to ruamel_yaml to comply with PEP 625, sometime in the future. The email doesn't say if namespace packages are no longer allowed, or how to deal with the very real clash with the pre-existing package ruamel_yaml.
I might not be able to adapt ruamel.yaml, in a way that does not negatively affect the 0.5 million daily downloads (and my own usage of the package) in time. My experience with other such service downgrades (Bitbucket, Readthedocs), has not been entirely positive.
Starting with 0.19.1 ruamel.yaml no longer has a dependency on ruamel.yaml.clibz nor on the old ruamel.yaml.clib.
Some deployment issues were reported on 0.19.0, due to the lack of proper pinning of the ruamel.yaml version used. Most of these issues had to do with the environment not having updated setuptools installed and the build dependency setuptools-zig for ruamel.yaml.clibz not being invoked, with Python falling back to invoking gcc (which was even less likely to be installed).
The already indicated simple solution of using:
python -m pip install --no-deps ruamel.yaml ruamel.yaml.clib
turned out not to work for at least one setup.
As I am not aware that you can create an install requirement that removes a dependency, the default (Cython) dependency is removed and you should use ruamel.yaml[libyaml] resp. ruamel.yaml[oldlibyaml] as your requirements. (this is the preferred way over using ruamel.yaml.clibz and ruamel.yaml.clib directly). If you are using ruamel.yaml in its default (round-trip, YAML(typ='rt')) mode, there is currently no advantage of installing either optional extension.
The C sources are functionally unchanged, but they are now always compiled (using setuptools-zig and ziglang) on your system, instead of being downloaded as pre-compiled wheels (if available). For this to function properly your Python (virtual) environment needs to have an up-to-date version of setuptools and wheels pre-installed.
The code to load ruamel.yaml.clib has priority over ruamel.yaml.clibz if both are installed. This compatibility will at least be available during the 0.19 ruamel.yaml series (so pin your usage of ruamel.yaml if necessary and report any problems).
The motivitation for this change is the availability, and easy of use, of Zig as the toolchain (in the form of ziglang on PyPi), so lenghty, non-optimized, pre-compilation and uploading to PyPI, is no longer necessary. The time spent on creating ~60 wheels and even more time wasted on dealing with CI providers (Appveyor not being updated to support 3.14, Github CI being slow, and charging for the use of your own computer, etc).
The split out of ruamel.yaml.clib after the 0.15.100 release, was also motivated by the time spent on generating .whl files even if only Python code was changed. The use of ziglang and setuptools-zig does make re-integration of the C sources into ruamel.yaml feasable, but there are no plans yet to make this happen.
The test matrix for ruamel.yaml.clibz, of course still has many dimensions:
Python versions: 3.9 - 3.14
OS-es: Linux, Alpine (musl), macOS, Windows
Architectures: Intel/AMD, Arm (and others), in 64 and some also in 32 bit versions
Zig version: ziglang < 0.16 is taken from PyPI
I try to test as much of the combinations as possible, trying at least all supported Python versions, including freethreading, on macOS-arm64, Linux arm64 (via docker containers), Ubuntu Linux-Intel, Linux musl intel (docker). And at least one Python version along each of the indicated positions of the dimensions above (e.g. Windows10 64bit was tested with Python 3.14, but I could not test the RISC-V architecture). As with generating .whl files previously (which I could not all test myself) I partly have to rely on the process of compilation/generation being likely correct, and feedback from actual users, of exotic (for me) platforms, is of course welcome.
There is new section, in the documentation, on the security of processing unchecked input.
The potentially breaking change announced for the 0.18 series, in that YAML(typ='unsafe') was going to be deprecated (now pending), has not yet been implemented, but is still considered. If you only use unsafe to dump, please use the new YAML(typ='full'), the result of that can be safely loaded with a default instance YAML(), as that will get you inspectable, tagged, scalars, instead of executed Python functions/classes. (You should probably add constructors for what you actually need, but I do consider adding a ruamel.yaml.unsafe package that will re-add the typ='unsafe' option. Please adjust/pin your dependencies accordingly if necessary.
Version 0.18.16 was the last one tested to be working with Python 3.8. Version 0.18.9 was the last one tested to be working with Python 3.7. Version 0.17.21 was the last one tested to be working on Python 3.5 and 3.6. The 0.16.13 release was the last that was tested to be working on Python 2.7.
There are two extra plug-in packages (ruamel.yaml.bytes and ruamel.yaml.string) for those not wanting to do the streaming to a io.BytesIO/StringIO buffer themselves.
If your package uses ruamel.yaml and is not listed on PyPI, drop me an email, preferably with some information on how you use the package (or a link to the repository) and I'll keep you informed when the status of the API is stable enough to make the transition.
For packaging purposes you can use a download of the tar balls of tagged source
At first I thought this won't have much impact, until short after I ran into a problem.
That was on Ubuntu 22.04 with intelmq installed with pip/from a local repo. Calling any intelmq program yielded:
# intelmqsetup
Traceback (most recent call last):
File "/usr/local/bin/intelmqsetup", line 33, in <module>
sys.exit(load_entry_point('intelmq', 'console_scripts', 'intelmqsetup')())
File "/usr/local/bin/intelmqsetup", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/opt/intelmq_src/intelmq/bin/intelmqsetup.py", line 25, in <module>
import pkg_resources
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3267, in <module>
def _initialize_master_working_set():
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3241, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3279, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 573, in _build_master
ws.require(__requires__)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 891, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 777, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ruamel.yaml' distribution was not found and is required by intelmq
ruamel.yaml was of course installed:
# pip install ruamel.yaml
Requirement already satisfied: ruamel.yaml in /usr/local/lib/python3.10/dist-packages (0.19.1)
I could work around it by replacing ruamel.yaml with ruamel_yaml in intelmq.egg-info/requires.txt
I'm not yet sure if we need to or should replace that now in the requires, as that behaviour could depend on the local versions of the packaging tooling.
Needs some research :/