Assistance with Debian packaging

[sorenstoutner]

Now that pyinstaller has been accepted into Debian I am working on packaging pyinstaller-hooks-contrib. I have a basic package prepared, but it currently skips a lot of tests. Some of them require components that are not currently in Debian, which I am unlikely to worry about at this time. But others I think can be addressed.

Some of these I have been able to figure out myself, but there are a few where you might be able to help me. My intention is to ask one question at a time.

When I include altair in the build environment, I receive the following error:

RecursionError: maximum recursion depth exceeded

https://salsa.debian.org/soren/pyinstaller-hooks-contrib/-/jobs/7201942#L6013

The python3-altair package in included in the build environment here:

https://salsa.debian.org/soren/pyinstaller-hooks-contrib/-/blob/debian/master/debian/control?ref_type=heads#L13

The version of python-altair in Debian is 5.0.2, which is a bit behind the upstream 5.5.0.

https://tracker.debian.org/pkg/python-altair

If you think the version is the problem I can work on updating python-altair, but this error message doesn't immediately point me toward a version problem.

For reference, builds that don't include python3-altair complete successfully.

https://salsa.debian.org/soren/pyinstaller-hooks-contrib/-/pipelines/826946

[bwoodsend]

Hmm, this is going to be tricky.

First warning is that requirements-test-libraries.txt is so large that we can't come close to installing the whole thing into one environment without dependency version conflicts (I'm guessing Debian removes at least some of more arbitrary constraints so you might manage more than we do?). We do all our testing incrementally, only installing and testing the subset of packages that have released a new version.

Second warning is that PyInstaller does struggle when environments become insanely large where the module->module dependency tree gets so deep that the stack depth exceeds the recursion limit. It's worst in packages that have large numbers of (usually optional or interchangeable) dependencies. You could probably get altair in if you pulled enough other packages out. Or we do implicitly raise the recursion limit for known bad offenders and I wouldn't mind doing the same for others.

pyinstaller-hooks-contrib/_pyinstaller_hooks_contrib/stdhooks/hook-langchain.py

Lines 19 to 25 in 6470ebd

	# Automatically raise recursion limit to ensure it is at least 5000; this attempts to mitigate recursion limit errors
	# caused by some import chains that involve langchain, but also depend on the build environment (i.e., other packages
	# installed in it).
	new_limit = 5000
	if sys.getrecursionlimit() < new_limit:
	logger.info("hook-langchain: raising recursion limit to %d", new_limit)
	sys.setrecursionlimit(new_limit)

Large environments do also slow the builds down. I don't know how much you consider that to be a factor when deciding how many libraries to include?

[sorenstoutner]

I didn't realize that you don't run all the tests each time.

In Debian, there is a CI system called autopkgtest that runs a series of tests every time a package is uploaded (and on every supported architecture), but also can be run every time a dependency is uploaded so that unexpected package breakage can be detected. Depending on hardware, the pyinstaller-hooks-contrib tests that are currently enabled can take 3 hours or more, so I can see why some people wouldn't want to run them all every time. But, if possible, I would like to run all the ones I can get to work in Debian every time.

Is there a single file I can patch to raise the recursion limit to 5000 for every test, or do I need to add the above code to every test that fails?

[rokm]

Is there a single file I can patch to raise the recursion limit to 5000 for every test, or do I need to add the above code to every test that fails?

Hmm, I suppose increasing the recursion limit in tests/conftest.py might do the trick?

But we could also consider simply raising it as part of PyInstaller initialization...

[sorenstoutner]

Thanks. I will try patching tests/conftest.py.

[sorenstoutner]

I have encountered another problem when including freetype in the test environment.

AssertionError: FT library file is not an absolute path!

https://salsa.debian.org/soren/pyinstaller-hooks-contrib/-/jobs/7212831#L6681

[rokm]

I have encountered another problem when including freetype in the test environment.

AssertionError: FT library file is not an absolute path!

https://salsa.debian.org/soren/pyinstaller-hooks-contrib/-/jobs/7212831#L6681

This likely means that freetype library was not collected. I'll set up a VM tomorrow to see if (and if, why) that is the case.

[sorenstoutner]

In case it is helpful, if freetype-py (provided by the python3-freetype binary package in Debian) is not present in the build environment, then the build finishes correctly.

https://salsa.debian.org/soren/pyinstaller-hooks-contrib/-/jobs/7212376

https://tracker.debian.org/pkg/freetype-py

The only difference between the succeeding build above and the failing build from the previous post is the inclusion of python3-freetype, which means pyinstaller-hooks-contrib is detecting its presence at least to some degree.

https://salsa.debian.org/soren/pyinstaller-hooks-contrib/-/commit/be4976e88956172387b1ea6255bac72e1c187f72

[rokm]

The problem is likely that our (as well as upstream) hook is currently assuming that it is dealing with freetype-py wheel, which bundles the freetype shared library. The debian-packaged version probably uses system-provided copy, which is currently not collected by the hook. So either the test should account for that possibility, or we should extend the hook to collect the system copy of the shared library (which is what we do in some other hooks that face similar dilemma).

[sorenstoutner]

That makes sense. I will currently just disable the test and proceed with the others for the time being, although I would like to fully address the issue at some point.

[sorenstoutner]

In case it is helpful to you, these are the contents of the python3-freetype package in Debian.

https://packages.debian.org/sid/all/python3-freetype/filelist

[sorenstoutner]

New question: I receive the following error when including pyproj in the test environment:

pyproj.exceptions.DataDirError: Valid PROJ data directory not found. Either set the path using the environmental variable PROJ_DATA (PROJ 9.1+) | PROJ_LIB (PROJ<9.1) or with `pyproj.datadir.set_data_dir`.

https://salsa.debian.org/soren/pyinstaller-hooks-contrib/-/jobs/7216265#L6586

[bwoodsend]

Feels rather surreal being able to apt-get install python3-pyinstaller...

I notice that it didn't pull in binutils for objdump? You seemed to think it would work out magically when I mentioned it last time. Likewise with python3-dev.

[bwoodsend]

pyproj's issues stem from its data coming from /usr/share (via proj-data) instead of being in the Python package. This is similar to how Conda does it so it should just be a case of extending an existing workaround for us.

[bwoodsend]

I'll try and go through the apt-get install-able packages and see what other trouble I can flush out in a debian:unstable Docker container. Should be quicker that way...

[sorenstoutner]

Feels rather surreal being able to apt-get install python3-pyinstaller...

I notice that it didn't pull in binutils for objdump? You seemed to think it would work out magically when I mentioned it last time. Likewise with python3-dev.

Sorry, I somehow missed when you mentioned that earlier. Are you saying that binutils and python3-dev should be dependencies of python3-pyinstaller?