Pyproject.toml-based installation, updated dependencies, and automatic Docker builds

[cailmdaley]

The shapepipe and sp_validation packages have inconsistent version requirements, causing dependency versions to change when installing both packages in the same environment. This PR attempts to make the dependencies of the two packages consistent (and adds a few new dependencies).

While I was at it, I switched form the old setup.py installation method to the modern pyproject.toml approach.

I also plan to set up automatic GitHub builds of a docker container for the package, building on top of the container for the shapepipe package.

[sfarrens]

Hey @cailmdaley, looking good so far. 👏 I added a few comments on things you can look into. One general point to note is that we probably won't want/need any pinned dependencies in the pyproject.toml as we can tag versions of the Docker image instead.

[cailmdaley]

thank you for the helpful comments Sam. to your general point, wouldn't it be better to specify python dependencies in the pyproject.toml since those dependencies should apply outside the container as well?

[cailmdaley]

I think this is ready to be merged pending review, I will do one more run through the whole script to make sure everything works.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[sfarrens]

Hi @cailmdaley, my apologies for being so slow to finalise this review. I have run the following tests:

1. pyproject.toml

• Build on CC-IN2P3 ✅ - build succeeds but I had to mess with the versions of scipy and statsmodels to import the package
• Build on macOS ❌ - build fails

3. Dockerfile

• Apptainer build on CC-IN2P3 ✅ - works out of the box
• Local Docker build on macOS ❓- it has not crashed but the build has been running for a very long time, certainly too long for real world users
• Docker pull of pre-built image from ghcr ✅ - no problem pulling, however I would argue that the resulting image is basically unusable because of how slow it runs on Apple Silicon
• I will try one final local build test on my Ubuntu machine tonight just for comparison

In summary, if the scope of this PR is simply to add a pyproject.toml setup and a Dockerfile that works only for what it absolutely has to work on, then I would say go ahead and merge this PR but just keep in mind that some future PRs will be needed to opened to make this a more usable tool for the community.

[cailmdaley]

Thank you Sam! I really appreciate you taking the time to check this. It's good to hear that the Docker build works on another system and a plain pip installation almost works out of the box.

Yes I think making things work for MacOS is beyond the scope of this PR. Making it usable more widely would be nice longer-term, but for now I think it's most important to get these changes merged so we can use them for the ongoing UNIONS cosmic shear analyses. I will go ahead and merge.