[RFC 0197] package sets by-name

[quantenzitrone]

This is currently a collection of ideas on how to handle package sets in the future.
Looking for feedback and ideas.

rendered

[Sigmanificient]

First time interacting with an RFC, i hope i am not making comment that are too dumb 😅

[Sigmanificient]

thanks for adding trees ❤️

[Sigmanificient]

i think a drawback from idea 3 is that we end up with attrpath in directory names

[SigmaSquadron]

I was discussing this privately with @Sigmanificient earlier today, and he helped me organise my thoughts on this.

[SigmaSquadron]

I think our implementation plan should be separated into four stages:

1. Create interfaces for package sets that uses the same functions as by-name to callPackage packages from sharded directories, with additional extensions for package set versioning.
2. Update nixpkgs-vet to expand the existing linting checks to the new package sets.
3. Migrate all package sets into the new interface when it makes sense to do so, or move the children of smaller sets into by-name as top-level packages.
4. Add new rules to nixpkgs-merge-bot to allow unprivileged merging to package sets.

We can't move into the next stage before the previous one is complete, so let's not think about the later stages just now.

[SigmaSquadron]

(Answering because this applies to idea 1)

I never thought it made sense for functions and derivation builders to be inside package sets. I think our Go infrastructure has the right idea: move them to the top-level, and version them in the attribute name. fishPlugins.buildFishPlugin can just be moved to the top-level, while python313Packages.buildPythonApplication becomes buildPython313Application on the top-level. We can deal with this at stage 3, where we migrate everything over.

[7c6f434c]

This proposal is awkward with Python where you have CPython and PyPy and then language version-dialects within them, and even worse for Common Lisp where you have same base language but multiple compatible implementations with their completely own versioning.

[quantenzitrone]

i also think keeping functions in package sets is fine

[SigmaSquadron]

(Answering because this applies to idea 1)

I generally think it's a good idea to think of Nixpkgs as having three types of package infrastructure systems:

1. Top-level packages, which nowadays go to by-name;
2. Simple package sets, like fishPlugins, which are easier to implement in this new interface as we can simply call them as we do by-name packages and shove the resulting set into fishPlugins from the top-level.
3. Versioned package sets, like python3Packages, which are by far the most complex. I think we should absolutely get the input of the primary maintainers of these sets here before we move forward. There are many concerns regarding updating versioned sets, and default aliases, as well as edge cases like python3.pkgs and python3Packages. In the end, having something that is no worse to maintain as it was before would be ideal.

[quantenzitrone]

4. Auto generated package sets like haskellPackages.

[quantenzitrone]

i want handle both 2 and 3 with this RFC

[SigmaSquadron]

(Answering because this applies to idea 1)

Some of them are easy enough. If we resolve the aforementioned issues with the python edge cases, it should be a simple matter of moving and renaming files. OCaml should also be pretty simple.

The generated sets, like Haskell, Node, Lisp and R, will be impossible, though. Those packages don't actually exist as separate files in Nixpkgs, and are instead imported from some other upstream.

This is something that again, we'll have to get the input of package set maintainers on. Moving everything over will end up being a case-by-case basis.

[SigmaSquadron]

That seems good to me. As mentioned in #197 (comment), this is conditional on the approval of the current versioned package set maintainers.

[infinisil]

In addition to the comments, also a meta-comment: While it's really great to create RFC documents like these, we don't need the lengthy RFC process anymore to get them approved, because if Nixpkgs committers can't agree on it, we can escalate to the SC-appointed Nixpkgs core team, which has the authority to make a call.

Probably still beneficial for Nixpkgs to keep discussions for such changes outside the Nixpkgs repo itself due to the noise there, but otherwise I'd suggest to PR this to an adr/¹ directory in Nixpkgs.

Footnotes

1. From https://adr.github.io/, which is very similar ↩

[7c6f434c]

What is a package set? There are some that do not have a direct top-level name. I think in general some places of the text is written for a package set that comes with its set of expressions, while there are cases where meaningfully sets share a large fraction of the expressions.

[Eveeifyeve]

Adding potentially an GitHub action bot that runs updates????

[SigmaSquadron]

Updates across package sets are very different. I don't think this should be a goal of this RFC, but instead something that can be developed separately as an updateScript that nixpkgs-update can pick up.