Proposal: Split the all-packages.nix file

[matthiasbeyer]

The all-packages.nix file is currently 15.1k lines long. This is too mach IMHO.

As far as I can tell, there are several sections, one for the environment, one for normal packages, one for libraries.

I think it would be a good idea to split this enormous file. While doing this, one could also sort the packages, as they are not entirely sorted as far as I can tell.

Any objections? Maybe I can try to do this in the next few days.

[copumpkin]

I think there was already a branch attempting mostly eliminate the big list via readDir and friends. I'd love it if it worked. Main thing to watch out for is being as lazy as possible so you don't accidentally traverse the entire filesystem with every evaluation, and providing a sensible overrides mechanism when you don't just want to pass in the default values in the callPackage scope.

I feel a bit more torn about just splitting it into smaller files, since I then no longer have an easy place to search for everything, and still have to manually maintain a big list o' stuff.

[matthiasbeyer]

@copumpkin I'd simply split them files and not try to invent something. Searching would still be possible via grep -r or ack or ag.

[vcunat]

Recently we had a long discussion about the manual(s) – whether they should be single-page or split. And that was just about readability of the result. There are subsets already not in that single file (haskell, python, etc.), but it still simplifies some things. Another problem is that the categories are rather ad-hoc, and recently we had a serious proposal to abandon them and just sort the whole file alphabetically. (Note that e.g. large fraction of packages contain some library used by another package.)

BTW, that Eelco's attempt wasn't much about splitting, IIRC, but mainly about getting rid of most of the lines completely. If we just split it up, we would need to read all the parts always, but that would be about the same speed as now, as long as it's only several files.

I don't yet see a solution that would be decisively and significantly better than what we have now.

[copumpkin]

@vcunat do you have a link to Eelco's attempt and some explanation (for posterity) of what was wrong with it?

[vcunat]

No, sadly, I can't find it ATM and I'm not even sure why it wasn't used in the end.

[vcunat]

Here, I found some ML reference http://comments.gmane.org/gmane.linux.distributions.nixos/16928 to ece61b7.

[matthiasbeyer]

@vcunat I like the approach the commit introduces, actually.

[matthiasbeyer]

Closing due to no progress anymore.

[fgaz]

That commit was reverted shortly after: e729559

What was the reason for that?

[matthiasbeyer]

I guess you should ask the committer, @edolstra

[edolstra]

It created an ugly asymmetry between packages with and without overrides. So to override a package you had to override both auto-packages.nix and all-packages.nix.