10081 packages disapeared from packages.json

[davidak]

Describe the bug

https://repology.org/repository/nix_unstable

was something changed about the exported package list?

Expected behavior
same number of packages or even more

Additional context
we used to have the most packages, but now AUR is on top again

https://repology.org/repositories/statistics/total

[tu-maurice]

Maybe #110346 is part of this.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-has-been-the-largest-repository-for-months/10667/15

[Atemu]

Might also be a redefinition of a "project" on the repology side, i.e. duplicates that weren't detected before. Projects != Packages.

Do we know the revisions of before and after? We could check whether anything went missing from packages.json in the tarball because that's the source of Repology's data.

[tu-maurice]

Well it must've happened on January 16th so we could use one commit from the 15th and 17th and see if there are suspicious changes in the that json file in between.

Using git log --pretty=format:'%h %cD %s' --grep "Merge pull request" we can pick a merge commit from the 15th e.g. 809dc63 and one from the 17th e.g. faed1fc

[Atemu]

We need to know the state of the unstable channel on those dates, it's days or sometimes even weeks behind master.

Don't really have time for this right now but https://channels.nix.gsc.io/nixos-unstable/history is probably what we need.

[davidak]

Well it must've happened on January 16th

It did appear in Repology on 16. January, but our channel take some time after a commit was done. So i would check like the 9. Jan..

There was a successful channel evaluation on 2021-01-07.

Channel evaluation: https://hydra.nixos.org/build/134563331 (f211631)
packages file: https://hydra.nixos.org/build/134563595

[davidak@gaming:~/Downloads]$ nix run nixpkgs.brotli -c brotli --decompress packages-2021-01-07.json.br 

[davidak@gaming:~/Downloads]$ jq '.packages | length' packages-2021-01-07.json 
78766

https://hydra.nixos.org/build/134705011 (257cbbc)
https://hydra.nixos.org/build/134703692

[davidak@gaming:~/Downloads]$ jq '.packages | length' packages-2021-01-10.json 
78849

https://hydra.nixos.org/build/135156379 (b3616bd)
https://hydra.nixos.org/build/135155191

[davidak@gaming:~/Downloads]$ jq '.packages | length' packages-2021-01-16.json
69776

https://hydra.nixos.org/build/135174384 (68398d2)
https://hydra.nixos.org/build/135174479

[davidak@gaming:~/Downloads]$ jq '.packages | length' packages-2021-01-17.json 
69784

Latest unstable channel evaluation:

https://hydra.nixos.org/build/137005286 (6b1057b)
https://hydra.nixos.org/build/137004487

[davidak@gaming:~/Downloads]$ jq '.packages | length' packages-2021-02-17.json 
70202

So the number of packages decreased by 8,990 between 2021-01-07 (f211631) and 2021-01-16 (b3616bd).

There was this change of emacsPackages to packages-config.nix on 2021-01-13 (#107152)

# get only the package names
[davidak@gaming:~/Downloads]$ jq '.packages | keys' packages-2021-01-07.json >packages-2021-01-07.txt

# see the diff
[davidak@gaming:~/Downloads]$ meld packages-2021-01-07.txt packages-2021-01-16.txt

And it's indeed the emacs packages that are missing on 2021-01-16!

Number of emacs2[67]Packages:

[davidak@gaming:~/Downloads]$ egrep 'emacs2[67]Packages' packages-2021-01-07.txt | wc -l
10081

Number of packages that are in packages-2021-01-07.txt but not in packages-2021-01-16.txt:

[davidak@gaming:~/Downloads]$ diff packages-2021-01-07.txt packages-2021-01-16.txt | grep '<' | wc -l
10466

Full list: https://gist.github.com/davidak/8bee60e5485259b42a21d9b7916093da

For example emacs26Packages.abgaben. It does not exist in packages-2021-02-17.json.

[davidak@gaming:~/Downloads]$ jq '.packages | keys' packages-2021-02-17.json | grep abgaben

It does exist today in the search: https://search.nixos.org/packages?channel=unstable&show=emacs26Packages.abgaben&from=0&size=50&sort=relevance&query=abgaben

But not on Repology: https://repology.org/projects/?search=abgaben&maintainer=&category=&inrepo=nix_unstable&notinrepo=&repos=&families=&repos_newest=&families_newest=

We should really be consistent here. The packages that nix can install from nixpkgs channel, search.nixos.org and Repology should be identical!

@adisbladis @garbas

[adisbladis]

@davidak emacs26Packages, emacs27Packages and so on were moved to their respective interpreters as emacs26.pkgs & emacs27.pkgs in #107152.
Partly this is to fix the asymmetry of not having package sets available for the emacs*-nox packages.

The old names now only exists as aliases which probably explains why they have disappeared from Repology.

I think this opens up a question regarding if we should remove dontRecurseIntoAttrs from emacs*.pkgs.
We've tried this before and it makes tools like nix search a lot slower since these package sets are pretty massive.

Maybe we should only expose emacs.pkgs for the default Emacs package but not for the rest?
That way we also don't count every package in the sets N times (where N is the number of distinct Emacs packages).

[davidak]

I think such tools should show packages only once, even when they exist for different interpreters like emacs26 and emacs27. But it should be obvious that you can use both.

The raw package list should still contain all i guess, so tools like repology get all nixpkgs has to offer.

But that is a big topic and it's about UX.

For example in the search. Now it looks like this:

But this would be less cluttered and more user friendly:

(I opened an issue for that NixOS/nixos-search#292)

Same in nix search.

[Atemu]

The bug is probably here:

nixpkgs/pkgs/top-level/packages-config.nix

Line 37 in b310f49

"emacs26.pkgs"

@adisbladis did update the set names here too but I don't think sub-attributes work in that construct.

[adisbladis]

@Atemu Nice catch! I've attempted to conjure up a fix in #113680 but it needs more testing before I'm comfortable merging.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/wheres-the-nodejs-apps/12064/10

[davidak]

Status: rPackages is fixed

nixpkgs/pkgs/top-level/packages-config.nix

Line 28 in c54c8f8

"rPackages"

See for example "tidytidbits" on search.nixos.org or repology.

In the package set list is also emacs27, but i don't find any packages.

nixpkgs/pkgs/top-level/packages-config.nix

Line 37 in c54c8f8

"emacs27.pkgs"

When i search for random packages that existed before, like "pony-mode" i don't find it now in search or repology.

https://gist.github.com/davidak/8bee60e5485259b42a21d9b7916093da/92925357cd2d5b530d19e40e909519147861c1b9#file-gistfile1-txt-L8784
https://search.nixos.org/packages?channel=20.09&from=0&size=50&sort=relevance&query=pony-mode

The name does exist in ./pkgs/applications/editors/emacs/elisp-packages/recipes-archive-melpa.json which is not found in github search, probably because the file is too big.

On repology you find only 1 emacs package in nixpkgs!

https://repology.org/projects/?search=emacs%3A&maintainer=&category=&inrepo=nix_unstable&notinrepo=&repos=&families=&repos_newest=&families_newest=
https://repology.org/projects/?search=emacs-&maintainer=&category=&inrepo=nix_unstable&notinrepo=&repos=&families=&repos_newest=&families_newest=

https://hydra.nixos.org/build/149921732

[davidak@gaming:~/Downloads]$ nix run nixpkgs.brotli -c brotli --decompress packages.json.br
[davidak@gaming:~/Downloads]$ jq '.packages | keys' packages.json | grep magit
  "vimPlugins.vimagit",
  "vscode-extensions.kahole.magit",

[davidak@gaming:~/Downloads]$ jq '.packages | keys' packages.json | grep pony-mode

[davidak@gaming:~/Downloads]$ jq '.packages | keys' packages.json | grep emacs
  "emacs",
  "emacs-all-the-icons-fonts",
  "emacs-nox",
  "emacsMacport",
  "haskellPackages.emacs-keys",
  "haskellPackages.emacs-module",
  "haskellPackages.yi-emacs-colours",
  "haskellPackages.yi-keymap-emacs",
  "pinentry-emacs",
  "qemacs",

So the package set emacs27 is still missing from packages.json.