nix copy runs in non-constant memory when destination is file

[AleXoundOS]

The issue is a subset of #1681, perhaps which was solved for other kinds of destinations but file:///.

nix version: 2.2.
I try the following command:

$ curl -L https://nixos.org/channels/nixos-19.03/store-paths.xz | xz -d > store-paths
$ xargs -P1 -n1 -a store-paths nix copy --from https://cache.nixos.org --to file:///data/software/nixos/19.03/cache\?compression=none

It's quickly observable with tools like htop that the memory usage grows proportionally to the size of currently downloaded parts of files which are not downloaded (or written to disk) fully yet. Most likely it's the uncompressed size. I mean the number 1161.2 out of 1774.2 MiB here:

[1/26/35 copied (1161.2/1774.2 MiB), 109.2/153.2 MiB DL] copying path '/nix/store/wfgrz42bpcl1r635dasfk7r236hm83az-ghc-8.6.4' from 'https://cache.nixos.org' to 'file:///data/software/nixos/19.03/cache'

Originally I observed the issue running such nix copy on NanoPi Neo2 with 512 MiB RAM. It runs extremely slow by spending most of the time to swapping, giving little chance to finish in half a month.

[AleXoundOS]

I also experience nix copy stalls with symptoms similar to #1988. However, I'm not 100% certain that it's nix copy fault and not linux kernel's broken swapping (swap page allocation failure) on NanoPi Neo2.

[stale]

I marked this as stale due to inactivity. → More info

[AriFordsham]

Is there any plan to do anything about this?

[AriFordsham]

Here are my measurements for a test run, copying ghc, a reasonably large package, with 2.3.10 and master:

$ nix --version
nix (Nix) 2.3.10

$ \time nix copy --from https://cache.nixos.org --to file://$PWD/experiments nixpkgs.ghc
[27 copied (2141.5 MiB), 193.5 MiB DL]
1023.02user 14.08system 13:59.09elapsed 123%CPU (0avgtext+0avgdata 2502236maxresident)k
49640inputs+397496outputs (42major+561463minor)pagefaults 0swaps

$ nix-env -f https://github.com/NixOS/nixpkgs/archive/master.tar.gz -iA nix

$ nix --version
nix (Nix) 2.3.15

$ \time nix copy --from https://cache.nixos.org --to file://$PWD/experiments nixpkgs.ghc
[27 copied (2141.5 MiB), 193.5 MiB DL]
998.50user 12.19system 13:31.29elapsed 124%CPU (0avgtext+0avgdata 2437176maxresident)k
0inputs+396544outputs (0major+420782minor)pagefaults 0swaps

As you see, no improvement.

13+ minutes and ~2.5 GB of memory to download 193 MB and extract/copy 2.2 GB seems excessive.

I let the scripts from https://discourse.nixos.org/t/using-nixos-in-an-isolated-environment/3369/19 run for several days before giving up - it spent hours stuck on big packages. This is holding up my workflow.

Let me know if there's any other measurements that would be helpful.

[edolstra]

Try nixUnstable, it should use less memory:

$ command time nix copy --to file:///tmp/binary-cache nixpkgs#ghc
496.56user 2.07system 8:18.74elapsed 99%CPU (0avgtext+0avgdata 119460maxresident)k
4764202inputs+799527outputs (0major+159389minor)pagefaults 0swaps

Also, note that binary caches use XZ compression by default, which is slow. Try adding compression=none:

$ command time nix copy --to file:///tmp/binary-cache?compression=none nixpkgs#ghc
2.52user 3.36system 0:06.01elapsed 98%CPU (0avgtext+0avgdata 24232maxresident)k
8730132inputs+8731390outputs (0major+28383minor)pagefaults 0swaps

[AriFordsham]

Yes, the memory problem seems sorted with nixUnstable:

991.37user 10.61system 15:52.72elapsed 105%CPU (0avgtext+0avgdata 587844maxresident)k
394008inputs+1140096outputs (2major+167343minor)pagefaults 0swaps

I'm sure I will also find the compression=none tip useful.

Thank you @edolstra !

[stale]

I marked this as stale due to inactivity. → More info

[AleXoundOS]

I haven't checked the nixUnstable fix for the issue (as 16 GiB RAM eventually is the minimum for evaluating nixpkgs for isolated environment). But based on @AriFordsham report I will close the issue.