slow nix build in devcontainer

[lentilus]

Describe the bug

Building nix flakes in a devpod devcontainer takes significantly longer (hours) compared to bare metal (seconds).

Steps To Reproduce
0. I am on a Fedora machine (not NixOS) but I think this is irrelevant for the issue

1. create a minimal .devcontainer.json with a nix feature

{
    "image":"mcr.microsoft.com/devcontainers/base:ubuntu",
    "features": {
        "ghcr.io/devcontainers/features/nix:1": {
            "version": "2.23.1",
            "extraNixConfig": "experimental-features = nix-command flakes, require-sigs = false"
        }
    }
}

2. spin up the devcontainer

# install devpod
nix-shell -p devpod

# add local docker as provider
devpod provider add docker

# launch devpod
devpod up .  --ide none

3. build something

ssh myproject.devod
nix build .#mything

Additional Context

I am trying to find a way to apply my home-manager configuration to a devcontainer. I cant wait hours for this to finish as it totally defeats the purpose of having devcontainers that you can quickly spin up whenever you need them.
Even though I am only concerned about my hm-flake, I think the issue is more general, as all nix-building seems very slow.
my home-manager config can be found here.

What i tried so far

My first thought was, that nix is just building way more stuff than it should not using cache.nixos properly, but I was able to verify that this is not the case. To narrow this down even further I did a nix build .#homeConfigurations.myuser.activationPackage on my host followed by

nix copy --no-check-sigs --to ssh-ng:devpodtesting.devpod ./result

# in devpod:
/nix/store/somehash-home-manager-generateion/activate

I really expected this to work smoothly but other than expected it is still taking about 15min just to install the home-manager-path.

I am fairly new to nix and I ran out of ideas, of how to noarrow down the issue further, so any pointers would very much be appreciated!
lentilus

[edolstra]

I couldn't reproduce this. Maybe you can add -vvvvv to see if Nix is hanging somewhere.

[lentilus]

Thanks for stepping in! I will post a -vvvvv log tonight. So far I can tell you that -vvv indicated that the "slowness" originates from the execution of the (bash) builder.

[lentilus]

Here it is: This is an excerpt of the output of nix build .#homeConfigurations.vscode.activationPackage -vvvvv (without previous copying of the store!)

download thread waiting for 100 ms
download buffer is full; going to sleep
download thread waiting for 100 ms
download buffer is full; going to sleep
download buffer is full; going to sleep
download thread waiting for 100 ms
download buffer is full; going to sleep
download thread waiting for 100 ms
download buffer is full; going to sleep
curl: Connection #0 to host cache.nixos.org left intact
finished download of 'https://cache.nixos.org/nar/1fvsc54n33wg7x9m8jwfpw066p15y55r6bnrhigy0cx3rn61ndim.nar.xz'; curl status = 0, HTTP status = 200, body = 79766112 bytes
download thread waiting for 10000 ms
lock released on '/nix/store/1rfa1p1yinw2zys5p224lf49xsd3za3d-llvm-17.0.6-lib.lock'
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms
download thread waiting for 10000 ms

---

Below is the output of just -vvv

  /nix/store/031cw5p43s7wmf7na4ypm5mnb7nqkk2r-rustc-1.77.2
  /nix/store/fvp921c5d0mic6vp36dnj7inl7i8pm87-rustc-wrapper-1.77.2
  /nix/store/d9ff8aqv537mlhpinncx6dwc7a5ky6gk-systemd-255.6
  /nix/store/qf85fzj9qx61y4nwa1vdis6jsdxg5210-tmux-3.4
  /nix/store/2dgkjj9z89ymaggbirp985viagxbhbm4-w3m-0.5.3+git20230121
  /nix/store/klfk7c979390gk26684pxlb6bkdd423n-xz-5.4.7-dev
  /nix/store/3fk9w1crc33rcnq3nil6cr44k1wafqrj-zoxide-0.9.4
  /nix/store/l74lmnfb989k6zwq5f6fplr9ikp1zk36-zstd-1.5.6-dev
copying path '/nix/store/9s6viar1n7285msyvgaabm8cymn9d11l-man-db-2.12.1' from 'https://cache.nixos.org'...
copying path '/nix/store/3fk9w1crc33rcnq3nil6cr44k1wafqrj-zoxide-0.9.4' from 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/nar/0bsfb2dxpivi1dv3jassapvbr2alm59223l4dr1cccscv2bpwmkj.nar.xz'...
downloading 'https://cache.nixos.org/nar/0zbbrvkpgig4qcxa9wj112ypymrd6706iwghppgg4sl0j5a3s57m.nar.xz'...
executing builder '/nix/store/5jw69mbaj5dg4l2bj58acg3gxywfszpj-bash-5.2p26/bin/bash'
using builder args '-e /nix/store/4vq3350zc6sqnibkqpgic2d6cvi1r9hq-substitute-all.sh'
[0/29 built, 2/0/127 copied (3.0/903.0 MiB), 0.8/187.3 MiB DL]

Note! The two logs are not from the exact same phase of the build, but both logs are from one of those periodically occuring slow parts of the build.

What I find a little confusting is that the second log suggest we are executing the builder while things arent moving. The more verbose log though suggests, that we are still downloading stuff during that phase (which appearantly is not going smoothly). Htop suggests that the nix-daemon is using maximum resources in those phases.

Additional Note: As far as I can tell, there are no network issues with the pod.

[lentilus]

The issue was fixed by #11171 . For details see the thread in #10630 .

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-08-12-nix-team-meeting-minutes-168/50561/1