Permission denied error when building symlink derivation

[rvl]

Describe the bug

My apologies if this is already fixed on master branch, but I couldn't identify any particular issue or PR which describes this issue.

Nix 1.19.2 fails to build a derivation containing a symlink to store path.

# test-symlink.nix
{ local ? "/home/rodney/ops/nixpkgs"
, pkgs ? import local {}
}: rec {
  direct-symlink = pkgs.runCommand "direct-symlink" {} ''
    ln -vs ${local}/.version $out
  '';
  indirect-symlink = pkgs.runCommand "indirect-symlink" {} ''
    ln -vs ${direct-symlink} $out
  '';
}

Steps To Reproduce

The error message is:

rodney@tethys:~/ops/nixpkgs % nix build --experimental-features "nix-command" -f test-symlink.nix && readlink -f result*
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
error:
       … while setting up the build environment

       error: getting attributes of path '/nix/store/ws9yl6ph10v79gx4p9ilhyxg214xf7i7-direct-symlink': Permission denied

Expected behavior

I expected both derivations to build and for the resulting store paths to be resolvable symlinks.

This is what happens with nix-2.18.1:

rodney@tethys:~/ops/nixpkgs % nix build --experimental-features "nix-command" -f test-symlink.nix && readlink -f result*
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
/home/rodney/ops/nixpkgs/.version
/home/rodney/ops/nixpkgs/.version

nix-env --version output

This is the nixUnstable package, corresponding to revision 2c7f3c0fb7c08a0814627611d9d7d45ab6d75335 of nixpkgs.

nix (Nix) 2.19.2

Additional context

Introduced by

• Bindmount files instead of hardlinking or copying to chroot #8965

You might ask, why build such a silly derivation? Well, it's how mkOutOfStoreSymlink works under home-manager.

See: nix-community/home-manager#4692

Priorities

Add 👍 to issues you find important.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-12-18-nix-team-meeting-minutes-113/37050/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/home-manager-mkoutofstoresymlink-and-nixunstable/37241/1

[infinisil]

I tried to reproduce this, but struggled. I wrote a NixOS test for this that shows no problems:

# nixos-test.nix
let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/d6863cbcbbb80e71cecfc03356db1cda38919523";
  pkgs = import nixpkgs {
    config = {};
    overlays = [];
    system = "x86_64-linux";
  };
in
pkgs.nixosTest {
  name = "test";
  skipTypeCheck = true;
  nodes.machine = {
    environment.etc.nixpkgs.source = nixpkgs;
    system.extraDependencies = [
      (pkgs.closureInfo {
        rootPaths = [ pkgs.stdenvNoCC.drvPath ];
      })
    ];
    nix.package = pkgs.nixUnstable;
    environment.etc."test-symlink.nix".text = ''
      # test-symlink.nix
      { local ? "/etc/nixpkgs"
      , pkgs ? import local {}
      }: rec {
        direct-symlink = pkgs.runCommand "direct-symlink" {} '''
          ln -vs ''${local}/.version $out
        ''';
        indirect-symlink = pkgs.runCommand "indirect-symlink" {} '''
          ln -vs ''${direct-symlink} $out
        ''';
      }
    '';
  };

  testScript = ''
    start_all()
    machine.succeed('nix-build /etc/test-symlink.nix')
    print(machine.succeed('nix-env --version'))
    print(machine.succeed('readlink -f result*'))
  '';
}

Run with

$ nix-build nixos-test.nix
[...]
machine # these 2 derivations will be built:
machine #   /nix/store/lqh27yh50v8zyv57cwk4l5zz2yg8046q-direct-symlink.drv
machine #   /nix/store/4p887hcx1bks8h65p0j3ghfnximkddyf-indirect-symlink.drv
machine # warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
machine # warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
machine # building '/nix/store/lqh27yh50v8zyv57cwk4l5zz2yg8046q-direct-symlink.drv'...
machine # '/nix/store/96664m3zyjnw2xyhpkyipxcmf5s7vv3w-direct-symlink' -> '/etc/nixpkgs/.version'
machine # building '/nix/store/4p887hcx1bks8h65p0j3ghfnximkddyf-indirect-symlink.drv'...
machine # '/nix/store/hq63b5zkir3ibsas5w9rkyp8gllgs6jx-indirect-symlink' -> '/nix/store
/96664m3zyjnw2xyhpkyipxcmf5s7vv3w-direct-symlink'
(finished: must succeed: nix-build /etc/test-symlink.nix, in 11.68 seconds)
machine: must succeed: nix-env --version
(finished: must succeed: nix-env --version, in 0.06 seconds)
nix-env (Nix) 2.19.2

machine: must succeed: readlink -f result*
(finished: must succeed: readlink -f result*, in 0.02 seconds)
/nix/store/61gw1axgxrvx0bnxv2dla6zhkgwv45zr-source/.version
/nix/store/61gw1axgxrvx0bnxv2dla6zhkgwv45zr-source/.version

(finished: run the VM test script, in 11.93 seconds)
test script finished in 11.96s
cleanup
kill machine (pid 7)
machine # qemu-kvm: terminating on signal 15 from pid 4 (/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/bin/python3.11)
(finished: cleanup, in 0.05 seconds)
kill vlan (pid 5)
/nix/store/7dwclnxw0xdvmqvrmi19qyyxqs1xwv03-vm-test-run-test

I don't have more time right now, but if somebody could adjust this NixOS test to show the problem, that would be great!

[rvl]

I don't have more time right now, but if somebody could adjust this NixOS test to show the problem, that would be great!

Here @infinisil, try this one:

# nixos-test.nix
let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/d6863cbcbbb80e71cecfc03356db1cda38919523";
  pkgs = import nixpkgs {
    config = {};
    overlays = [];
    system = "x86_64-linux";
  };

  user = "alice";
  target = "/home/${user}/file";
in
pkgs.nixosTest {
  name = "test";
  skipTypeCheck = true;
  nodes.machine = {
    system.extraDependencies = [
      (pkgs.closureInfo {
        rootPaths = [ pkgs.stdenvNoCC.drvPath ];
      })
    ];
    nix.package = pkgs.nixUnstable;
    nix.nixPath = ["nixpkgs=${nixpkgs}"];
    environment.etc."test-symlink.nix".text = ''
      { pkgs ? import <nixpkgs> {} }: let
        direct-symlink = pkgs.runCommand "direct-symlink" {} '''
          ln -vs ${target} $out
        ''';
        indirect-symlink = pkgs.runCommand "indirect-symlink" {} '''
          ln -vs ''${direct-symlink} $out
        ''';
      in
        indirect-symlink
    '';
    users.users.${user}.isNormalUser = true;
    systemd.tmpfiles.rules = ["f ${target} 0644 ${user} ${user} - test"];
  };

  testScript = ''
    start_all()
    print(machine.succeed('nix --version'))
    machine.succeed("su -l ${user} -c 'nix-build /etc/test-symlink.nix'")
    machine.succeed("su -l ${user} -c 'readlink -f result'")
    machine.succeed("su -l ${user} -c 'test `readlink -f result` = ${target}'")
  '';
}

I believe that the three necessary conditions for failure are:

1. Symlink target is outside of /nix/store
2. nix-build is invoked from a normal user, not root.
3. nix.package = pkgs.nixUnstable;

[infinisil]

Thanks, using bisection I was able to confirm that #8965 (as already suspected by @cole-h) indeed is the culprit for this regression.

[Ericson2314]

When it does work like, what do the references look like? Is it possible the hardlinking was incorrectly following the symlink so we referencing paths not in our closure?!

[Artturin]

Rudimentary workaround/fix #9723

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-01-08-nix-team-meeting-minutes-114/38156/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/an-additional-nix-installed-by-home-manager/42298/1

[sdht0]

I was just wondering: is this bug not considered a breaking change critical enough to be prioritized? Stable nix releases > 2.18 breaks home-manager users as per this bug.

Does the behavior being relied upon by home-manager not come under stability guarantees? Asking because the discourse report marked this as Annoying but not critical.

[rvl]

For me personally, it's a critical issue because my home-manager config which used to work nicely is now broken.

However I think home-manager could work around the issue by building symlinks within a nix store path directory, rather than as top-level nix store paths. So from that perspective it's "annoying but not critical."

For stability guarantees I guess we would need to know which behaviours (if not all) should be stable, and what does correct behaviour look like. Perhaps this is the more important issue. What is the meaning of a symlink at the top level of a nix store?

[alois31]

Not only does this lead to "permission denied" errors when the symlink target is not accessible to the build user, it also introduces a huge impurity when it is.

[SrEstegosaurio]

I don't know but impermanence is pretty broken for me right now and by being in a root-on-tmpfs system this is quite critical.