Can't use local substituter when mounted read-only

[thomasjm]

I'm trying to bind-mount my /nix dir into a Docker container and use it as a local substituter. This works fine when I mount read/write:

docker run -it -v /nix:/host_nix/nix nixos/nix

> nix-build -E 'with import <nixpkgs> {}; pkgs.hello' --extra-substituters /host_nix/
...works fine, uses the local substituter to avoid hitting cache.nixos.org

But, I don't want the container to be able to modify my /nix dir if it happens to be run as root. So I tried to do the same thing with a read-only mount (note the :ro).

docker run -it -v /nix:/host_nix/nix:ro nixos/nix

> nix-build -E 'with import <nixpkgs> {}; pkgs.hello' --extra-substituters /host_nix/
warning: remounting /host_nix/nix/store writable: Operation not permitted
...proceeds to hit cache.nixos.org

As you can see, just because we're running as root, nix seems under the mistaken impression that it can remount this directory. When that fails, it fails to use the substituter.

It looks like maybe this is due to the "substituter" code path reusing the code for dealing with a normal local store?

Expected behavior

Read-only bind mounts should work as local substituters.

nix-env --version output

nix-env (Nix) 2.8.0

[thomasjm]

Note: the code path in question seems to get triggered here.

I tried to dodge it by running the container as non-root, but then I promptly ran into another error a few lines lower.

[arianvp]

Another workaround is to use the new overlay store functionality. You would use /host_nix as the lowerdir and then things should just work TM . Except for that the lower store should really be "immutable" which is not the case in this case.