nativeBuildInputs gives non native dependencies when used with overrides in all-packages.nix

[symphorien]

Issue description

nativeBuildInputs puts binaries of the wrong architecture on the PATH. That is, binaries cannot be executed on the build architecture.

Note, I am somewhat afraid I could just have misunderstood everything about cross compilation in nixpkgs.

Steps to reproduce

consider this file:

with import <nixpkgs> {
      crossSystem = { config = "aarch64-unknown-linux-gnu"; };
};
stdenv.mkDerivation {
      name = "foo";
      nativeBuildInputs = [ pkgconfig ];
      strictDeps = true;
      buildCommand = ''
            echo PATH=$PATH
            echo nbi=$nativeBuildInputs
            pkg-config --version
            touch $out
      '';
}

When building it I get

these derivations will be built:                                                                                                                                                                       
  /nix/store/layqrjhacamrnpf4fy8k7ww05kzyy8j5-foo-aarch64-unknown-linux-gnu.drv                                                                                                                        
building '/nix/store/layqrjhacamrnpf4fy8k7ww05kzyy8j5-foo-aarch64-unknown-linux-gnu.drv'...                                                                                                            
PATH=/nix/store/mwsk6nin4rxsbghfhqxzk9lcsk09nbbl-pkg-config-0.29.2-aarch64-unknown-linux-gnu/bin:/nix/store/bqp3d4yhpx6h60fv8jww5id0nkm44034-patchelf-0.9/bin:/nix/store/63j8mi5vjyxfc89rhjknwxaxqg3a2s
x2-paxctl-0.9/bin:/nix/store/xswqhzlmxyc1s21kqdiyb7hl62560gpm-aarch64-unknown-linux-gnu-stage-final-gcc-debug-wrapper-7.3.0/bin:/nix/store/dsn74vd7k0mqqmkc7plgdhbav2v9v515-aarch64-unknown-linux-gnu-s
tage-final-gcc-debug-7.3.0/bin:/nix/store/cpxkl6k01lh3wkaw2fv8xz8v60j43n4s-glibc-2.27-aarch64-unknown-linux-gnu-bin/bin:/nix/store/wm8va53fh5158ipi0ic9gir64hrvqv1z-coreutils-8.29/bin:/nix/store/gp0aj
2lxhrlkqf0g4f38dg5hqg45fnad-aarch64-unknown-linux-gnu-binutils-wrapper-2.30/bin:/nix/store/nncs4b6jsbfzx9c8i3p27qrj3z74fh9d-aarch64-unknown-linux-gnu-binutils-2.30/bin:/nix/store/cpxkl6k01lh3wkaw2fv8
xz8v60j43n4s-glibc-2.27-aarch64-unknown-linux-gnu-bin/bin:/nix/store/wm8va53fh5158ipi0ic9gir64hrvqv1z-coreutils-8.29/bin:/nix/store/wm8va53fh5158ipi0ic9gir64hrvqv1z-coreutils-8.29/bin:/nix/store/g5dl
pwd44kd75i71nwzii8w4bp4inxwk-findutils-4.6.0/bin:/nix/store/2wdj4rznh37jd44lxz9nn6ymnbqs9axp-diffutils-3.6/bin:/nix/store/ny5p32137wfyzdm485xfdck21w1gyl3g-gnused-4.5/bin:/nix/store/9f89z51na7w931aja8
lqlmhqny9h16cj-gnugrep-3.1/bin:/nix/store/zdadpaj5z8k9ifkgsc16yvfjn4wdv0r2-gawk-4.2.1/bin:/nix/store/lxbaya2mzjc3xzbcsykrcbvcf5vxvfia-gnutar-1.30/bin:/nix/store/ix13jm9a1jfkcg1fs3skagyz4s1w77r6-gzip-
1.9/bin:/nix/store/1iih7pgc7krhis13zaq8ajdcb2hd10d9-bzip2-1.0.6.0.1-bin/bin:/nix/store/ql3azyviwyc6w73h6prk1j2945skfapf-gnumake-4.2.1/bin:/nix/store/czx8vkrb9jdgjyz8qfksh10vrnqa723l-bash-4.4-p23/bin:
/nix/store/61m8xswm9xxwb3ddw1za2cxvcikgk656-patch-2.7.6/bin:/nix/store/26lgqf0ja6rx8dnz972a3f56vfxmmmv5-xz-5.2.4-bin/bin                                                                               
nbi=/nix/store/mwsk6nin4rxsbghfhqxzk9lcsk09nbbl-pkg-config-0.29.2-aarch64-unknown-linux-gnu                                                                                                            
/nix/store/jbkam81ja9pn2jz11qq433i619q56ihm-stdenv-linux/setup: line 1261: /nix/store/mwsk6nin4rxsbghfhqxzk9lcsk09nbbl-pkg-config-0.29.2-aarch64-unknown-linux-gnu/bin/pkg-config: cannot execute binar
y file: Exec format error                                                                                                                                                                              
builder for '/nix/store/layqrjhacamrnpf4fy8k7ww05kzyy8j5-foo-aarch64-unknown-linux-gnu.drv' failed with exit code 126                                                                                  
error: build of '/nix/store/layqrjhacamrnpf4fy8k7ww05kzyy8j5-foo-aarch64-unknown-linux-gnu.drv' failed  

But I expected pkg-config to come from the build architecture (here x86_64-linux), not the cross (native/host) one.

I get the same result if I put pkg-config in depsBuildBuild.

Technical details

Checked on current nixos-18.03, on 18.09 18.09.932.09195057114 (Jellyfish) and on unstable: 9f88282

[matthewbauer]

Yes - here you need to use buildPackages. The splicing magic only happens when you use callPackage.

[symphorien]

Indeed wrapping mkDerivation in callPackage solves the problem. But my real problem was when cross compiling talloc, which uses callPackage. It seems giving arguments to callPackage breaks the spliciing:

with import <nixpkgs> {
      crossSystem = { config = "aarch64-unknown-linux-gnu"; };
};
callPackage ({ stdenv, fetchurl, python, pkgconfig, readline, libxslt
, docbook_xsl, docbook_xml_dtd_42}:
stdenv.mkDerivation {
      name = "foo";
      nativeBuildInputs = [python];
      strictDeps = true;
      buildCommand = ''
            echo PATH=$PATH
            echo nbi=$nativeBuildInputs
            python --version
            touch $out
      '';
}) { python = python2; }

This also puts the wrong python2 in the PATH.
If I remove python = python2 it builds.
If this is "expected", please close the issue, but I think this behavior is unwanted.

[symphorien]

My point is:
if you move python to nativeBuildInputs here

nixpkgs/pkgs/development/libraries/talloc/default.nix

Line 14 in 06fb025

buildInputs = [

and then build pkgsCross.whatever.talloc, python will have the wrong architecture.
If you remove python = python2 here

nixpkgs/pkgs/top-level/all-packages.nix

Line 12201 in 06fb025

python = python2;

then splicing is done correctly and python can be run when building talloc.
So the minimal working example above is a symptom of a real problem.

[matthewbauer]

Yeah, I think this is a known problem (although no one has created an issue for it yet, so this will do). It's pretty difficult because all of the splicing happens in Nixpkgs and we have no reference to the "buildPackages" python2 when you do python = python2. I think this is often solved by doing python = buildPackages.python2 instead.

In terms of solutions, there's not really a good solution to me. We could make callPackage support strings so you could do something like:

  python = "python2";

Then have callPackage replace the string with buildPackages."python2" for you as needed. We definitely don't want it using buildPackages.python though.

This is more of a bandaid than anything. There is definitely a need for better solutions.

[bennofs]

What if we used with pkgs.__splicedPackages; in all-packages.nix... I suspect this might give a recursion error though? (not sure if nix is lazy enough for this) But in theory, it should be possible to write it in a way such that python2 refers to the spliced package?

[matthewbauer]

What if we used with pkgs.__splicedPackages; in all-packages.nix... I suspect this might give a recursion error though? (not sure if nix is lazy enough for this) But in theory, it should be possible to write it in a way such that python2 refers to the spliced package?

/cc @Ericson2314

[Ericson2314]

Wow, that's a nice idea! I had written off splicing as unreadably shitty, but I guess there are things to do to make it better until the day we get rid of it entirely.

[Ericson2314]

Made #58327. The risk is doubly-spliced packages where we include an alias or something (instead of just includeing for .override) and then will end up with a doubly-spliced package with all the invariants broken.

[nspin]

For now, is it alright to use explicit references to __splicedPackages in all-packages.nix, taking care to avoid double splicing? For example:

talloc = callPackage ../development/libraries/talloc {
  python = __splicedPackages.python2;
};

[symphorien]

The reproducer of the open post still hits the bug.

[roberth]

Perhaps we could throw or warn when we detect that an input in nativeBuildInputs is incompatible.

[domenkozar]

The workaround doesn't work anymore on 21.05:

$ cat cross-compile-shell.nix 
{ nixpkgs ? fetchTarball "https://github.com/NixOS/nixpkgs/archive/0f8f64b54ed07966b83db2f20c888d5e035012ef.tar.gz"
, pkgs ? (import nixpkgs {}).pkgsCross.aarch64-multiplatform
}:

pkgs.callPackage ({ mkShell, zlib, pkg-config }: mkShell {
  nativeBuildInputs = [ pkg-config ];
  buildInputs = [ zlib ];
}) {}

$ nix-shell cross-compile-shell.nix --run "pkg-config"
/tmp/nix-shell-492070-0/rc: line 1: pkg-config: command not found

[sternenseemann]

@domenkozar This is working as intended. pkg-config is prefixed for the cross case, i. e. it is aarch64-unknown-linux-gnu-pkg-config in this case. So a good reminder to always use $PKG_CONFIG instead of pkg-config :)

[Artturin]

Perhaps we could throw or warn when we detect that an input in nativeBuildInputs is incompatible.

#204387

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cross-compiling-perl-packages-like-stow-riscv-aarch/39221/1