Cross bootstrap tools have broken stackprotector support

[r-burns]

Describe the bug
The cross-compiled bootstrap tools produced by make-bootstrap-tools-cross.nix have faulty support for stack protector hardening (libssp) during the resulting stdenv bootstrap chain. The issue manifests as linker errors when building the stage4 gcc from the static stage3 libraries during stdenv bootstrap. This means that when initially bootstrapping a system from cross-compiled bootstrap tools, an ugly workaround is needed:

--- a/pkgs/stdenv/linux/default.nix
+++ b/pkgs/stdenv/linux/default.nix
@@ -259,7 +259,14 @@ in
   (prevStage: stageFun prevStage {
     name = "bootstrap-stage3";
 
-    overrides = self: super: rec {
+    overrides = self: super: let
+      wrapStage3 = pkg: (pkg.override {
+        stdenv = self.makeStaticLibraries self.stdenv;
+      }).overrideAttrs (oa: {
+        # Work around faulty stackprotector support in cross bootstrap tools
+        hardeningDisable = (oa.hardeningDisable or []) ++ [ "stackprotector" ];
+      });
+    in rec {
       inherit (prevStage)
         ccWrapperStdenv
         binutils coreutils gnugrep
@@ -268,10 +275,10 @@ in
       # Link GCC statically against GMP etc.  This makes sense because
       # these builds of the libraries are only used by GCC, so it
       # reduces the size of the stdenv closure.
-      gmp = super.gmp.override { stdenv = self.makeStaticLibraries self.stdenv; };
-      mpfr = super.mpfr.override { stdenv = self.makeStaticLibraries self.stdenv; };
-      libmpc = super.libmpc.override { stdenv = self.makeStaticLibraries self.stdenv; };
-      isl_0_20 = super.isl_0_20.override { stdenv = self.makeStaticLibraries self.stdenv; };
+      gmp = wrapStage3 super.gmp;
+      mpfr = wrapStage3 super.mpfr;
+      libmpc = wrapStage3 super.libmpc;
+      isl_0_20 = wrapStage3 super.isl_0_20;
       gcc-unwrapped = super.gcc-unwrapped.override {
         isl = isl_0_20;
       };

To Reproduce
(The following steps are slightly modified from my original use case, to ease repro on commodity x86_64 hardware. You don't need any --option system fudging if you have a foreign arch system you can cross-compile to.)
Steps to reproduce the behavior:

1. Checkout any recent commit. I'm on f288a8c.
2. Patch make-bootstrap-tools-cross.nix for compilation to x86_64.

--- a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix
+++ b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix
@@ -13,6 +13,7 @@ in lib.mapAttrs (n: make) (with lib.systems.examples; {
   armv6l     = raspberryPi;
   armv7l     = armv7l-hf-multiplatform;
   aarch64    = aarch64-multiplatform;
+  x86_64 = gnu64;
   x86_64-musl  = musl64;
   armv6l-musl  = muslpi;
   aarch64-musl = aarch64-multiplatform-musl;

3. Build cross-compiled bootstrap tools. (We'll force the build system to be an i686 here, to take the cross-compilation codepaths on ordinary x86 hardware.)
   nix build -f pkgs/stdenv/linux/make-bootstrap-tools-cross.nix --option system i686-linux x86_64.build
4. Copy result/on-server/bootstrap-tools.tar.xz to pkgs/stdenv/linux/bootstrap-files/ and set it as the x86_64 bootstrap.

--- a/pkgs/stdenv/linux/bootstrap-files/x86_64.nix
+++ b/pkgs/stdenv/linux/bootstrap-files/x86_64.nix
@@ -2,8 +2,11 @@
 (import ./i686.nix) //
 
 {
+  /*
   bootstrapTools = import <nix/fetchurl.nix> {
     url = "http://tarballs.nixos.org/stdenv-linux/x86_64/c5aabb0d603e2c1ea05f5a93b3be82437f5ebf31/bootstrap-tools.tar.xz";
     sha256 = "a5ce9c155ed09397614646c9717fc7cd94b1023d7b76b618d409e4fefd6e9d39";
   };
+  */
+  bootstrapTools = ./bootstrap-tools.tar.xz;
 }

5. Build stdenv: nix build -f . stdenv
6. Observe undefined reference to ``__stack_chk_guard' linking errors near the end of the stage 4 gcc build. The stage3 static library dependencies of gcc (gmp, mpfr, mpc, and isl) must be overridden to disable stackprotector hardening for gcc to build.

Log snippet:

/build/build/./prev-gcc/xg++ -B/build/build/./prev-gcc/ -B/nix/store/0v9kxf2w2rqvd5410vnc5lrqciiryaxd-gcc-10.2.0/x86_64-unknown-linux-gnu/bin/ -nostdinc+>
/build/build/./prev-gcc/xg++ -B/build/build/./prev-gcc/ -B/nix/store/0v9kxf2w2rqvd5410vnc5lrqciiryaxd-gcc-10.2.0/x86_64-unknown-linux-gnu/bin/ -nostdinc+>
/build/build/./prev-gcc/xg++ -B/build/build/./prev-gcc/ -B/nix/store/0v9kxf2w2rqvd5410vnc5lrqciiryaxd-gcc-10.2.0/x86_64-unknown-linux-gnu/bin/ -nostdinc+>
  cc1-checksum.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a   ..>
/build/build/./prev-gcc/xg++ -B/build/build/./prev-gcc/ -B/nix/store/0v9kxf2w2rqvd5410vnc5lrqciiryaxd-gcc-10.2.0/x86_64-unknown-linux-gnu/bin/ -nostdinc+>
      cp/cp-lang.o c-family/stub-objc.o cp/call.o cp/class.o cp/constexpr.o cp/constraint.o cp/coroutines.o cp/cp-gimplify.o cp/cp-objcp-common.o cp/cp-u>
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: /nix/store/4sssnshkvw4i722j0qrkdhyjcmx71l62-isl-0.20/lib/libisl.a(isl_map.o): in func>
(.text+0x82b): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0x8b0): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0x927): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0xa29): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0xaa3): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: /nix/store/4sssnshkvw4i722j0qrkdhyjcmx71l62-isl-0.20/lib/libisl.a(isl_map.o):(.text+0>
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: /nix/store/4sssnshkvw4i722j0qrkdhyjcmx71l62-isl-0.20/lib/libisl.a(isl_map.o): in func>
(.text+0x82b): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0x8b0): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0x927): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0xa29): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0xaa3): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: /nix/store/4sssnshkvw4i722j0qrkdhyjcmx71l62-isl-0.20/lib/libisl.a(isl_map.o):(.text+0>
collect2: error: ld returned 1 exit status
make[3]: *** [../../gcc-10.2.0/gcc/lto/Make-lang.in:88: lto1] Error 1
make[3]: *** Waiting for unfinished jobs....
collect2: error: ld returned 1 exit status
make[3]: *** [../../gcc-10.2.0/gcc/lto/Make-lang.in:92: lto-dump] Error 1
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: /nix/store/4sssnshkvw4i722j0qrkdhyjcmx71l62-isl-0.20/lib/libisl.a(isl_map.o): in func>
(.text+0x82b): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0x8b0): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0x927): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0xa29): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0xaa3): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: /nix/store/4sssnshkvw4i722j0qrkdhyjcmx71l62-isl-0.20/lib/libisl.a(isl_map.o):(.text+0>
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: /nix/store/4sssnshkvw4i722j0qrkdhyjcmx71l62-isl-0.20/lib/libisl.a(isl_map.o): in func>
(.text+0x82b): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0x8b0): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0x927): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0xa29): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: (.text+0xaa3): undefined reference to `__stack_chk_guard'
/nix/store/467843qgpic7qqhsq1q1ciikr775h9lz-binutils-2.35.1/bin/ld: /nix/store/4sssnshkvw4i722j0qrkdhyjcmx71l62-isl-0.20/lib/libisl.a(isl_map.o):(.text+0>
collect2: error: ld returned 1 exit status
make[3]: *** [../../gcc-10.2.0/gcc/c/Make-lang.in:85: cc1] Error 1
collect2: error: ld returned 1 exit status
make[3]: *** [../../gcc-10.2.0/gcc/cp/Make-lang.in:120: cc1plus] Error 1
rm gcc.pod
make[3]: Leaving directory '/build/build/gcc'
make[2]: *** [Makefile:4906: all-stageprofile-gcc] Error 2

Expected behavior
Successful build of stdenv using cross-compiled bootstrap files

Additional context
I first encountered this months ago when using cross-built bootstrap tools for powerpc64le, and assumed it was an arch-specific issue. However, when later using native-built bootstrap tools I realized the stackprotector hardening could be reenabled in the stage3 libraries. It wasn't until I tested i686 -> x86_64 bootstrap that I realized this was a more general issue.

Notify maintainers
Cross infra CODEOWNERS: @Ericson2314 @matthewbauer

[DieracDelta]

Maybe this isn't the morally correct fix, but I'm also hitting this problem... Thoughts on merging the provided fix into Nixpkgs?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/getting-started-with-nix-on-ppc64le/12712/2

[r-burns]

Hmm, I think I would want input from someone more versed in stdenv. AFAICT the stage3 static libs are only used for the stage4 gcc, but that gcc is the one used in stdenv. So these libraries would never be exposed directly in nixpkgs, but would be used transitively by gcc when every package is built. So I'd need to know what issue the stackprotector hardening addresses, and if that's potentially an issue for us to disable it for dependencies of stdenv's gcc. It looks like in most cases, when stackprotector causes issues, we just turn it off. But that only seems to be the case for leaf packages; I would imagine stdenv requires much more scrutiny.

[DieracDelta]

For some reason, this is actually making it into nixpkgs for me. E.g. the cross compiler I grab here gives a gcc cross compiler from x8664 to riscv64. When I try to compile a kernel with that cross compiler toolchain (riscv64-unknown-linux-gnu-*), I get stack protector errors at link time (exactly as you've described). I'm not sure which stage gcc this is; perhaps this is completely different. I just pattern matched on the error you've described.

[piperswe]

So I'd need to know what issue the stackprotector hardening addresses, and if that's potentially an issue for us to disable it for dependencies of stdenv's gcc.

The stack protector guards against buffer overflow in the built artifact. Hardening of a compiler's stack protector should have no impact on that compiler's output, and since this compiler is only used to compile stage4 I'd say this change is fine.

[r-burns]

since this compiler is only used to compile stage4 I'd say this change is fine.

This compiler /is/ stage4, and is exposed as stdenv.cc.

Realistically, I think we have bigger fish to fry w.r.t. the security of nixpkgs. But I'm not a security expert so I'd want some more reassurance here.

[piperswe]

This compiler /is/ stage4, and is exposed as stdenv.cc.

Oh, it might be worth it to rebuild the compiler in stage4 then.

[r-burns]

Hmm, I just found #40797 in the issue backlog and am wondering if it's related.

[stale]

I marked this as stale due to inactivity. → More info

[trofi]

Proposed the possible fix as #181943

[r-burns]

Confirmed this has been resolved by trofi's sysroot fix. Thanks!