nixos/binfmt: Add QEMU wrapper to preserve argv[0]

[zhaofengli]

Motivation for this change

This PR enables argv[0] preservation for the QEMU binfmt-misc rules generated by boot.binfmt.emulatedSystems. It allows certain packages that rely on argv[0] in their build process (e.g., coreutils for its tests) to correctly build under binfmt.

Traditionally, the emulator invoked by binfmt-misc doesn't know what the original argv[0] is when running an executable, because only the absolute path to the foreign executable is passed. Currently, the emulator will simply use the full path as argv[0]. This leads to problems with multi-call executables as well as diff-based test methodologies used in various packages. For example, coreutils relies on output comparison in many test cases:

uniq: test 145.p: stderr mismatch, comparing 145.p.3 (expected) and 145.p.E (actual)
*** 145.p.3     Mon Oct 18 07:11:43 2021
--- 145.p.E     Mon Oct 18 07:11:44 2021
***************
*** 1,7 ****
! uniq: invalid argument 'badoption' for '--group'
  Valid arguments are:
    - 'prepend'
    - 'append'
    - 'separate'
    - 'both'
! Try 'uniq --help' for more information.
--- 1,7 ----
! /build/coreutils-8.32/src/uniq: invalid argument 'badoption' for '--group'
  Valid arguments are:
    - 'prepend'
    - 'append'
    - 'separate'
    - 'both'
! Try '/build/coreutils-8.32/src/uniq --help' for more information.

binfmt-misc now has the "P" flag that, when enabled, will add an extra argument after the absolute path to the foreign binary containing the original argv[0]. However, this also means that CLI compatibility with existing emulator executables is broken. SUSE has been carrying a patch since 2011 to add -binfmt wrappers that translate the new arguments into ones that regular QEMU accepts (-0 for overriding argv[0]). There have been attempts to resolve this issue upstream, but so far there seems to be no consensus.

This PR adds a statically-linked wrapper for QEMU that translates the arguments. It's possible to rewrite the arguments in the wrapper script that we use (#118926), but that still leaves some remaining issues (e.g., extraneous output when setting LD_PRELOAD).

Tested by building coreutils for riscv64-linux on x86_64-linux under binfmt (before, after).

Things done

• Tested on platform(s)
  • riscv64-linux on x86_64-linux
  • aarch64-linux on x86_64-linux
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• 21.11 Release Notes (or backporting 21.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[oxalica]

Dup of #118926

[zhaofengli]

Whoops, wasn't aware of that PR and GitHub wasn't returning good results for my searches. However, I still think the implementation here is more robust and only affects QEMU (not wine, for example).

[zhaofengli]

I implemented the suggestion from @oxalica in #115406 (comment) and coreutils can now be built under binfmt. I did a --check build and the results match what I got from native riscv64. The old version is here for comparison.

[oxalica]

Also we should update the PR title.

[zhaofengli]

Looks like extra-sandbox-path cannot contain the path to the executable itself, otherwise there will be some interesting conflicts when you try to build nix.conf.

To reproduce:

• Activate emulatedSystems using the previous commit
• After activation, run nix-build --check $(nix-store -qd /etc/nix/nix.conf) on the same machine

[oxalica]

Just need more docs. It works pretty fine now.
I've started a build of riscv64 coreutils with binfmt from this PR. Hope it could pass tests.

[oxalica]

riscv64-linux coreutils built and passed all tests with QEMU wrapper from this PR now.

[risicle]

True to your word this does appear to fix the LD_PRELOAD error message with binfmt - would you mind adding a test such as the following for this?

  ldPreload = makeTest {
    name = "systemd-binfmt-ld-preload";
    machine = {
      boot.binfmt.emulatedSystems = [
        "aarch64-linux"
      ];
    };
    testScript = let
      helloAarch64 = pkgs.pkgsCross.aarch64-multiplatform.hello;
      libredirectAarch64 = pkgs.pkgsCross.aarch64-multiplatform.libredirect;
    in ''
      machine.start()

      assert "error" not in machine.succeed(
          "LD_PRELOAD='${libredirectAarch64}/lib/libredirect.so' ${helloAarch64}/bin/hello 2>&1"
      ).lower()
    '';
  };

[risicle]

@ofborg test systemd-binfmt

(I want to see what happens when this runs on an aarch64 machine)

[risicle]

Great work.

[0x4A6F]

Tests successfully run against rebased master.

LGTM