Support extra compatible architectures

[lheckemann]

This is based on @cleverca22's patch (cleverca22/nix-misc). It allows building for additional architectures locally. This can be useful with binfmt-misc or compatible architectures (e.g. ARM32-capable aarch64 machines). Additionally, it enables 32-bit ARM seccomp support.

Questions:

• Thoughts on disabling i686-linux on x86_64-linux by default?
• What about the seccomp setting? I experienced crashes caused by seccomp while testing on an ARM32-capable aarch64 machine, but I'm not sure how this will behave on non-ARM32-capable ones — in case it fails, I didn't throw an error. I'm not sure if this is appropriate, feedback would be appreciated.

[shlevy]

I'm lukewarm on this. Partly because it seems like kind of a niche use case, and partly because I don't really trust qemu-user enough (though that's a gut feeling that's not well-founded). I'm open to be swayed if there are compelling use cases.

Either way needs docs, ideally with pointers to how you might use this with qemu-user

[shlevy]

IMO this should default ti i686-linux on x86_64-linux

[lheckemann]

Makes sense.

[Ericson2314]

Shouldn't we have all the entries from from https://github.com/NixOS/nix/pull/1916/files#diff-81a5a969aed0f3f9d6dc4e99b92925cdL60 ?

[lheckemann]

I'm not sure having all of them is a good idea, given how unreliable actually building them is: building armv6 stuff on an armv7 machine will often result in code that doesn't actually run on armv6 because of annoying autodetection in various build systems. Users who want to risk that can easily set this setting as appropriate.

[bjornfor]

partly because I don't really trust qemu-user enough (though that's a gut feeling that's not well-founded).

That is my impression too. Relevant: NixOS/nixpkgs#33553.

[lheckemann]

Oh yes, the docs. Whoops :)

As for its actual use — I'm not saying this absolutely should be used with qemu-user; I've had a bit of trouble with qemu-user too. My primary motivation was to allow ARM32-capable aarch64 machines to build ARM32 stuff. Another option for this would of course be to have both 32-bit and 64-bit nix installed on the machine, but at that point working out which to use, especially in automated invocations, could get hairy.

[shlevy]

Oh, yeah, that's a solid use case 👍

[cleverca22]

another edgecase this can fix, some of the kernel options in nixos lack 32bit x86 support, and there is currently no way to tell nix about that

but if the automatic 64bit supports 32bit too code is removed, we can declare the 32bit support in the config file

[lheckemann]

Added some documentation, renamed the option, added i686-linux on x86_64-linux by default. I'm not 100% happy with the description, advice on improving it would be appreciated. I'll also squash the commits before this gets merged.

[edolstra]

I don't really see the use case for this option. Nix needs to know how to build something for a particular platform type. For example, on x86_64-linux, we can support i686-linux because there is a personality() syscall to do so. Merely adding a platform to an extra-platforms option isn't enough.

[edolstra]

This warning suggests that extra-platforms is a really bad idea. I mean, it allows you to completely lie to Nix about the platform type (e.g. do a x86_64-darwin build on x86_64-linux), thus producing a completely wrong build result, which could then end up in a binary cache.

[cleverca22]

if you lie and dont configure things properly, it will simply fail to execute the darwin bash/gcc/bootstrap tools, and never get anywhere

[lheckemann]

I've tested this and it works on an aarch64 machine with 32-bit support. Nixpkgs does need quite a few modifications for this to work (NixOS/nixpkgs#21471 should fix this properly) correctly, but I have got it producing binaries that work fine on a 32-bit only ARM machine successfully.

[edolstra]

Why does Nixpkgs need modification?

[lheckemann]

localhost:/tmp# nix build -f '<nixpkgs>' --argstr system armv7l-linux hello
[1 built, 1 copied (0.7 MiB), 0.0 MiB DL]
localhost:/tmp# uname -a
Linux localhost 4.9.20 #1 SMP Tue Sep 12 19:34:09 UTC 2017 aarch64 GNU/Linux
localhost:/tmp# file result/
bin/    share/
localhost:/tmp# file result/bin/hello
result/bin/hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/lhsrzki5bx9bj2ccxibxw5scfckh3p01-glibc-2.27/lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped
localhost:/tmp# result/bin/hello
Hello, world!

The reason nixpkgs needs modification is that a lot of packages will incorrectly detect the system (using uname AFAIK) to build for as aarch64 when we actually want to build for armv7 using armv7 tools.

[edolstra]

Right, that demonstrates exactly why this option is a bad idea. You could do a armv7 build and end up with a aarch64 binary, a rather significant impurity. For x86_64-linux/i686-linux we have personality() to prevent that. Maybe aarch64 has something similar?

IMHO this PR should be reverted. Or at the very least extra-platforms should be undocumented.

[lheckemann]

Maybe aarch64 has something similar?

It doesn't, as far as I know.

You could do a armv7 build and end up with a aarch64 binary, a rather significant impurity.

Exactly the same thing can still be achieved by using an armv7 nix on the same machine, so the absence of this option doesn't eliminate this impurity risk.

[edolstra]

It doesn't, as far as I know.

Maybe seccomp can be used to fake the uname result?

Exactly the same thing can still be achieved by using an armv7 nix on the same machine, so the absence of this option doesn't eliminate this impurity risk.

Yes, there is the same problem when running i686-linux on x86_64-linux since then it doesn't know that it should use personality(). I guess we should move the build-time determination of the platform to runtime.

[lheckemann]

Maybe seccomp can be used to fake the uname result?

Sure, if that works!

Yes, there is the same problem when running i686-linux on x86_64-linux since then it doesn't know that it should use personality(). I guess we should move the build-time determination of the platform to runtime.

Yes, but I'm also strongly in favour of having nixpkgs tell the build systems what to build for explicitly as per NixOS/nixpkgs#21471.

What I don't want is to end up having machines (probably more powerful than any "pure" 32-bit ARM machine) that can run a 32-bit user space but not allowing nix to build 32-bit arm stuff on them, just because they also support aarch64.

[Ericson2314]

Maybe seccomp can be used to fake the uname result?

That + NixOS/nixpkgs#21471 seems like a good thing.

[dezgeg]

There is already some discussion on the uname thing at NixOS/nixpkgs#25240. So, there's no personality() flag that is good enough. And seccomp (except until this SECCOMP_RET_USER_NOTIF someone proposed on LKML goes in) would mean using ptrace(), which is pretty much a no-go. I also don't expect NixOS/nixpkgs#21471 to ever happen (if it means that configure scripts will start spewing those configure: error: cannot run test program while cross compiling. things at users building native stuff).

What to me sounds like the best next plan is to try exporting build_alias to the host triplet in native builds, which should make autotools not call uname without hopefully breaking anything else. Sadly, I think some of the known offenders (on the ARMv6-on-ARMv7 case at least) like perl and openssl don't use autotools...

[dezgeg]

BTW so far I've found out it's easiest to boot a ARMv7 KVM virtual machine on aarch64 to sidestep this problem :-)

[Ericson2314]

if it means...

I don't expect to disable all of those and live either, but I'd like to do the rest of that issue.

[dezgeg]

(Getting a bit off-topic, but...) Well, the condition for that spew is literally:

AS_IF([test "$cross_compiling" = yes],
  [m4_default([$4],
	   [AC_MSG_FAILURE([cannot run test program while cross compiling])])],
  [_AC_RUN_IFELSE($@)])
])

so there is no way to simultaneously have autotools in cross-compilation mode and have those compile+run tests getting executed.

[dezgeg]

I tried the quick hack of doing a export build_alias='armv6l-unknown-linux-gnueabihf' in stdenv and the results were quite good; Ruby created a lib/ruby/vendor_ruby/2.4.0/armv6l-linux-eabihf as expected and GMP stopped looking in /proc/cpuinfo and enabling NEON instructions (IIRC GMP currently has an impurity problem on x86 as well where it does similar things). Perl and OpenSSL didn't get fixed as expected (since they don't use autotools).

[lopsided98]

I built an armv6l system on armv7l, and it seems to be working well. So far, I've only had to modify openssl and I haven't run into any other illegal instructions yet.

@dezgeg Where did you set build_alias? In mkDerivation, or somewhere else? I was thinking about setting configurePlatforms to [ "build" ] by default, but this fails because it passes --build to all configure scripts, which often fails if the package does not use autotools. Should configurePlatforms set build/host/target_alias instead?

[dezgeg]

See NixOS/nixpkgs@master...dezgeg:uname-hack.

[dezgeg]

Seems like this SECCOMP_RET_USER_NOTIF is gaining some traction (https://lwn.net/SubscriberLink/756233/1f23df75d146c393/) so hopefully we get to use it for uname() emulation.

In the meantime I've experimented with a dirty kernel hack to look at the system environment variable and base the uname -m result on that: https://github.com/dezgeg/nixpkgs/blob/arm-work/pkgs/os-specific/linux/kernel/uname-hack.patch

Don't use yet, it still has some problems with the huge environment variables that happen in nixpkgs stdenv.

[lopsided98]

I've been building for armv6l on armv7l using build_alias and a fix for openssl to prevent it from detecting the platform. I'm sure there are packages that still have problems, but it has been working well for my use cases. My armv6l machines are so slow that this is the only way I can feasibly run NixOS on them, so I would like to see this configuration be better supported in nixpkgs.