Always cross compile

[Ericson2314]

Exherbo, another linux distribution that has done good work with cross-compiling, always builds cross compilers---i.e. --host --build and --target are always passed to gcc's build system even if some of those platforms are the same: http://git.exherbo.org/arbor.git/tree/packages/sys-devel/gcc/gcc.exlib#n187. The big advantage here is by using the cross-compiling code path in all cases, there's less to maintain, and actual cross compilation is less likely to rot assuming native compilation will always be better tested.

In #21268 (specifically 633feb4 but I'll probably rebase at some point breaking that link), I introduce always-defined buildPlatform hostPlatform and targetPlatform for the same reasons (I keep around nullable crossSystem and not-always present stdenv.cross for compatibility). So if/when that PR is merged would be a good time to tackle this. that PR will probably be closed as a bunch else happened separately, including always defining buildPlatform hostPlatform and targetPlatform and removing crossSystem and stdenv.cross altogether.

Naturally this would be a gcc-caused mass rebuild on Linux. Darwin shouldn't be as affected as LLVM, by default, includes all the targets we care about.

[Ericson2314]

@DavidEGrayson's work in https://github.com/DavidEGrayson/nixcrpkgs/ should greatly accelerate this!

[Ericson2314]

@cleverca22 points out https://github.com/taktoa/arcane-chat/blob/master/default.nix has a bunch of hacks needed to get windows to work. Would be good to keep these things in mind.

[Ericson2314]

OK, hope this actually happens this cycle 🤞.

[wmertens]

This will be seriously amazing!

So after a quick look, this change requires changing all occurrences of crossSystem to hostPlatform or targetPlatform? Is there no way to guess the correct value at evaluation time?

In other words, it's a huge single-shot change and after that it will just work?

If so, I propose we declare a week "cross-compile week" and only merge changes that make this work. Once everything builds on staging we merge to master and we're done?

Or is there more to it? Will there be things that break?

[Ericson2314]

The crossSystem vs hostPlatform things are actually all taken care of, now! What's left is:

• LLVM-based cross compilation #36867 LLVM-based cross compilation
• Change configureFlags on binutils and gcc to be [ "host" "build" "target" ]. Watch out for this arm (armv5 I think it is?) case where then things fail to build. Also do for ghc and any other compiler where we have configureFlags set today. Also make the various cases of targetPrefix unconditionally hostPlatform.config "-". (Must be done with the previous.

[Ericson2314]

stdenv.cross and `crossSystem are already sufficiently purged!

The first thing to do is just make all targetPrefixes unconditional, and also --target for compilers that might use it. Then start passing --build and --host to everything else. @lheckemann found a safer alternative for that second step.

[lheckemann]

Said alternative is setting the build_alias, host_alias and target_alias environment variables. However, I'm not sure if this is really a desirable solution, both because it'll only work for autotools-based configure scripts and because I'm not sure this way of setting the platforms is actually considered a public/documented/stable interface by autotools.

[wmertens]

So when everything is cross-compiled, will the hashes of something cross-compiled on platform A to platform T be the same as platform B to T?

…

On Tue, Mar 27, 2018 at 1:15 PM Linus Heckemann ***@***.***> wrote: Said alternative is setting the build_alias, host_alias and target_alias environment variables. However, I'm not sure if this is really a desirable solution, both because it'll only work for autotools-based configure scripts and because I'm not sure this way of setting the platforms is actually considered a public/documented/stable interface by autotools. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21471 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADWlkAUjoLYAcY44QQdUpda1jG27RR7ks5tih9pgaJpZM4LXFz4> .

[lheckemann]

No. The derivation hashes will be different, and as a result the output hashes will also be different. In an ideal scenario the only differences between cross-compiled and natively-compiled outputs will be the store paths, but I highly doubt that we'll achieve that in practice.

[wmertens]

Well, maybe we can achieve that with a CAS store, but only if the bytecode is 100% the same… It would be great if we could exclude the host platform from the input hashes…

…

On Tue, Mar 27, 2018 at 2:04 PM Linus Heckemann ***@***.***> wrote: No. The derivation hashes will be different, and as a result the output hashes will also be different. In an ideal scenario the only differences between cross-compiled and natively-compiled outputs will be the store paths, but I highly doubt that we'll achieve that in practice. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21471 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADWloXVPFtZbrT_4rK4ySTA-nSuIEG_ks5tiirYgaJpZM4LXFz4> .

[Ericson2314]

@wmertens Yes that is a goal, but it will be very hard. CAS / intentional store is necessary, but not sufficient, as the input binaries for the build tools would not be anywhere near bit-for-bit the same.

[Also, you mean build platform not host platform, BTW. The names are indeed awful.]

[lheckemann]

@dezgeg raised the concern of tests not being run when cross-compiling. Does exherbo maybe have a solution that we can steal?

[eternaleye]

@lheckemann: My experience on Exherbo has been that tests run just fine when build == host (build == run), and would likely run just fine any time host (run) is runnable (such as i686 or x32 on an x86_64 builder). That could be further expanded using binfmt_misc and qemu-user.

[Ericson2314]

I can't find it well on my phone, but the answer must be in https://git.exherbo.org/paludis/paludis.git/

[eternaleye]

This is the relevant bit, but is overconservative - it just checks for build == host (build == run); though it misnames 'host'/'run' as 'target'.

[Ericson2314]

@eternaleye err we mean how are (autoconf-based) configure scripts invoked. I see default_src_configure but not where the various env vars it uses are defined.

[eternaleye]

@Ericson2314: All the DEFAULT_SRC_CONFIGURE_* variables are set by individual packages. They aren't the defaults for src_configure; they allow packages to parameterize the default implementation of src_configure.

You probably want to look at the definition of econf instead.

[Ericson2314]

@eternaleye
https://git.exherbo.org/paludis/paludis.git/tree/paludis/repositories/e/ebuild/exheres-0/build_functions.bash?h=cross#n155 huh so --build and --host indeed are always passed if CBUILD and CHOST are defined. Do you all indeed never have configure run programs then? Brave!

[Gaelan]

I'm running into this when trying to compile for aarch32 on aarch64. I've been sending PRs to set configurePlatforms = ["host" "build"]; on a per-package basis, but is there anything stopping us from doing it globally at this point?

[wmertens]

Let's put it in staging and watch the world burn :)

[stale]

I marked this as stale due to inactivity. → More info

[Ericson2314]

Still pertinent, and in fact #87909 is getting new attention.

[stale]

I marked this as stale due to inactivity. → More info

[sternenseemann]

We did make some progress towards this: #123419, #119625, … (?)

[wmertens]

One future concern: would we be ok with declaring cross-compiled binaries as equivalent, no matter what the build platform was?

If so, provided we have a trusted mapping (being looked at with the intensional efforts), how would we go about generating all the input hashes that map to the same build?

[Ericson2314]

@wmertens I don't think we should calculate them up front. The space of possible build platforms is very, very large.

[wmertens]

@Ericson2314 another option would be to somehow replace the build platform input hash with a placeholder when calculating the input hash? Then all cross-compiling build tools for a target platform would map to the same dependency input hash?

[stale]

I marked this as stale due to inactivity. → More info

[Lahvuun]

I'll use this chance to mention that I've been looking into cross compilation with Nix, and my conclusion is that the build platform shouldn't go into an input hash at all. It makes no sense to have two bit-for-bit identical packages with different hashes simply because one was built natively and another one cross compiled. The host platform (on which the package will be used) should be the one included in the input hash. And for packages that have no native code (for example text files, like configuration or Python programs) platform shouldn't matter at all.

If you get rid of that, then the other source of difference is the compiler. Here, you basically want native and cross compilers built from the same source with the same codegen options to have different store paths, but affect input hashes in the same way. I don't think this is possible without extending Nix, and even then this kind of rule bending feels like it goes against the core Nix philosophy.

[Ericson2314]

@Lahvuun I suspect with CA derivations we could do "cache cheating as a plugin" to allow people to experiment with mixing builds without baking in logic in Nix contra to our morals.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixlang-how-do-you-find-all-uses-of-a-declaration/18369/13

[Mindavi]

This PR is just about done now: #87909

I'm thinking it might be merged just after branch-off of 22.05?

[sternenseemann]

Only a step towards it, however, since prefixing the compiler by default is still missing.

[Mindavi]

Yeah, that'll be an interesting exercise