Static executables on other platforms: Cross compilation via Nix

[wolfgangwalther]

We have been building a static executable for x64-linux for quite a while now. From this, the minimal docker image is created.

It would be really cool, if we could build the static executable on other platforms, too - and then create a multi-layer docker image, which provides minimal image size for different platforms.

One way to do this would be to run a nix environment on linux x64 and build the static executable for other platforms via cross-compiling.

This issue collects the current state of affairs.

---

I have been building GHC 9.4.8 as a cross-compiler for aarch64 successfully. However, when compiling PostgREST and its dependencies a major challenge comes up quickly: Template Haskell.

Template Haskell is a challenge, because it needs to be run during compilation. Because the cross-compiler compiles for the target platform, the compiled TH code can't be executed on the build platform. GHC will then fail like this:

Couldn't find a target code interpreter. Try with -fexternal-interpreter

There are a few different possible approaches how to deal with that:

1. Remove all Template Haskell from PostgREST and its dependencies. This is not easy, because a few transitive dependencies use TH and for example the whole hasql ecosystem depends on them. We also use it ourselves for the Paths_ module, although that could probably be worked around.
2. Compile and execute TH natively and then pass the resulting splices to the cross-compiler. The idea is, that most Template Haskell applications are merely "code generation", e.g. helpers to create a lot of similar shaped haskell code, which is then compiled to the actual program. Many of those uses are actually independent of the target system - so it could be possible to compile and execute those splices on the build system and then pass the generated code to the cross-compiler. I experimented with that and I was able to cross-compile almost all our dependencies that way. I was doing it manually and this quickly got very annoying, that's why I didn't finish this, yet. This could possibly be automated, but it would never be the "clean" solution. Once any of our dependencies would use "unsafe" Template Haskell, this might result in hard to debug errors.
3. GHC comes with a ghc-iserv executable. This is basically a wrapper around the GHCi library - so it allows to execute compiled code. For a GHC cross-compiler, this executable is built for the target platform. This can therefore be run with qemu and passed to -fexternal-interpreter. This should make Template Haskell just work. I tried this with GHC 9.4.8 and was able to make it work - but at some point hit a compiler bug in GHC 9.4.8 when cross-compiling to aarch64. My understanding is, that this bug was fixed in GHC 9.6 and 9.8 - but not in the 9.4 series. Therefore, I tried upgrading to GHC 9.6 / 9.8, but couldn't make that work, yet. GHC 9.6 doesn't build for pkgsStatic in nixpkgs, yet - and GHC 9.8 does build for pkgsStatic, but fails with the same TH-related error message as mentioned above, but already for pkgsStatic (no cross-platform-compiling involved, yet). This is tracked in GHC 9.6, 9.8, 9.10 TemplateHaskell doesn't work in pkgsStatic NixOS/nixpkgs#275304.

The last approach is the most promising in principle and is also tracked in NixOS/nixpkgs#248979 upstream.

[wolfgangwalther]

Once we can make this work, let's not forget Raspberry PI as a target: #3080 and #3086

[wolfgangwalther]

Remove all Template Haskell from PostgREST and its dependencies. This is not easy, because a few transitive dependencies use TH and for example the whole hasql ecosystem depends on them. We also use it ourselves for the Paths_ module, although that could probably be worked around.

I have been working on this a while ago. All Template Haskell is removed by now, except for hasql and swagger2. hasql is already updated upstream, we just need to update to the new version. To remove swagger2, we will need to do #1698.

[wolfgangwalther]

All Template Haskell is removed by now, except for hasql and swagger2. hasql is already updated upstream, we just need to update to the new version. To remove swagger2, we will need to do #1698.

The last time I tried updating to the new hasql version, I think the result was that we'd need to update to at least GHC 9.6 for the nix build. But the static build is currently broken for GHC 9.6 in nixpkgs - because of Template Haskell.

So it seems like we will need to solve the OpenAPI / swagger2 stuff first, before we can go ahead.