Build SDK with musl

[TheOneWithTheBraid]

This tracker is for issues related to:

• Common Front End (CFE) and kernel

• Dart VM

• dev_compiler

• Whether you are using: Void Linux x64 musl

It it somehow possible to build Dart using musl compiler instead of glibc?

[mraleph]

Right now we don't provide a way to build Dart with musl compiler.

[PureTryOut]

I've looked into this a bit and the initial blocker is the fact that the build process requires a pre-build dart binary dynamically linked to glibc. Thus you'd need to bootstrap Dart without using pre-built binaries, and so far I've not found instructions or any efforts to do so.

[CamilleScholtz]

Any update on this?

[PureTryOut]

Well, Alpine Linux has been shipping Dart for quite a bit.
https://pkgs.alpinelinux.org/package/edge/testing/x86_64/dart

Bootstrapped from the Windows version of Dart through Wine...

[a4z]

The wine/dart does not help really if you run docker on Mac M1
(yes I know that I can change the arch of a container)
also, adding wine might remove the size advantage, and it's a super ugly workaround

It would be really nice if dart-SDK would build on alpine. There are quite some other projects that require it and have now real inconvenient situations for users.
One example is static blog site generators that used the old ruby sass and now switch to dart sass. And ups, the container you had does not work anymore. Sory user. That is kind of really bad

[PureTryOut]

I agree, it's really, really bad. But it's a way to get something running. Musl itself isn't really a problem, Dart only needs a few small patches to get it working which should be easy enough to upstream.

The issue is the bootstrapping. Dart requires Dart to compile, but when Dart doesn't run on the platform... well you need to have a workaround. In Alpine's case this is Wine right now, but I rather do a full from-source bootstrap, starting from the very last version that doesn't require itself to compile (which is as far as I can see Dart 2.0.0-dev.8.0) and then going up version by version till you're on the most recent one. It's a better chain of trust but also makes it easier to compile it for new architectures, as Wine on anything non-x86 is not that great.
As I said earlier I looked into this and I have made quite a bit of progress since then, but didn't manage to finish it yet. Currently I have Dart 2.0.0-dev.56.0 running natively on Musl compiled through 3 earlier Dart versions, without requiring Wine or glibc anywhere. I'm stuck here however, the next step would've been Dart 2.0.0-dev-65.0 but it fails for me there at the moment.

It's a pain, and I wish Google cared at least a bit about this. 😢

[mraleph]

We could merge the necessary runtime patches (given that they are small).

As for bootstrapping. I don't think we will provide a full from-source solution, but we could theoretically provide a solution which does not require a prebuilt Dart binary: we simply need to maintain a fixed copy of Dart front-end compiled to Kernel AST, and update it whenever Kernel AST format changes or whenever we update SDK prebuilts. Bootstrapping process can then use this prebuilt CFE to compile the rest of Dart code.

I see that Alpine is quite popular for Docker, so maybe we should consider doing that. /cc @athomas @mit-mit

[PureTryOut]

Using a pre-built anything is not great from a distribution point of view, but it would be way better already than having to bootstrap from a Windows pre-built Dart. So I'd welcome that change yes, thanks.

I believe Alpine is the most popular image for Docker containers, but there are also people happy if Dart (and especially Flutter) would run on Alpine outside of Docker 😉

As for the patches used currently to built for Musl:

• https://git.alpinelinux.org/aports/tree/testing/dart/host-compiler.patch
• https://git.alpinelinux.org/aports/tree/testing/dart/musl-reinterpret.patch
• https://git.alpinelinux.org/aports/tree/testing/dart/musl-sigsetjmp.patch

That is with -Werror off, but I'd consider that small 😉

[PureTryOut]

Indeed, gcompat is a much better bootstrap than Wine already. Would pre-built binaries be provided somewhere so Alpine can built a Musl-native build without having a glibc system somewhere?

[a4z]

The Alpine Wiki mentions three ways to run glibc programs

• gcompat
• flatpack
• chroot

ref: https://wiki.alpinelinux.org/wiki/Running_glibc_programs

There are also several user projects, one that might be interesting for you could be
https://hub.docker.com/r/frolvlad/alpine-glibc/

I do not know if anything of that could help you with the bootstrap problem, but I thought it will also not harm to add this info here

[PureTryOut]

I know how to run such things, but the current glibc Dart doesn't run through gcompat. You need a Dart build with #50998 for that, a.k.a. not the binaries Google currently offers for download. Which is why I'm asking if they're going to offer binaries with tcmalloc disabled for download so that can be used to build a properly native Musl binary.

[leviem1]

I know it's bad practice to +1 these things, but I thought I might add that a few widely used Jekyll plugins have started upgrading their dependencies (e.g. jekyll-include-cache), which included a new transient dependency on the sass-embedded gem, which requires Dart, and most Jekyll build containers are Alpine, so you might be seeing more activity around this.

The workaround suggested was to pin the version of a dependency, which will get people by for now, but nobody should have to remain committed to this workaround forever for a number of reasons. We could all probably change our deployment pipelines, but I think this would affect enough people to justify the benefit of adding musl support.

Thanks for all of the hard work so far! Looks like you all have made some pretty awesome progress lately! Can't wait for this change!

[ntkme]

I started a project to release unofficial Dart SDK for musl: https://github.com/dart-musl/dart

Unofficial releases for musl is available here: https://github.com/dart-musl/dart/releases

Also available as alpine based container: docker pull ghcr.io/dart-musl/dart

---

The project works as follow:

• Bootstrap the previous version (currently 2.17.0) of Dart required to build new Dart on Debian with dart_use_tcmalloc=false.
• Patch the new version (currently 2.18.7):
  • Use correct cross compile target for alpine.
  • Use alpine provided host clang toolchain.
  • A few other minor changes...
• Replace debian based sysroot with alpine based sysroot.
• Replace the bootstrap SDK in source tree with the custom build with no tcmalloc.
• Install gcompat on alpine so that the bootstrap SDK works.
  • Note that this only works on 64-bit systems due to gcompat issue: https://git.adelielinux.org/adelie/gcompat/-/issues/357
• Copy 32-bit musl to 64-bit alpine to support cross compile 32-bit.
• Compile or cross-compile with dart_use_tcmalloc=false against alpine based sysroots.

The whole build and release process is done by GitHub Actions, and you can see the source code if you're interested .

I'd happy to contribute this to upstream, but without access to Google's internal infrastructure it is impossible to do it properly. The exact build steps and patch sets I used are open source, so if anyone from Google is willing to take it with Google's infrastructure to provide official musl support, please free feel.

---

Known issues:

The arm build do not work on qemu-user-static, which seems to be a long existing qemu bug where pthread_getattr_np keeps loop over mremap until the address become negative. A previous report of the same issue can be found here: https://www.openwall.com/lists/musl/2017/06/15/9.

There might also be other unknown issues. Please give it a try.

[mraleph]

@ntkme could you make PRs with code patches you have mentioned before? I am happy to merge them if they are reasonable.

We also started discussions internally on providing a bootstrap process which does not require a dart binary to be available (by publishing a Kernel binary version of CFE to CIPD and providing GN changes which allow you to use this binary instead of full dart binary to bootstrap). I can't provide concrete timeline on when this becomes available - but this is something we are going to look at.

[ntkme]

@mraleph That sounds great. It will take some time to prepare the PRs as what I did was to replace the Debian sysroot/etc with Alpine sysroot/etc. To upstream, I will have to change the build scripts to supports both.

[ntkme]

PR #51044 is up. I explicitly split it into 4 commits hope for an easier review, but looks like Gerrit just combine them as one. So I will briefly explain here:

1. Add script for downloading alpine sysroot. Replace boolean flag dart_use_debian_sysroot with string flag dart_sysroot, which can be "", "alpine" or "debian". Setup compiler target & flags for dart_sysroot == "alpine".
2. Force use alpine's host clang when dart_sysroot == "alpine", as Google's provided clang doesn't work.
3. No -Werror when dart_sysroot == "alpine", as musl has different headers than glibc which caused quite a lot warnings.
4. Fix reinterpret_cast of NULL.

[ntkme]

Dart with tcmalloc can be built against on musl with a proper re-configure described in the documentation: https://github.com/dart-lang/sdk/blob/main/third_party/tcmalloc/README.dart

The document mentioned:

Make sure that include/config.h defines HAVE_UCONTEXT_H on Linux,

However, using the provided configure_command does not define HAVE_UCONTEXT_H: https://github.com/dart-lang/sdk/blob/main/third_party/tcmalloc/configure_command

It turns out --enable-cpu-profiler is required to run AC_PC_FROM_UCONTEXT, which will define HAVE_UCONTEXT_H:
https://github.com/gperftools/gperftools/blob/bf8b714bf5075d0a6f2f28504b43095e2b1e11c5/configure.ac#L270-L273