Release musl builds

[hayd]

Previously I was able to use --target x86_64-unknown-linux-musl on rusty_v8 and deno (excluding the plugin). denoland/rusty_v8#49

It's unclear to me if it's best to only target x86_64-unknown-linux-musl or to additionally support it. e.g. sccache seems to only provide binaries for that target.

The one concern is that glibc is required for plugins (but I don't know if that means glibc cannot be used on a binary created with musl 🤷‍♂ ).

This is useful for platforms without glibc, examples alpine linux and amazon linux (which is occasionally a pain building, so would be helpful if a compatible binaries was in deno's CI).

xlink: #3243 #1658 #1495 #3356

[hayd]

Some progress using rust-musl-builder, but run into GLIBC issue (maybe?):

https://gist.github.com/hayd/ae5cbe81117863fff98c2f0c877f2b34

/usr/bin/ld: /home/rust/src/deno/target/x86_64-unknown-linux-musl/release/deps/deno-9173e3687c7172b6: hidden symbol `__dso_handle' isn't defined
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

Note: In this context it seems like rusty_v8 builds:

cargo build --package rusty_v8
   Compiling ... #snipped 
   Compiling rusty_v8 v0.1.0
    Finished dev [unoptimized + debuginfo] target(s) in 33m 20s

Not sure what that means...

But it seems like it's the deno(bin) that fails (rather than rusty_v8):

Building [=====================================================> ] 256/257: deno(bin)
# same failure

[bnoordhuis]

Not sure if it's helpful but __dso_handle is normally provided by libgcc's (not glibc's) crtbegin.o. It sounds like that's not getting linked in on your system?

On my system:

$ dpkg -S `cc -print-file-name=crtbegin.o`
libgcc-9-dev:amd64: /usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o

$ nm -o `cc -print-file-name=crtbegin.o` | grep __dso_handle
/usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o:0000000000000000 D __dso_handle

[hayd]

🤷‍♂ I think the objective is for musl to statically link, specifically not to link them to that specific system... this might already be possible with some flags?

(The environment is in the Dockerfile.)

cc @chrmoritz ?

[bnoordhuis]

I think the objective is for musl to statically link

crtbegin.o is linked in statically, not dynamically.

[hayd]

It is in the docker image too, perhaps an env issue? (e.g. RUST_FLAGS or CFLAGS? 😬 )

rust@c3b60eef5da1:~/src/deno/cli$ dpkg -S `cc -print-file-name=crtbegin.o`
libgcc-7-dev:amd64: /usr/lib/gcc/x86_64-linux-gnu/7/crtbegin.o
rust@c3b60eef5da1:~/src/deno/cli$ nm -o `cc -print-file-name=crtbegin.o` | grep __dso_handle
/usr/lib/gcc/x86_64-linux-gnu/7/crtbegin.o:0000000000000000 D __dso_handle

rust@c3b60eef5da1:~/src/deno/cli$ env
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
LESSCLOSE=/usr/bin/lesspipe %s %s
GN_ARGS=clang_use_chrome_plugins=false treat_warnings_as_errors=false use_sysroot=false use_glib=false use_gold=true clang_base_path="/tmp/clang"
HOSTNAME=c3b60eef5da1
RUST_BACKTRACE=full
GN_VERSION=latest
PKG_CONFIG_ALL_STATIC=true
DENO_VERSION=0.31.0
OPENSSL_DIR=/usr/local/musl/
LIBZ_SYS_STATIC=1
OPENSSL_LIB_DIR=/usr/local/musl/lib/
NINJA=/bin/ninja
PWD=/home/rust/src/deno/cli
PKG_CONFIG_ALLOW_CROSS=true
HOME=/home/rust
PG_CONFIG_X86_64_UNKNOWN_LINUX_GNU=/usr/bin/pg_config
CLANG_BASE_PATH=/tmp/clang
DEP_OPENSSL_INCLUDE=/usr/local/musl/include/
TERM=xterm
SHLVL=1
DENO_BUILD_MODE=release
PQ_LIB_STATIC_X86_64_UNKNOWN_LINUX_MUSL=1
TARGET=musl
GN=/bin/gn
PATH=/home/rust/.cargo/bin:/usr/local/musl/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NINJA_VERSION=1.8.2
OPENSSL_INCLUDE_DIR=/usr/local/musl/include/
OPENSSL_STATIC=1
LESSOPEN=| /usr/bin/lesspipe %s
_=/usr/bin/env

[hayd]

@jeromegn Do you have any tips/suggestions here, noting your comment:
emk/rust-musl-builder#65 (comment)

Did the musl binaries you created for fly work on other linux systems?

[jeromegn]

I'm not sure I have much to add. I mostly used the linked Dockerfile and it worked. Inspired (read: ripped off) from: https://hub.docker.com/r/alexmasterov/alpine-libv8

That said, there are alpine images with glibc compiled in there, that might help. It's a lot of trial and error to get it to compile. And then the performance can vary a bit between musl and glibc.

[kesor]

If anyone is interested, I have compiled DENO with V8 in Alpine Linux Docker container - without glibc, but completely using the Alpine muslc library.

https://gist.github.com/kesor/68df53a5d76784a235ca6b0e7efed4d9

Enjoy!

[hayd]

@kesor This is great, good work! Do these (or some of these) options make sense to be incorporated as gn args?
That patch looks scary to maintain. :)

[kesor]

@kesor This is great, good work! Do these (or some of these) options make sense to be incorporated as gn args?
That patch looks scary to maintain. :)

I spent hours trying to find GN args to pass via env var or some other way. Couldn’t find how. My knowledge of GN is limited, probably someone with more experience can find the way. Didn’t mean to provide that patch as “the solution” only as a proof of concept.

PS: Alpine has other llvm versions as well, maybe with llvm8 the binary will be smaller?

Also note that the eventual binary still has dependency on libgcc at the least, maybe other dependencies I didn’t quite catch, it does run with the provided docker image resulting from the Dockerfile build.

[hayd]

In llvm10 there is no_inline_line_tables which shrinks the binary.
I think the solution is that BUILD.gn is modified to allow arguments that support what your diff does :)

Are both these required? Do you have any idea of the reasons?

-      ldflags += [ "-Wl,--color-diagnostics" ]
+      ldflags += [  ]

-  libs = []
+  libs = ["execinfo"]

[kesor]

In llvm10 there is no_inline_line_tables which shrinks the binary.
I think the solution is that BUILD.gn is modified to allow arguments that support what your diff does :)

Are both these required? Do you have any idea of the reasons?

-      ldflags += [ "-Wl,--color-diagnostics" ]
+      ldflags += [  ]

-  libs = []
+  libs = ["execinfo"]

Color diagnostics was reported as an invalid option for the linker. Wouldn’t work with it. I also think that there was a different linker used for parts of the build for some reason. Maybe that can be the root cause and resolving it can solve the arguments issue.

libexecinfo was also required because of failing linking. I don’t really know the reason why.

The most annoying thing was the custom location of the header files which was not automatically detected, and I couldn’t find a way to provide an include path via GN args. I would also assume that this path will change for different llvm/alpine versions.

[hayd]

I would also assume that this path will change for different llvm/alpine versions.

Yes, I think we'd want to pass this (as a GN_ARGS)...?

@ry do you have any pointers in adding these/or suggest whether this a good/bad way forward?

https://gist.github.com/kesor/68df53a5d76784a235ca6b0e7efed4d9#file-rusty-changed-build-diff

[nathanblair]

Also note that the eventual binary still has dependency on libgcc at the least

Is this referring to the libgcc alpine package? I wanted to avoid this dependency by compiling with musl and the linux musl headers in order for deno to be a fully static binary. This can be achieved with node but I'm not sure how far down the rust toolchain requires glibc.

[kesor]

Also note that the eventual binary still has dependency on libgcc at the least

Is this referring to the libgcc alpine package? I wanted to avoid this dependency by compiling with musl and the linux musl headers in order for deno to be a fully static binary. This can be achieved with node but I'm not sure how far down the rust toolchain requires glibc.

@nathanblair Note that libgcc is not the same as glibc, whereas musl is an alternative glibc, there is no "special" alternative for libgcc on Alpine.

[nizox]

Are both these required? Do you have any idea of the reasons?

-  libs = []
+  libs = ["execinfo"]

I had a similar problem with execinfo and was able to workaround it with:
nizox/v8@f66447b

Also, you should be able to replace the libgcc dependency with libunwind: nizox/chromium_build@39a009c

[lygstate]

Can deno static link to c library to resolve issue like golang does?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hayd]

I think this one could probably stay open?

[Cloudef]

I have managed to build rusty_v8.a for musl, but deno fails with:

error: /build/source/target/release/deps/libdeno_ops-4e7e641473573b40.so: cannot allocate memory in static TLS block

[Cloudef]

Works after switching to older zig (version 0.8.1)

/nix/store/zqik2z7dxn57z66kqf0qymjbfb8smmlz-hello-world-0.1.0-aarch64-unknown-linux-musl/bin/lambda-hello-world: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped

[130] void@voidlinux> ./result/bin/lambda-hello-world   
v8 version: 10.8.168.4
thread 'main' panicked at 'Missing AWS_LAMBDA_FUNCTION_NAME env var: NotPresent', /sources/lambda_runtime-0.7.0/src/lib.rs:62:65
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
qemu: uncaught target signal 6 (Aborted) - core dumped

Attached is
librusty_v8-aarch64-unknown-linux-musl.zip
librusty_v8.a for aarch64-unknown-linux-musl. It should work with v0.54.0 so be careful. I'll put up my toolchain and build files some other place at some point.

[Cloudef]

Succesfully deployed and working in AWS:

jari@jv-m1-mbp> curl -i -X POST -H 'Content-Type: application/json' -d '{"src": "Deno.core.ops.v8_version()"}' redacted.url
HTTP/2 200
date: Fri, 11 Nov 2022 07:57:05 GMT
content-type: application/json
content-length: 23
x-amzn-requestid: eb28be2d-1d08-468e-a7b0-d98749ba3f2d
x-amz-apigw-id: bbT8iGwbNjMFTPQ=
x-amzn-trace-id: Root=1-636e0050-624e648873eac695624d1e45;Sampled=0

{"result":"10.8.168.4"}

[Propolisa]

Succesfully deployed and working in AWS:

jari@jv-m1-mbp> curl -i -X POST -H 'Content-Type: application/json' -d '{"src": "Deno.core.ops.v8_version()"}' redacted.url
HTTP/2 200
date: Fri, 11 Nov 2022 07:57:05 GMT
content-type: application/json
content-length: 23
x-amzn-requestid: eb28be2d-1d08-468e-a7b0-d98749ba3f2d
x-amz-apigw-id: bbT8iGwbNjMFTPQ=
x-amzn-trace-id: Root=1-636e0050-624e648873eac695624d1e45;Sampled=0

{"result":"10.8.168.4"}

Hey @Cloudef, does this indicate you've gotten a musl build working? I would be much happier to continue with a current pipeline automation project with Deno rather than Golang, but the complexity + size of gcc requirement is currently pushing me toward the latter. Musl builds would make life a lot easier, even if small things sometimes break :)

[Cloudef]

@Propolisa I've got deno_core to build, I don't use deno directly myself, but I guess deno could be able to build as well, since it just uses deno_core as the problematic dep?

I pushed the build files here: https://github.com/Cloudef/nix-zig-stdenv/tree/master/sketchpad
If you aren't familiar with nix, then I suggest reading a bit about it first. The default.nix file in that folder is commented.

One thing to note is that, building rusty-v8 from source needs qemu-binfmt, this may need to be workarounded in future. I only support aarch64 for now as well. When building rust project with deno_core, you may need to provide older zig with --argstr zig-version 0.8.1 on linux. This seems to be a bug in zig's recent linux linker.

To use the prebuilt static lib, run nix store add-file librusty_v8-aarch64-unknown-linux-musl.a this adds it to the nix store.

I've tested building on both aarch64-darwin and x86_64-linux and confirmed the builds working. rusty-v8 obviously can't be build from aarch64-darwin due to the qemu-binfmt requirement, but you can use the produced static lib after.

TL;DR

Get nix from https://nix.dev/tutorials/install-nix

I want to use prebuilt librusty-v8

curl -O 'https://github.com/denoland/deno/files/9923527/librusty_v8-aarch64-unknown-linux-musl.zip' 
unzip librusty_v8-aarch64-unknown-linux-musl.zip
nix store add-file librusty_v8-aarch64-unknown-linux-musl.a
git clone https://github.com/Cloudef/nix-zig-stdenv.git
cd nix-zig-stdenv/sketchpad
# you might need '--argstr zig-version 0.8.1' here if you are on linux
nix-build -A deno-core-hello-world --argstr target aarch64-unknown-linux-musl
file result/bin/deno-core-hello-world
# result/bin/deno-core-hello-world: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped

I want to build everything from scratch

You need to be on a linux system for this with qemu-binfmt emulation set for aarch64 binaries.
qemu-binfmt setup depends on the distro, go read the docs, or you can just be on a aarch64 linux system :)

git clone https://github.com/Cloudef/nix-zig-stdenv.git
cd nix-zig-stdenv/sketchpad
nix-build -A deno-core-hello-world --arg with-rusty-v8 true --argstr target aarch64-unknown-linux-musl
file result/bin/deno-core-hello-world
# result/bin/deno-core-hello-world: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped

I don't care about the hello-world, give me the static lib for rusty-v8

git clone https://github.com/Cloudef/nix-zig-stdenv.git
cd nix-zig-stdenv/sketchpad
nix-build -A rusty-v8 --argstr target aarch64-unknown-linux-musl
ls result/lib/librusty_v8.a
# result/lib/librusty_v8.a

Wait what is this nix? Why do I need it

Simply put, it makes your life easier. Go learn it.

[Propolisa]

@Cloudef really appreciate the quality of this reply. I am short for free time to set this environment up, but perhaps another may be able to use your writeup to attempt building deno itself sooner than I. Cheers!

[benatkin]

Hmm, it seems like a true Deno alpine container image might be just around the corner. The current one uses FROM:alpine-glibc and I'm not sure that even belongs in the official Docker project. It's nice but it's based on an unofficial base image and IMO the official Deno Docker project should only use official base images.

[mdekstrand]

I have successfully built a musl binary (on Alpine) that passes the WPT tests (although it does not pass cargo test yet). The only source code edit needed was the one I raised in #17739; the rest could be handled with environment variables:

export GN_ARGS='use_custom_libcxx=false v8_enable_backtrace=false v8_enable_debugging_features=false use_lld=false symbol_level=0 v8_builtins_profiling_log_file=""'
export CLANG_BASE_PATH=/usr
export V8_FROM_SOURCE=1
export TARGET=x86_64-alpine-linux-musl

[jirutka]

I’ve packaged Deno for Alpine Linux (package deno); it’s currently available in the testing repository (in edge). All tests pass on x86_64, a few tests fail on aarch64.

[danopia]

I’ve packaged Deno for Alpine Linux (package deno); it’s currently available in the testing repository (in edge).

This is lovely for testing, thank you!
[edit]
And now that it's moved from testing to community, super easy to use in a basic Alpine Linux Dockerfile:

FROM alpine:edge
RUN apk add --no-cache deno

For comparison, this produces a 95MB image vs the 127MB denoland/deno:alpine-1.35.1 image. I'll give it a spin by deploying on some x86_64.

[mokeyish]

I use openwrt, I hope the official can provide musl builds.

[jirutka]

I use openwrt, I hope the official can provide musl builds.

You can package it yourself for OpenWrt…

[mokeyish]

I use openwrt, I hope the official can provide musl builds.

You can package it yourself for OpenWrt…

It's very hard for me, many errors.

[jirutka]

Did you use patches from the deno aport on Alpine Linux?

[arkxfly]

Did you use patches from the deno aport on Alpine Linux?

I copy deno from alpine to openwrt. it throws

Error loading shared library libzstd.so.1: No such file or directory (needed by ./deno)
Error loading shared library libffi.so.8: No such file or directory (needed by ./deno)
Error loading shared library libz-ng.so.2: No such file or directory (needed by ./deno)
Error loading shared library libsqlite3.so.0: No such file or directory (needed by ./deno)
Error loading shared library libicui18n.so.73: No such file or directory (needed by ./deno)
Error loading shared library libicuuc.so.73: No such file or directory (needed by ./deno)
Error relocating ./deno: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating ./deno: _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating ./deno: _ZNKRSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEE3strEv: symbol not found
Error relocating ./deno: _ZNKRSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEE3strEv: symbol not found
Error relocating ./deno: _ZNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEEC2ERKNS_12basi

[CosmicToast]

it throws

None of these are the build system, you have some missing libraries (your package definition likely lacks them as runtime dependencies) and have some relocation errors in the c++ standard library (usually suggesting it wasn't linked or available, which is likely given the other errors).
It ultimately has little to do with deno itself, and more to do with your platform and packaging.

[szmarczak]

I don't think FFI works on Alpine because:

1. https://github.com/denoland/deno/blob/708a6196930a1267e577805aed380e3715eaae10/ext/ffi/dlfcn.rs#L15C5-L15C12
2. https://github.com/OpenByteDev/dlopen2/blob/e2167bfcb30c062f81e44e1d8195f3781956fb71/dlopen2/src/raw/unix.rs#L5
3. https://github.com/rust-lang/libc/blob/acc7bb1a38e2b9d6a18f28279d80d464c4c25089/src/unix/mod.rs#L1153
4. https://git.musl-libc.org/cgit/musl/tree/src/ldso/dlopen.c?h=v1.2.5#n6

[CosmicToast]

I don't think FFI works on Alpine because:

1. https://github.com/denoland/deno/blob/708a6196930a1267e577805aed380e3715eaae10/ext/ffi/dlfcn.rs#L15C5-L15C12
2. https://github.com/OpenByteDev/dlopen2/blob/e2167bfcb30c062f81e44e1d8195f3781956fb71/dlopen2/src/raw/unix.rs#L5
3. https://github.com/rust-lang/libc/blob/acc7bb1a38e2b9d6a18f28279d80d464c4c25089/src/unix/mod.rs#L1153
4. https://git.musl-libc.org/cgit/musl/tree/src/ldso/dlopen.c?h=v1.2.5#n6

This is a misunderstanding. Musl (like every other libc currently available) only allows for dlopen in dynamically linked executables. Linked above is the stubbed implementation that's present when statically linking against musl, while the dynamic version can be found here.

Whether your deno will run FFI code based on dlopen will thus depend on how you build deno.

It's technically possible to have a statically linked executable that can call dlopen, but you'd need to build linker capabilities into the libc to make this happen, and then you may have incompatibilities (e.g. re: memory management) when actually loading said extensions.
Regardless, no one's done this on Linux so far (there are more exotic platforms that have comparable capabilities, but they're out of scope for the topic).

[szmarczak]

Looks like you're right! Sorry for confusion.

[balupton]

I’ve packaged Deno for Alpine Linux (package deno); it’s currently available in the testing repository (in edge). All tests pass on x86_64, a few tests fail on aarch64.

@jirutka any chance of getting a more modern Deno version released to your apk?

[jd1378]

probably solves denoland/deno_docker#373 and denoland/deno_docker#350