Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make AppImages run on Alpine #1015

Closed
probonopd opened this issue Dec 8, 2019 · 32 comments · May be fixed by #1168
Closed

Make AppImages run on Alpine #1015

probonopd opened this issue Dec 8, 2019 · 32 comments · May be fixed by #1168

Comments

@probonopd
Copy link
Member

Alpine has libc6-compat which provides ld-linux, so in theory it should run there, but in practice we are getting:

/patchelf-0.10 # apk add libc6-compat
(1/1) Installing libc6-compat (1.1.22-r3)
OK
/patchelf-0.10 # ./appimagetool-390-x86_64.AppImage 
Error relocating ./appimagetool-390-x86_64.AppImage: gnu_dev_makedev: symbol not found
@probonopd
Copy link
Member Author

Documentation says:

Conforming to
The makedev(), major(), and minor() functions are not specified in POSIX.1, but are present on many other systems.
Notes
These interfaces are defined as macros. Since glibc 2.3.3, they have been aliases for three GNU-specific functions: gnu_dev_makedev(), gnu_dev_major(), and gnu_dev_minor(). The latter names are exported, but the traditional names are more portable.

@TheAssassin
Copy link
Member

Would a statically linked binary (e.g., using musl internally) work on every system? Can we safely assume that? I mean, we have a dependency on FUSE, but that's more API than ABI, and any FUSE client should be able to talk to any distro, no matter what libc they use.

@TheAssassin
Copy link
Member

CC #877

@CosmicToast
Copy link

There is a library named libgcompat that provides most of the additional (non-standard) features in glibc for musl-based systems.
When I was trying to do some fairly specific stuff it got me a bit further in the process (ncurses would act weird), but it should be most of the solution here - https://code.foxkit.us/adelie/gcompat
(There's also a GitHub mirror)

I think the optimal solution will eventually be to build things against musl (ideally statically) and package all the deps in the appimage.
That way it'll even run on "older" distributions, and has less of an associated cost (due to the size and scope of musl and co.).
Appimages will always take up more space than "native" counterparts, but it makes up for it immensely using everything it brings in - leaning into that aspect seems natural, especially given how cheap storage is, relative to everything else (within reasonable bounds, but I do encourage commenters to-be to look at their filesystem usage patterns).

@CosmicToast
Copy link

Would a statically linked binary (e.g., using musl internally) work on every system? Can we safely assume that?

That's a fairly safe assumption. ABI incompatibilities are a thing, as well as kernel headers, but I've done some fairly extreme things in the past (such as statically linked shells and utilities for use in a system that hasn't been updated in about a decade).
Musl allowing true static linking is very important relative to glibc in that use-cases.

@TheAssassin
Copy link
Member

The problem with glibc is that we need the right ld-linux as linker, too, right? I don't know exactly how musl works in detail.

I think we need to finally extract and separate the runtime from the rest of the code, then we can create the regular builds with their semi-static linking and also experiment with truly static runtime builds.

@probonopd
Copy link
Member Author

Some instinct tells me that sooner or later we'll end up making the runtime a version of ld-linux, so that it doesn't have to rely on one from the system... (musl's is even MIT licensed)

@CosmicToast
Copy link

CosmicToast commented Feb 19, 2020

The problem with glibc is that we need the right ld-linux as linker, too, right? I don't know exactly how musl works in detail.

That gcompat project I mentioned provides a linker (compatibility stub*) too, but yes, that's part of it.

I think we need to finally extract and separate the runtime from the rest of the code, then we can create the regular builds with their semi-static linking and also experiment with truly static runtime builds.
Some instinct tells me that sooner or later we'll end up making the runtime a version of ld-linux, so that it doesn't have to rely on one from the system... (musl's is even MIT licensed)

Both of these sound interesting.
Also, consider me available for general help (pings and such) - I believe the general concept of AppImages to be the best way for general linux distribution, for a variety of reasons.
I'm also a maintainer of a few packages on Alpine Linux and the co-founder of Abyss OS - in which we plan to use the general AppImage idea (not necessarily the main implementation, it's a bit early to decide on that) to distribute most user-facing applications (by way of packaging them ourselves), so I'm somewhat invested as it is 🙂.

I'm fairly busy, but knowing where and what to poke at to understand the codebase would be useful as well, if possible (as much as I abhor C).

@TheAssassin
Copy link
Member

@5paceToast thanks for your offer. We've already found enough reasons to make a AppImage type 3, in which we intend to solve a lot of problems. Your input will be welcome.

@TheAssassin
Copy link
Member

I've started a draft, comments welcome: https://github.com/TheAssassin/type3-runtime

(Please continue the discussion there, here it's off topic.)

@probonopd
Copy link
Member Author

I don't think we need a new type in order to address the points mentioned in this thread; this imho all can be achieved by a new runtime for type 2 images (i.e., a squashfs or other filesystem appended to an ELF binary).

@wenerme
Copy link

wenerme commented Feb 26, 2021

About one year now, any progress here to support musl ?

@probonopd
Copy link
Member Author

Looks like no one is working on it.

@vinniec
Copy link

vinniec commented Mar 10, 2021

Has anyone tried with this? https://github.com/sgerrand/alpine-pkg-glibc
I tried without a positive result but I actually don't know if it should have worked or not.

@anki-code
Copy link

anki-code commented Mar 10, 2021

Hi @vinniec! Thank you for the link. Yes! It works!

I've made a test with the xonsh shell and report the successfull result - sgerrand/alpine-pkg-glibc#153 (comment)

@mensinda
Copy link

mensinda commented Aug 8, 2021

There is also https://github.com/mensinda/libRuntime, which will give you a statically linked runtime that can be passed to appimagetool with --runtime-file.

However, this won't really make a difference if the actual packaged program also depends on glibc, since the host version of the c library is used in general. The way I work around this is by also packaging musl or glibc (including ld-linux.so) and then launching LD_LIBRARY_PATH=.../AppDir/usr/lib /path/to/ld-linux.so .../AppDir/usr/bin/myprog.

@probonopd
Copy link
Member Author

@mensinda do you have such an AppImage for download somewhere? I'd like how it fares on helloSystem (FreeBSD).

@mensinda
Copy link

@probonopd Yes, I have some "AppImages" here. For reference, they are as a demonstration for mesonbuild/meson#7518.

I am not sure if they really are AppImages according to the spec, since I am packaging python3, ninja, pkgconf, and Meson. Also, the "AppImage" self extracts when an actual Meson project is configured.

These binaries are built in the latest Alpine docker container and the complete c library is packaged (including the musl *ld-linux.so). However, instead of relying on LD_LIBRARY_PATH (which messes stuff up for Meson, since we call other binaries outside the AppDir) I modified the rpath instead.

I have successfully tested the 64bit binary on Ubuntu 10.04 (not that you can actually do anything useful there).

@toniz4
Copy link

toniz4 commented Sep 13, 2021

Any reason why this doesn't work? I'm missing a package or something like that?

/tmp λ curl -LO 'https://github.com/AppImage/AppImageKit/releases/download/continuous/appimagetool-x86_64.AppImage'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   632  100   632    0     0   1468      0 --:--:-- --:--:-- --:--:--  1466
100 2121k  100 2121k    0     0  1677k      0  0:00:01  0:00:01 --:--:-- 7954k
/tmp λ chmod +x appimagetool-x86_64.AppImage
/tmp λ ./appimagetool-x86_64.AppImage
This doesn't look like a squashfs image.

Cannot mount AppImage, please check your FUSE setup.
You might still be able to extract the contents of this AppImage
if you run it with the --appimage-extract option.
See https://github.com/AppImage/AppImageKit/wiki/FUSE
for more information
open dir error: No such file or directory
/tmp λ ldd appimagetool-x86_64.AppImage
        /lib64/ld-linux-x86-64.so.2 (0x7fc91d075000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7fc91d075000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fc91d075000)
        libz.so.1 => /lib/libz.so.1 (0x7fc91d05b000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fc91d075000)
Error relocating appimagetool-x86_64.AppImage: gnu_dev_makedev: symbol not found
/tmp λ patchelf --add-needed libgcompat.so.0 appimagetool-x86_64.AppImage
/tmp λ ldd appimagetool-x86_64.AppImage
        /lib64/ld-linux-x86-64.so.2 (0x7f829bb73000)
        libgcompat.so.0 => /lib/libgcompat.so.0 (0x7f829bb61000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f829bb73000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f829bb73000)
        libz.so.1 => /lib/libz.so.1 (0x7f829bb47000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f829bb73000)
        libucontext.so.1 => /lib/libucontext.so.1 (0x7f829bb42000)
        libobstack.so.1 => /usr/lib/libobstack.so.1 (0x7f829bb3d000)
/tmp λ ./appimagetool-x86_64.AppImage
This doesn't look like a squashfs image.

Cannot mount AppImage, please check your FUSE setup.
You might still be able to extract the contents of this AppImage
if you run it with the --appimage-extract option.
See https://github.com/AppImage/AppImageKit/wiki/FUSE
for more information
open dir error: No such file or directory
/tmp λ

Unlike libc6-compat, gcompat defines gnu_dev_makedev, so why it doesn't work?

@probonopd
Copy link
Member Author

Great question. I don't know the answer, sorry.

@xproot
Copy link

xproot commented Sep 16, 2021

xproot@cedric:~/AppImages$ ./browservice-v0.9.2.2-aarch64.AppImage 
bash: ./browservice-v0.9.2.2-aarch64.AppImage: No such file or directory
xproot@cedric:~/AppImages$ ls
browservice-v0.9.2.2-aarch64.AppImage

Is this like a well known alpine issue?

@CosmicToast
Copy link

xproot@cedric:~/AppImages$ ./browservice-v0.9.2.2-aarch64.AppImage 
bash: ./browservice-v0.9.2.2-aarch64.AppImage: No such file or directory
xproot@cedric:~/AppImages$ ls
browservice-v0.9.2.2-aarch64.AppImage

Is this like a well known alpine issue?

This is a well known linux issue.
Exec failures are reported this way.
Usually, it's a consequence of wrong architecture or missing libraries/symbols - normally you'd try running it through file(1) and ldd(1).

However, in this case, it's likely using the original stub, which dynamically links to glibc (which is obviously not present), which is indeed what this (unresolved, unlikely to be resolved) bug is about.
Similarly, you would find running musl-dynlinked binaries a challenge on a glibc distribution without installing musl.

@toniz4
Copy link

toniz4 commented Sep 16, 2021

xproot@cedric:~/AppImages$ ./browservice-v0.9.2.2-aarch64.AppImage 
bash: ./browservice-v0.9.2.2-aarch64.AppImage: No such file or directory
xproot@cedric:~/AppImages$ ls
browservice-v0.9.2.2-aarch64.AppImage

Is this like a well known alpine issue?

I get this when not using gcompat. Using gcompat i get the error shown above.

s-zeid added a commit to s-zeid/AppImageKit that referenced this issue Jan 28, 2022
When running an AppImage under [gcompat][0] (e.g. on Alpine Linux),
`fopen("/proc/self/exe", "rb")` opens the dynamic linker (e.g.
`/lib/ld-musl-x86_64.so.1`) instead of the AppImage itself.  However,
`readlink("/proc/self/exe", ...)` does give the path to the AppImage as
expected, even under gcompat.

This commit fixes this problem by always using the link target for
`/proc/self/exe` in places that read the AppImage instead of the link
itself.  Without this commit, running an AppImage under gcompat results
in [the error message "This doesn't look like a squashfs image." from
squashfuse][1].  With this commit, AppImages run as expected under
gcompat.

In order to make `--appimage-help` and
`--appimage-portable-{home,config}` work under gcompat, I also move the
calculation of the `fullpath` variable in `main()` to earlier in the
function and change `print_help()` and `portable_option()` to use it
instead of calculating the fullpath separately.  (When
`$TARGET_APPIMAGE` is set, since `realpath()` is (already) used instead
of `readlink()` in that case, this change could result in a different
path being used in help output and when _creating_ the portable home and
config directories with the respective command line options, but
`fullpath` is already being used to find existing portable directories
when running an AppImage, so this should not affect existing portable
installations.)  For consistency, I also rename the `fullpath` variable
in `main()` and the corresponding arguments in `print_help()` and
`portable_option()` to `appimage_fullpath`.

Fixes AppImage#1015 on Alpine Linux systems with gcompat installed, for
AppImages made with this changeset applied.

Tested on Alpine Linux edge x86_64 and postmarketOS (based on Alpine)
edge aarch64.

[0]: https://git.adelielinux.org/adelie/gcompat/
[1]: https://github.com/vasi/squashfuse/blob/e51978cd6bb5c4d16fae9eee43d0b258f570bb0f/util.c#L81-L82
s-zeid added a commit to s-zeid/AppImageKit that referenced this issue Jan 28, 2022
When running an AppImage under [gcompat][0] (e.g. on Alpine Linux),
`fopen("/proc/self/exe", "rb")` opens the dynamic linker (e.g.
`/lib/ld-musl-x86_64.so.1`) instead of the AppImage itself.  However,
`readlink("/proc/self/exe", ...)` does give the path to the AppImage as
expected, even under gcompat.

This commit fixes this problem by always using the link target for
`/proc/self/exe` in places that read the AppImage instead of the link
itself.  Without this commit, running an AppImage under gcompat results
in [the error message "This doesn't look like a squashfs image." from
squashfuse][1].  With this commit, AppImages run as expected under
gcompat.

In order to make `--appimage-help` and
`--appimage-portable-{home,config}` work under gcompat, I also move the
calculation of the `fullpath` variable in `main()` to earlier in the
function and change `print_help()` and `portable_option()` to use it
instead of calculating the fullpath separately.  (When
`$TARGET_APPIMAGE` is set, since `realpath()` is (already) used instead
of `readlink()` in that case, this change could result in a different
path being used in help output and when _creating_ the portable home and
config directories with the respective command line options, but
`fullpath` is already being used to find existing portable directories
when running an AppImage, so this should not affect existing portable
installations.)  For consistency, I also rename the `fullpath` variable
in `main()` and the corresponding arguments in `print_help()` and
`portable_option()` to `appimage_fullpath`.

Fixes AppImage#1015 on Alpine Linux systems with gcompat installed, for
AppImages made with this commit applied.

Tested on Alpine Linux edge x86_64 and postmarketOS (based on Alpine)
edge aarch64.

[0]: https://git.adelielinux.org/adelie/gcompat/
[1]: https://github.com/vasi/squashfuse/blob/e51978cd6bb5c4d16fae9eee43d0b258f570bb0f/util.c#L81-L82
@s-zeid
Copy link
Contributor

s-zeid commented Feb 1, 2022

I've also opened https://git.adelielinux.org/adelie/gcompat/-/issues/349 for this.

@probonopd
Copy link
Member Author

If I understand it right, it Looks like https://github.com/eth-cscs/spack-batteries-included is providing a solution for this. Should we backport these changes into the AppImage runtime?

Differences and improvements over AppImage runtime
spack.x uses zstd for faster decompression;
spack.x itself is an entirely static binary;
spack.x does not need to dlopen libfuse.so

Reference:
#1120 (comment)
cc @haampie

@probonopd
Copy link
Member Author

probonopd commented May 7, 2022

I believe the AppImages from https://github.com/probonopd/go-appimage/releases/tag/continuous which are using the experimental static runtime might work on Alpine (if fusermount is installed and the fuse kernel module is loaded) but I fail to get DHCP and internet access running when booting into

https://dl-cdn.alpinelinux.org/alpine/v3.12/releases/x86_64/alpine-standard-3.12.0-x86_64.iso

Does anyone know how to test this?

@s-zeid
Copy link
Contributor

s-zeid commented May 7, 2022

On v3.12, setup-interfaces, press enter a few times for defaults, and service networking restart. On newer Alpine versions v3.15 or later, you can just do setup-interfaces -r.

Also, you may want to setup repositories: setup-apkrepos -1, and (if desired) uncomment community in /etc/apk/repositories then apk update.

@s-zeid
Copy link
Contributor

s-zeid commented May 7, 2022

And appimagetool from your first link does run on my Alpine edge system.

@probonopd
Copy link
Member Author

Discussion continues in

@xplshn
Copy link

xplshn commented Apr 15, 2024

If I understand it right, it Looks like https://github.com/eth-cscs/spack-batteries-included is providing a solution for this. Should we backport these changes into the AppImage runtime?

Differences and improvements over AppImage runtime
spack.x uses zstd for faster decompression;
spack.x itself is an entirely static binary;
spack.x does not need to dlopen libfuse.so

Reference: #1120 (comment) cc @haampie

It uses the same method as flatpaks, runtimes contain a small Linux system... In this case, it contains gtar, python3, binutils and other misc programs and libraries that were compiled against glibc but then modified to link against a local glibc (a glibc installed inside of the runtime)

@xplshn
Copy link

xplshn commented Apr 15, 2024

If only the guides for packaging AppImages told developers to ALWAYS TRY to statically link using musl-gcc or an Alpine container or install... AppImages would be smaller and would be actually portable.

@probonopd
Copy link
Member Author

Everything has upsides and downsides. But yes, I like the approach you suggest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.