feat(arm64): enable single-binary builds

[mudler]

Description

This PR tries to add a job to build on aarch64 by cross-compiling.

In the current state, it only builds the llama.cpp backend (non-cuda).

I've tried compiling other backends and cuda but didn't had success yet.I'm leaving the CI configured for installing CUDA on the host, however that needs another iteration to figure out what's still wrong with it.

As a note, I've also added in the startup logics to allow to ship libraries as part of the backend assets and load those during start, under the lib directory, for instance, sibiling of the current directory we use to extract the grpc backends.

In my tests with the binary it needed other libraries as well (which aren't part of this PR) in order to start llama.cpp correctly. This is not addressed in this PR, but I will likely follow up to leverage the lib loading to ship libraries along with the binary as well.

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	ad3fe0a
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/66656de39a6804000885103c

[sozercan]

@mudler looks like this needs cuda packages to be installed

another option i was thinking is that merge release into container builds (using qemu for arm64 builds), and have an optional container target that just copies the binaries into scratch, then output them with --output. however, i think the container builds take really long so this will take quite some time unless this can be cached.

[mudler]

@mudler looks like this needs cuda packages to be installed

another option i was thinking is that merge release into container builds (using qemu for arm64 builds), and have an optional container target that just copies the binaries into scratch, then output them with --output. however, i think the container builds take really long so this will take quite some time unless this can be cached.

yep that was actually what I had in mind as well.. but thought was worth giving a shot in any case as with cross-arch should supposedly be faster 🤞

however, as we build already an arm64 container maybe at the end that's what we have to end up doing to squeeze CI cycles

[sozercan]

nvidia isn't publishing cuda libraries for arm64 for 12.3 for some reason, this need to be 12-4 or 12-5. I am guessing runtime libraries needs to match that too

[mudler]

we can start to think also to bundle that as part of the binary and then load libraries from there in runtime, that way we skip this annoying bit for users who are going to use static binaries

[mudler]

some of that logic implemented in 4412036 - it should allow us to carry over libs. I've just tried the arm64 builds and "almost" works in my env. llama.cpp doesn't start as there is a libstdc++ incompatibility with the host

[mudler]

something else is off - probably the cross arch compiler isn't even picked up properly

[mudler]

long time I didn't do this - bare with me a bit. @sozercan any input is helpful as CI cycles takes a while

[mudler]

ok let's start step by step and have first an arm64 binary cross-compiled, will see what to do next with cuda - seems it's not being picked up correctly

[mudler]

aarch64 binary is compiled successfully for the aarch64, however in my test failed:

/tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback 
/tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback)
/tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback)
/tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback)
/tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback)
/tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback)
/tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback: /lib/aarch64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by /tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback)

file /tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback 
/tmp/localai/backend_data/backend-assets/grpc/llama-cpp-fallback: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=dcff3a71166550acb940d410870543505d17aa94, for GNU/Linux 3.7.0, with debug_info, not stripped

so we are almost there. I'm trying to get a baseline now and merge what we have, we can iterate from there to add more backends to it

[mudler]

I want to try next to embed the libs (like nvidia too) into the final binary. will take a while to test, but if that works then we don't have to worry much more also in that specific case. If that won't work I could take it a little bit further in trying by pivoting root (from the user).

Anyhow, leaving libs handling for a follow-up, polishing now.