Consider updating all current container images

[jamesmunns]

For consistency, we should update all of the currently existing container images to the current LTS environment when possible (I believe that is ubuntu 16 04), and try to update any other component as well, such as openssl.

• aarch64-linux-android
• aarch64-unknown-linux-gnu
• arm-linux-androideabi
• arm-unknown-linux-gnueabi
• arm-unknown-linux-musleabi
• armv7-linux-androideabi
• armv7-unknown-linux-gnueabihf
• armv7-unknown-linux-musleabihf
• asmjs-unknown-emscripten
• i586-unknown-linux-gnu
• i686-linux-android
• i686-pc-windows-gnu
• i686-unknown-freebsd
• i686-unknown-linux-gnu
• i686-unknown-linux-musl
• mips64el-unknown-linux-gnuabi64
• mips64-unknown-linux-gnuabi64
• mipsel-unknown-linux-gnu
• mips-unknown-linux-gnu
• powerpc64le-unknown-linux-gnu
• powerpc64-unknown-linux-gnu
• powerpc-unknown-linux-gnu
• s390x-unknown-linux-gnu
• sparc64-unknown-linux-gnu
• sparcv9-sun-solaris
• thumbv6m-none-eabi
• thumbv7em-none-eabi
• thumbv7em-none-eabihf
• thumbv7m-none-eabi
• wasm32-unknown-emscripten
• x86_64-linux-android
• x86_64-pc-windows-gnu
• x86_64-sun-solaris
• x86_64-unknown-dragonfly
• x86_64-unknown-freebsd
• x86_64-unknown-linux-gnu
• x86_64-unknown-linux-musl
• x86_64-unknown-netbsd

[jamesmunns]

Actually, we may not want to do this, because of #85

[japaric]

@jamesmunns this is clearly a problem. rust-lang/rust is using ubuntu 16.04 but compiles toolchains from scratch that target glibc 2.14. If we move our images to 16.04 and use the packaged (cross) toolchain then our glibc will be 2.23 and all the binaries produced with cross will have that glibc requirement (2.23) instead of being compatible with 2.14+.

We should do what rust-lang/rust is doing but that will pretty much make all CI build time out because crosstool-ng takes like 30+ mins to run on a fast machine. It will take way longer on Travis machines.

An option here may be to produce binaries for the toolchain locally (e.g. on my laptop), push the binaries somewhere and have our images simply pull that. That should save us the toolchain build time. I expect that toolchain versions will change rarely so this semi-automatic process of producing binaries shouldn't be too onerous. We will need to rebuild the toolchains everytime we bump the ubuntu version of the images though.

[japaric]

produce binaries for the toolchain locally

but still within a docker container whose version matches the one we are using in our images

[jamesmunns]

@japaric I can also add a .gitlab-ci.yml and render the images on my CI server.

If you can set up the crosstool-ng part (I have tried briefly and failed in the past), I would be happy to help with either spreading that change to all containers, or at least providing build support.

[japaric]

Also, doing what rust-lang/rust does pretty much kills the chance of ever having #149 working, I think. The reason is that anything provided by the distribution has been compiled with the newer toolchain and linking that into the binary could potentially introduce newer GLIBC symbols and kill the portability gained by compiling our own toolchain.

[jamesmunns]

@japaric is there any way to make this "opt-in"? e.g. provide max compat by default, but have an escape hatch for users who don't care about it, and just want their stuff to work for their specific use case?

[japaric]

If you can set up the crosstool-ng part

It will be pretty much the same as the what rust-lang/rust does. Relevant scripts are:

• crosstool-ng.sh
• build-toolchains.sh
• sample .config file

[japaric]

is there any way to make this "opt-in"?

No idea. Mixing stuff compiled using different toolchains sounds like a recipe for disaster. I'd rather not recommend doing that in any official manner. Let people try it if they want but they'll have to deal with the problems they encounter themselves.

[gnzlbg]

We should do what rust-lang/rust is doing but that will pretty much make all CI build time out because crosstool-ng takes like 30+ mins to run on a fast machine

could a toolchain from https://toolchains.free-electrons.com be used?

[japaric]

@gnzlbg AFAICT, the glibc shipped with those toolchain is even newer (glibc 2.24) than ubuntu 16.04's (2.23). rust-lang/rust is using 2.14. (these versions are for ARMv7).

[bgermann]

Just a note on the *-pc-windows-gnu images: They should be updated independently from the Linux targeting images to 18.04 LTS when it is available, because of the wine version. The 17.04 version is not available on the ubuntu download archives anymore.

Maybe the other non-Linux targeting images should use the newest LTS as well as the glibc version does not matter for them.

Edit: Available as #199

[bgermann]

I would suggest using Debian Jessie (glibc 2.19) for the -linux-gnu targets. The only popular, supported distribution with 2.14 <= glibc < 2.19 is RedHat/CentOS 7. As Debian also uses apt/dpkg, chances are that there are only a few package names to change in the build system.

[jamesmunns]

We may need to revisit this, I am not sure all of the existing images based on old ubuntu distros still build at the moment, so the cached docker images will still work, but someone trying to modify the existing images will get an unexpected build failure.

[briansmith]

I think the considerations vary per architecture. In the case of ARM architectures, glibc 2.15 vs glibc 2.16 is significant because glibc 2.16 introduced getauxval() and getauxval() is needed for runtime CPU feature detection (NEON, crypto extensions, etc.). Thus I think it would be good to upgrade ARM/Aarch64 targets to glibc 2.16 or later, regardless of what is done for x86/x86_64.

[gnzlbg]

getauxval() is needed for runtime CPU feature detection (NEON, crypto extensions, etc.).

getauxval does indeed make runtime CPU feature detection simpler, but Rust standard library does not require it. That is, unless stdsimd has a bug. Please open an issue there if that's the case.

Having said this, rust-lang/libc has shifted its libc support policy from "the oldest libc possible" to "only supporting the newest libc". This is because some/most libc/system library implementations constantly break ABI compatibility with older versions, most users are using the latest version, and we currently don't have any way to support multiple incompatible versions (e.g. the platform version is not part of the target triple).

[briansmith]

getauxval does indeed make runtime CPU feature detection simpler, but Rust standard library does not require it. That is, unless stdsimd has a bug. Please open an issue there if that's the case.

FWIW, my project avoids using stdsimd because of the fallback logic that stdsimd has when it can't use getauxval(). My project calls getauxval() directly.

Having said this, rust-lang/libc has shifted its libc support policy from "the oldest libc possible" to "only supporting the newest libc".

Thanks. Do you have a link to some documentation or discussion about that change, so I can learn more about it?

[gnzlbg]

FWIW, my project avoids using stdsimd because of the fallback logic that stdsimd has when it can't use getauxval(). My project calls getauxval() directly.

Please open an issue. We have refactored the run-time feature detection support of stdsimd into a normal Rust crate, and we plan to make it more configurable so that those applications for which the defaults are not well suited can avoid re-inventing the wheel. Using getauxval or failing should be easy to support. If you are using a new Linux kernel >= 4.11, you can skip getauxval and call msr directly, so there are a couple of alternatives here that you might want to explore.

Do you have a link to some documentation or discussion about that change, so I can learn more about it?

rust-lang/libc#1142
rust-lang/libc#570

but basically the policy shift has been happening in PRs since currently there isn't an alternative. If most users use a newer system library version with a backwards incompatible change, we can currently either fix libc by doing a breaking change or leave it as is. We typically roll these breaking changes very slowly, particularly on weird platforms without much testing. But libc doesn't currently have a way to distinguish two OS versions, so it has to pick one, which often means things are broken in the other.

[TotalKrill]

What is the current status of a container update?

[therealprof]

@rust-embedded/tools As per discussion on the 21/03 meeting we need to make a decision how we would like to continue with cross. The current state is very much broken while at the same time a lot of crates (for embedded most prominently svd2rust) depend on cross for CI. Any opinions?

[gnzlbg]

Maybe we could maintain two sets of images, one using the most recent versions of everything, and one using the oldest version of everything that "works". If the images stop working (e.g. Ubuntu goes EOL or similar) and fixing them requires bumping them, then we do so. That would essentially duplicate the number of build jobs / time per build job on cross' CI, and would significantly increase the maintainer burden, but I don't think there is a way to make cross usable for both types of users: those who want to use newer glibc / kernel features, and those who want super-backward-compatible binaries.

[ryankurte]

i’ve been thinking for a while it’d be good to refactor the images so we have a common base image with all the standard utilities and the platform specific images all extend this (which i think will help build times).

docker supports piping in configuration files, so one way would be to create a build matrix and programmatically generate base-target images, then have cross default to whatever the latest-X while allowing people to specify alternatives.

i’d also propose that we run image building (and maybe even CI) for cross outside of travis-ci because it’s so demanding compared to almost everything else.

[jamesmunns]

I'm also working on an experimental alternative to Cross called Naught. The idea would be to split the maintenance of the tool from the images, largely leaving the images to community maintenance status, and no longer publishing pre-built versions of the containers.

I have some of my design goals up here, I'm planning on working on this over the next week or so.

[ryankurte]

@jamesmunns maybe a discussion issue over there would be good, but, why not skip the repo listing and just take arbitrary image names (which presumably you could drop into .cargo/config), and let docker deal with building or pulling as required?

[DoumanAsh]

Considering that dependencies such ring may require getauxval on some platforms, wouldn't it be best to update at least these targets? For example on armv7-unknown-linux-gnueabihf you will be unable to compile ring as it requires spin there, which makes rustls unusable

P.s. For these who will find need I published fresh build for armv7-unknown-linux-musleabihf but without OpenSSL
Also surprisingly bigger than older one in japaric's but at least it can compile rustls which is what I use.

[DoumanAsh]

@jamesmunns Could you maybe share your thoughts on whether we can expect some decision or should we start making own images/options instead of relying on cross?

[gnzlbg]

FWIW most rust-lang/ crates like libc, stdarch, etc. use Ubuntu 19.04 to be able to test newer APIs.

[gnzlbg]

We should do what rust-lang/rust is doing but that will pretty much make all CI build time out because crosstool-ng takes like 30+ mins to run on a fast machine. It will take way longer on Travis machines.

Azure Pipelines provides 10 concurrent jobs and up to 6 hour jobs I think.

[gdesmott]

Maybe we could maintain two sets of images, one using the most recent versions of everything, and one using the oldest version of everything that "works".

I'm also facing issue with images being too old (need newer version C libraries) making cross unusable for my needs. Having two set of images would solve this.

[gdesmott]

So ideally we should provide images based on:

• Ubuntu 16.04 for those carying most about backward compat, that's the current images
• The latest Ubuntu LTS (18.04 atm)
• Maybe also the current Debian Stable, those images seem useful for users having to install external libs using multi-archs, see Unable to add packages for other architectures #149

Does that make sense? Would you consider expanding the set of provided images to include those?

[Alexhuszagh]

All images have been updated to 16.04, and the update to 20.04 is tracked in #517.