CNIOBoringSSL compiler error on armv6 armv7 Linux using clang 3.8

[wlisac]

I'm attempting to build SwiftNIO SSL on armv6 and armv7 Linux, but I'm running into a couple compiler errors with Swift 5.1 and Swift 5.0.1.

The issue appears to be specific to using a clang 3.8 compiler. Ideally we'd be using clang 7 (which is what ships with official Swift 5 releases), but we're currently stuck using clang 3.8 with some ARM toolchains / Linux distributions.

I've discussed this a bit with @Lukasa and it looks like an issue with BoringSSL.

Steps to Reproduce

One way to reproduce the errors is to compile SwiftNIO SSL using Docker Desktop for macOS with armv6 and armv7 base images. Docker Desktop for macOS will use qemu to emulate the architecture.

I maintain Docker images for Swift on ARM at wlisac/swift-on-balena that we can use for the base images.

Dockerfile

ARG FROM_IMAGE

FROM $FROM_IMAGE

RUN git clone https://github.com/apple/swift-nio-ssl.git

WORKDIR swift-nio-ssl

RUN swift package resolve

RUN swift build --jobs 1

armv7, Swift 5.1, Ubuntu Bionic, clang version 6.0.0-1ubuntu2

✅ Builds successfully.

docker build --build-arg FROM_IMAGE=wlisac/armv7hf-ubuntu-swift:5.1-bionic-build .

armv7, Swift 5.0.1, Ubuntu Bionic, clang version 6.0.0-1ubuntu2

✅ Builds successfully.

docker build --build-arg FROM_IMAGE=wlisac/armv7hf-swift-ubuntu:5.0.1-bionic-build .

armv7, Swift 5.1, Ubuntu Xenial, clang version 3.8.0-2ubuntu4

❌ hrss.c build error (see below)

docker build --build-arg FROM_IMAGE=wlisac/armv7hf-ubuntu-swift:5.1-xenial-build .

armv7, Swift 5.0.1, Ubuntu Xenial, clang version 3.8.0-2ubuntu4

❌ hrss.c build error (see below)

docker build --build-arg FROM_IMAGE=wlisac/armv7hf-ubuntu-swift:5.0.1-xenial-build .

armv7, Swift 5.1, Debian Stretch, clang version 3.8.1-24

❌ hrss.c build error (see below)

docker build --build-arg FROM_IMAGE=wlisac/armv7hf-debian-swift:5.1-stretch-build .

armv7, Swift 5.0.1, Debian Stretch, clang version 3.8.1-24

❌ x25519-asm-arm build error (see below)

docker build --build-arg FROM_IMAGE=wlisac/armv7hf-debian-swift:5.0.1-stretch-build .

armv6, Swift 5.1, Debian Stretch, clang version 3.8.1-24+rpi1

❌ x25519-asm-arm build error (see below)

docker build --build-arg FROM_IMAGE=wlisac/rpi-swift:5.1-build .

armv6, Swift 5.0.1, Debian Stretch, clang version 3.8.1-24+rpi1

❌ x25519-asm-arm build error (see below)

docker build --build-arg FROM_IMAGE=wlisac/rpi-swift:5.0.1-build .

Errors

hrss.c build error

/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:253:41: error: cannot convert between vector values of different size ('int16x8_t' (vector of 8 'int16_t' values) and 'int')
  a = (vec_t)vshrq_n_s16(((int16x8_t)a) << 15, 15);
                         ~~~~~~~~~~~~~~ ^  ~~
/usr/lib/llvm-3.8/bin/../lib/clang/3.8.0/include/arm_neon.h:23593:20: note: expanded from macro 'vshrq_n_s16'
  int16x8_t __s0 = __p0; \
                   ^~~~
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:268:33: error: cannot convert between vector values of different size ('vec_t' (aka 'uint16x8_t') and 'int')
    vec_t next_carry_s = a_s[i] >> 15;
                         ~~~~~~ ^  ~~
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:269:12: error: cannot convert between vector values of different size ('vec_t' (aka 'uint16x8_t') and 'int')
    a_s[i] <<= 1;
    ~~~~~~ ^   ~
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:274:33: error: cannot convert between vector values of different size ('vec_t' (aka 'uint16x8_t') and 'int')
    vec_t next_carry_a = a_a[i] >> 15;
                         ~~~~~~ ^  ~~
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:275:12: error: cannot convert between vector values of different size ('vec_t' (aka 'uint16x8_t') and 'int')
    a_a[i] <<= 1;
    ~~~~~~ ^   ~
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:288:33: error: cannot convert between vector values of different size ('vec_t' (aka 'uint16x8_t') and 'int')
    vec_t next_carry_s = a_s[i] << 15;
                         ~~~~~~ ^  ~~
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:289:12: error: cannot convert between vector values of different size ('vec_t' (aka 'uint16x8_t') and 'int')
    a_s[i] >>= 1;
    ~~~~~~ ^   ~
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:294:33: error: cannot convert between vector values of different size ('vec_t' (aka 'uint16x8_t') and 'int')
    vec_t next_carry_a = a_a[i] << 15;
                         ~~~~~~ ^  ~~
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/hrss/hrss.c:295:12: error: cannot convert between vector values of different size ('vec_t' (aka 'uint16x8_t') and 'int')
    a_a[i] >>= 1;
    ~~~~~~ ^   ~
9 errors generated.

x25519-asm-arm build error

/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/curve25519/asm/x25519-asm-arm.S:56:1: error: out of range pc-relative fixup value
ldr r4,=0
^
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/curve25519/asm/x25519-asm-arm.S:57:1: error: out of range pc-relative fixup value
ldr r5,=254
^
/swift-nio-ssl/Sources/CNIOBoringSSL/crypto/curve25519/asm/x25519-asm-arm.S:75:1: error: out of range pc-relative fixup value
ldr r7,=960
^

[weissi]

@wlisac oh, this is surprising. Swift stuff will actually only work correctly if the clang used is the clang that ships with Swift. Since Swift 5.0, clang should actually be shipped with Swift and it should always be the same version.

Your errors look like in the ARM toolchain the wrong clang is selected. Next to swiftc, do you have a clang binary? If yes, what's the output of /path/to/swift/usr/bin/clang -version?

[weissi]

@wlisac btw, it's best to never install the host distribution's clang and always use the one that comes with Swift.

[weissi]

@wlisac you might be seeing https://bugs.swift.org/browse/SR-10633

[uraimo]

do you have a clang binary?

The problem is that no, on Linux/ARMv* we don't, same thing for libicu/libcxx, right now the stuff from the host is being used.
Until now we were using a custom set of build-script options to build the toolchain, and those didn't include options like --libicu/--install-libicu that are now part of the default preset on Linux.
When I tried to use the buildbot_linux preset on ARM, I noticed that clang still wasn't copied over to the toolchain install path, and that SPM compilation failed due to missing libicu.

I did a quick attempt at identifying the reason but couldn't understand why both are missing and not copied when needed (e.g. at least when SPM is being built, in the case of libicu).

So, unreadable description above aside, there is something broken on linux/arm where we copy clang and a few other components like libicu and libcxx to build the toolchain.

Edit: rephrased,sorry.

[wlisac]

@weissi yeah, hopefully we can move away from using the host system's clang in the near future, but as @uraimo mentioned this has been a non-trivial task so far on armv6 / armv7 Linux.

In the meantime, it seems like BoringSSL should still be able to compile using clang 3.8, but I totally understand if fixing this is out of scope for SwiftNIO SSL.

Unfortunately, for the time being, we're a bit stuck being unable to use SwiftNIO SSL on armv6 / armv7 devices where the host system uses clang 3.8.

As an aside, the aarch64 (armv8 64 bit) toolchain by futurejones/swift-arm64 does include Swift's clang and SwiftNIO SSL works as expected since #141 was merged.

[weissi]

@wlisac hmm, that’s really unfortunate that Swift on ARM isn’t actually a full Swift distribution. Clang 3.8 is over 3 years old and I’m not sure where we should draw the line on what’s supported and what not. The official Swift distributions all ship a clang that matches the ClangImporter that swiftc uses internally. Mixing a very recent ClangImporter (contained in swiftc) with a 3 years old clang seems very fragile and works more by accident than by design, all the sanitizers won’t work either because they require all the compilers to be updated in lockstep.

I understand that this probably isn’t the answer you wanted to hear. Does vanilla BoringSSL work with these old clangs?

CC @Lukasa WDYT?

[weissi]

@uraimo ok, thanks for the explanation. Is there already a Swift bug for this? I feel we need to really fix Swift here rather than adding workarounds to NIO SSL.

[Lukasa]

Vanilla BoringSSL does not work on Clang 3.8 on armv6/armv7. This is not to say that it couldn't, just that it doesn't. My gut feel is that the BoringSSL team are unlikely to do this work themselves, but they might accept a patch that does it. That's not necessarily going to be easier than trying to bring along a better clang in Swift on ARM, in particular dealing with the ASM issue which seems fairly profound.

[wlisac]

@weissi @Lukasa – yeah, makes sense and sounds fair to me.

For clarity, the Swift on ARM toolchains for aarch64 are full Swift distributions that include clang.

I think for now I can work around this on armv6 / armv7 by using Ubuntu Bionic with clang 6 or by using Debain Buster with clang 7.

Thanks for taking the time to look into this and provide input 🙂 should we close this issue?

[weissi]

@wlisac up to you really. It seems that the fixes need to come from the BoringSSL or Swift on armv7 teams and probably not us. But I'm also happy keeping this issue around because the title is not wrong, it just isn't in scope for us to fix.