Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to compile 0.28.0 for Windows and musl platforms with vendored openssl #446

Open
jszwedko opened this issue Feb 12, 2022 · 7 comments

Comments

@jszwedko
Copy link

jszwedko commented Feb 12, 2022

Hey!

Apologies, I tried to nail this down myself, but have been banging my head against it all day and thought maybe you would have some insight.

We have a project (https://github.com/vectordotdev/vector) using rust-rdkafka that we are trying to upgrade from 0.27.0 to 0.28.0 to include de421ba but our cross compilation for musl-based platfroms as well as native builds for Windows have errors linking a vendored openssl that they didn't for 0.27.0.

For the cross compilation to musl this looks like:

 = note: /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /target/aarch64-unknown-linux-musl/debug/deps/librdkafka_sys-315d339ecd8675f4.rlib(rdkafka_ssl.c.o): in function `rd_kafka_ssl_set_certs':
          /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1348: undefined reference to `ENGINE_load_ssl_client_cert'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /target/aarch64-unknown-linux-musl/debug/deps/librdkafka_sys-315d339ecd8675f4.rlib(rdkafka_ssl.c.o): in function `rd_kafka_ssl_ctx_term':
          /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1421: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /target/aarch64-unknown-linux-musl/debug/deps/librdkafka_sys-315d339ecd8675f4.rlib(rdkafka_ssl.c.o): in function `rd_kafka_ssl_ctx_init_engine':
          /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1442: undefined reference to `ENGINE_by_id'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1444: undefined reference to `ENGINE_by_id'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1453: undefined reference to `ENGINE_ctrl_cmd_string'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1456: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1463: undefined reference to `ENGINE_ctrl_cmd_string'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1465: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1472: undefined reference to `ENGINE_ctrl_cmd_string'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1473: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1480: undefined reference to `ENGINE_init'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1481: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /target/aarch64-unknown-linux-musl/debug/deps/librdkafka_sys-315d339ecd8675f4.rlib(rdkafka_ssl.c.o): in function `rd_kafka_ssl_ctx_init':
          /cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1627: undefined reference to `ENGINE_free'
          collect2: error: ld returned 1 exit status
          
  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

For Windows this looks like: https://gist.github.com/jszwedko/5e959ee2cc4c1508238ad8a146d20192

They seem plausibly related.

The features we use are:

rdkafka v0.28.0 cmake_build,libz,ssl,tokio,zstd

For openssl:

openssl v0.10.38 vendored

You can see the CI build failures here: https://github.com/vectordotdev/vector/runs/5029404180?check_suite_focus=true

I put together an simplified example repo that seems to reproduce it for me using just rdkafka with the ssl/vendored feature here: https://github.com/jszwedko/rdkafka-vendored-ssl

In there I run:

cargo build  --target x86_64-unknown-linux-musl

Any insight would be appreciated 🙂

@benesch
Copy link
Collaborator

benesch commented Feb 22, 2022

Hm, nothing changed about the build system between v0.27.0 and v0.28.0 that I can see, so really not sure what's going wrong. I wonder if it's librdkafka itself that's misbehaving. You could try adjusting vectordotdev/vector#10302 to update rdkafka without updating rdkafka-sys. That would help to isolate the problem, and would let you pick up the bugfix without bumping librdkafka itself.

Otherwise my reply in #447 (comment) applies here too. If you can re-run with cargo build -vv I can take a quick skim, but I'm afraid my time for debugging cross-compiling issues is basically zero.

@jszwedko
Copy link
Author

Thanks for taking a peek @benesch ! I reran a clean build of my dummy project with cargo build -vv. The log is here:

https://gist.github.com/d52bf10d2ea38df0658a79f4bd90c5f7

I'm curious if anything jumps out to you.

@jszwedko
Copy link
Author

I attempted to just upgrade rdkafka to 0.28.0 and keep rdkafka-sys at 4.0.0, but ran into the same failure.

@0x003e
Copy link

0x003e commented Jun 18, 2022

@jszwedko try this: #479 (comment)

@neuronull
Copy link

@jszwedko try this: #479 (comment)

Hi all, I'm investigating this issue with @jszwedko .

@0x003e , I tried that solution and while it did not work in moving from 0.27 to 0.28 , it did help reveal something intriguing.

If I specify the following, I get the same error as in the original comment (undefined reference to ENGINE_*).

rdkafka = { git="https://github.com/fede1024/rust-rdkafka", rev="v0.27.0", default-features = false, features = ["ssl-vendored", "libz", "tokio", "zstd"] }

However, if I override the rdkafka-sys to the version in v0.26.0 , compilation succeeds.

[patch.'https://github.com/fede1024/rust-rdkafka']
rdkafka-sys = { path = '../rust-rdkafka-old/rdkafka-sys' }

(where rust-rdkafka-old is a checkout of v0.26.0)

I believe @0x003e is on to something with this comment:
#479 (comment)

I also saw here: https://github.com/edenhill/librdkafka/releases/tag/v1.7.0
, that one of the features of v1.7.0 of librdkafka, was to include support for OpenSSL Engine... (which is what all of the linker errors are centered on).

Wondering if the openssl-sys version that rdkafka-sys specifies (0.9.48) needs to be upgraded, to a version that includes OpenSSL Engine support. It appears the openssl-sys version in rdkafka-sys has not been modified since 2019.

@neuronull
Copy link

neuronull commented Jun 28, 2022

On further inspection, I found that it is not the SSL version that is the problem.
The problem is that whenever cross-compiling for musl , the openssl-sys is setting the no-engine flag when configuring the openssl for compilation.

This combined with the fact that librdkafka v1.7.0 introduced a dependency on the SSL Engine feature, means that the cross compilation won't work without a modification somewhere.

It seems either openssl-sys needs to allow SSL Engine feature for musl cross-compilation, or rust-rdkafka / librdkafka needs to make SSL Engine dependency configurable.

This is where the openssl-sys is setting the no-engine flag:

https://github.com/alexcrichton/openssl-src-rs/blob/0e9ec2f8b30ee89a8d1a486e100e03718a960665/src/lib.rs#L197-L200

@trinity-1686a
Copy link

it's now possible to force-enable engine on musl thanks to alexcrichton/openssl-src-rs#151

[build-dependencies]
openssl-src = { version = "111.24.0" , features = ["force-engine"] }

On alpine it requires to install linux-headers (otherwise it indeed fails to compile due to missing linux/version.h). Some part of the engine module depends on linux >= 4.1.0 (any version not EOL is fine), checked both at compile time and runtime by openssl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants