Profile disabled on alpine 3.13 with a runtime error loading the native extension

[rwc9u]

Current behaviour

See the following in our log after upgrading ddtrace from 0.49 to 1.3

W, [2022-08-18T04:28:20.869948 #1]  WARN -- ddtrace: [ddtrace] Profiling was requested but is not supported, profiling disabled: There was an error loading the profiling native extension due to 'RuntimeError Failure to load ddtrace_profiling_native_extension.2.5.9_x86_64-linux-musl Due to Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/vendor/gems/ruby/2.5.0/gems/ddtrace-1.3.0/lib/datadog/profiling/../../../../libdatadog-0.7.0.1.0-x86_64-linux/vendor/libdatadog-0.7.0/x86_64-linux/libdatadog-x86_64-unknown-linux-gnu/lib/libddprof_ffi.so)' at '/app/vendor/gems/ruby/2.5.0/gems/ddtrace-1.3.0/lib/datadog/profiling/load_native_extension.rb:22:in `<top (required)>''

When I do some additional debugging on our pod I see the following

bash-5.1$ ldd `bundle exec gem which ddtrace_profiling_native_extension.2.5.9_x86_64-linux-musl.so`
Your RubyGems version (3.0.3) has a bug that prevents `required_ruby_version` from working for Bundler. Any scripts that use `gem install bundler` will break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by running `gem update --system 3.2.3`
	/lib/ld-musl-x86_64.so.1 (0x7f22219c4000)
	libruby.so.2.5 => /usr/local/lib/libruby.so.2.5 (0x7f222169b000)
	libddprof_ffi.so => /app/vendor/gems/ruby/2.5.0/gems/ddtrace-1.3.0/lib/../../libdatadog-0.7.0.1.0-x86_64-linux/vendor/libdatadog-0.7.0/x86_64-linux/libdatadog-x86_64-unknown-linux-gnu/lib/libddprof_ffi.so (0x7f22213ea000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f22219c4000)
	libgmp.so.10 => /usr/lib/libgmp.so.10 (0x7f2221384000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f222136a000)
	librt.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f22219c4000)
	libpthread.so.0 => /lib/ld-musl-x86_64.so.1 (0x7f22219c4000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7f22219c4000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/vendor/gems/ruby/2.5.0/gems/ddtrace-1.3.0/lib/../../libdatadog-0.7.0.1.0-x86_64-linux/vendor/libdatadog-0.7.0/x86_64-linux/libdatadog-x86_64-unknown-linux-gnu/lib/libddprof_ffi.so)
Error relocating /app/vendor/gems/ruby/2.5.0/gems/ddtrace-1.3.0/lib/../../libdatadog-0.7.0.1.0-x86_64-linux/vendor/libdatadog-0.7.0/x86_64-linux/libdatadog-x86_64-unknown-linux-gnu/lib/libddprof_ffi.so: __res_init: symbol not found
Error relocating /app/vendor/gems/ruby/2.5.0/gems/ddtrace-1.3.0/lib/../../libdatadog-0.7.0.1.0-x86_64-linux/vendor/libdatadog-0.7.0/x86_64-linux/libdatadog-x86_64-unknown-linux-gnu/lib/libddprof_ffi.so: gnu_get_libc_version: symbol not found

I'm thinking there are some additional packages I need to run under alpine, but not sure which ones?

Environment

• ddtrace version:
• 1.3
• Ruby version:
• 2.5.9
• Operating system:
• alpine 3.13
• Relevant library versions:

[ivoanjo]

Hey @rwc9u, thanks for the report and sorry for the issue!

From your log message, the failure is caused by the wrong version of libddprof_ffi.so being used by dd-trace-rb.

In particular, inside the libdatadog gem we ship two copies of that library -- one for "regular" glibc linux, and another for alpine/musl:

• libdatadog-0.7.0/x86_64-linux-musl/libdatadog-x86_64-alpine-linux-musl/lib/libddprof_ffi.so
• libdatadog-0.7.0/x86_64-linux/libdatadog-x86_64-unknown-linux-gnu/lib/libddprof_ffi.so

But in your case, the glibc variant is getting picked, which is incorrect, and leads to the issue.

I'm still investigating what may cause this, I'll update this ticket once I know more :)

[ivoanjo]

Found it! It seems to be a bug in rubygems 😅

I was able to reproduce the issue using the ruby:2.5.9-alpine3.13 docker image. We ask Rubygems which variant of the library to use, and here's what I see on that docker image:

/ # ruby -e "pp Gem::Platform.local.to_s"
"x86_64-linux"
/ # gem --version
3.0.3
/ # gem update --system
Updating rubygems-update
Fetching rubygems-update-3.3.20.gem
Successfully installed rubygems-update-3.3.20
Installing RubyGems 3.3.20
  Successfully built RubyGem
  Name: bundler
  Version: 2.3.20
  File: bundler-2.3.20.gem
Bundler 2.3.20 installed
RubyGems 3.3.20 installed
Regenerating binstubs
Regenerating plugins

# ... a bunch more output

RubyGems system software updated
/ # gem --version
3.3.20
/ # ruby -e "pp Gem::Platform.local.to_s"
"x86_64-linux-musl"

With the old version of rubygems shipped, Gem::Platform.local.to_s returns x86_64-linux which is incorrect; after upgrading rubygems, it correctly returns x86_64-linux-musl.

Let me know if upgrading rubygems fixes the issue for you! I'll still try to investigate if we can detect when rubygems is buggy and still pick the right variant.

EDIT: You'll need to install/reinstall ddtrace after upgrading rubygems.

[rwc9u]

Thx so much for the quick response.

[ivoanjo]

Thx so much for the quick response.

Happy to help!

I actually found a way to work around this issue directly in the libdatadog gem, and released the fix as libdatadog 0.7.0.1.1.

Thus, either one of:

• Upgrading to the latest rubygems with gem update --system
• Upgrading to the latest libdatadog

and then installing/reinstalling ddtrace should get you over the issue.

Feel free to reach out if there's anything else we can help with :)