Ruby google-protobuf linux precompiled gem is not compatible with linux-musl

[ntkme]

What version of protobuf and what language are you using?
Version: 4.26.1 or higher on aarch64 / 4.29.3 on x86_64
Language: Ruby

What operating system (Linux, Windows, ...) and version?
Alpine Linux 3.19.1

What runtime / compiler are you using (e.g., python version or gcc version)
ruby 3.3.1 (2024-04-23 revision c56cd86388) [aarch64-linux-musl]
ruby 3.3.6 (2024-11-05 revision 75015d4c1f) [x86_64-linux-musl]
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux-musl]

What did you do?
Steps to reproduce the behavior:

/ # gem install google-protobuf
Successfully installed google-protobuf-4.26.1-aarch64-linux
1 gem installed
/ # ruby -e "require 'google/protobuf'"
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- google/protobuf_c (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:15:in `rescue in <top (required)>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:12:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:57:in `<module:Protobuf>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:15:in `<module:Google>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:14:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from -e:1:in `<main>'
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': Error loading shared library ld-linux-aarch64.so.1: No such file or directory (needed by /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/3.3/protobuf_c.so) - /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/3.3/protobuf_c.so (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:13:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:57:in `<module:Protobuf>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:15:in `<module:Google>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:14:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from -e:1:in `<main>'
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- google/protobuf (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from -e:1:in `<main>'

What did you expect to see
google-protobuf should work on linux-musl just like how it works on linux-gnu

What did you see instead?
google-protobuf prebuilt library is strictly linked to glibc and it's not compatible with musl-libc

Anything else we should know about your project / environment
Please see rake-compiler's documentation regarding recent changes in rubygems in regarding to -linux, -linux-gnu, and -linux-musl platforms: https://github.com/rake-compiler/rake-compiler-dock?tab=readme-ov-file#linux-gnu-and-musl-important-details

[ntkme]

As a workaround, uninstalling the prebuilt gem and reinstall platform ruby to build from source fixes the issue:

/ # gem uninstall google-protobuf
Successfully uninstalled google-protobuf-4.26.1-aarch64-linux
/ # apk add alpine-sdk
...
/ # gem install --platform ruby google-protobuf
Fetching google-protobuf-4.26.1.gem
Building native extensions. This could take a while...
Successfully installed google-protobuf-4.26.1
1 gem installed
/ # ruby -e "require 'google/protobuf'"

For bundler, the following in Gemfile would work:

gem 'google-protobuf', force_ruby_platform: true if RUBY_PLATFORM.include?('linux-musl')

[ntkme]

There are a few options here:

1. Do not ship aarch64-linux gem, and let both aarch64-linux-gnu and aarch64-linux-musl platforms fallback to platform ruby and compile from source.
2. Ship the current flavor, but label it as aarch64-linux-gnu, so that it does not get installed incorrectly on aarch64-linux-musl.
3. Upgrade rake-compiler, and ship separate aarch64-linux-gnu and aarch64-linux-musl gems.

Again, please refer to rake-compiler's documentation on details: https://github.com/rake-compiler/rake-compiler-dock?tab=readme-ov-file#linux-gnu-and-musl-important-details

[JasonLunn]

@dazuma - what's the expected platform support for linux-musl?

[ntkme]

As said early, even if we don't "officially" support linux-musl, we can avoid it from breaking out of box. - Just ship the gem with label -linux-gnu instead of -linux using latest rake-compiler.

[henrahmagix]

I'd really appreciate if @ntkme's comment could be actioned pls ☺️

ship the gem with label -linux-gnu instead of -linux using latest rake-compiler

[ntkme]

This issue used to be aarch64-linux-musl only.

However, with 4.29.3 release, that x86_64-linux prebuilt now triggers Segmentation fault on x86_64-linux-musl:

• google-protobuf 4.29.2 (x86_64-linux) worked on musl: https://github.com/sass-contrib/sass-embedded-host-ruby/actions/runs/12606460485/job/35136617853
• google-protobuf 4.29.3 (x86_64-linux) crashed on musl with Segmentation fault: https://github.com/sass-contrib/sass-embedded-host-ruby/actions/runs/12712732880/job/35438995154

This issue can be worked around by force compiling from source by adding the following in Gemfile:

gem 'google-protobuf', force_ruby_platform: true if RUBY_PLATFORM.include?('linux-musl')

@JasonLunn I highly suggest that we take action now to update rake-compiler and split linux-gnu and linux-musl gems as soon as possible.

[ntkme]

The statistics from my own gem that has 10M+ total download shows that roughly 10% of x86_64-linux are from linux-musl, which is definitely not a small number. E.g. As of writing, the download counts for the latest version show:

• https://rubygems.org/gems/sass-embedded/versions/1.83.1-x86_64-linux-gnu - 24,734
• https://rubygems.org/gems/sass-embedded/versions/1.83.1-x86_64-linux-musl - 2,983

The statistics for google-protobuf might be different, but the total download for google-protobuf is way higher that even only 5% would affect tons of users. I know lots of users in ruby community uses docker.io/library/ruby:alpine image, some even use alpine for production, thus it's critical to not break linux-musl out of box even if we decide to not ship linux-musl prebuilt - an error from bundler asking user to install a compiler is way better than seeing Segmentation fault.

[ntkme]

I was probably the earliest adaptor of splitting linux-gnu and linux-musl prebuilt, as I introduced it way before rake-compiler added support, using my own custom cross-platform build scripts. I have been through lots of trouble with rubygems/bundler's bug around this area in terms of platform detection and even helped fixing a few bugs in rubygems/bundler.

From my own experience with it, I can say that as of now it's reliable with ruby >=3.1 and the default rubygems/bundler or higher version from them. In fact, nokogiri, one of the most downloaded gem of all time, introduced split -linux-gnu and -linux-musl with 1.18.0 release this month: https://rubygems.org/gems/nokogiri/versions
sparklemotion/nokogiri@4c15c5c

Now it's time to adopt -linux-gnu and -linux-musl gems.

[dazuma]

I share the desire to split the builds into -linux-gnu and -linux-musl as soon as possible. However, we cannot until April 2025 because our support policy doesn't drop Ruby 3.0 until then. Once we're able to drop support for Ruby 3.0, we'll definitely split the builds with the next protobuf release.

[ntkme]

@dazuma In fact shipping separate -gnu and -musl does not require dropping Ruby 3.0.

The way it works is that you can set different ruby version and rubygems version requirements on the platform ruby gem and native prebuilt gems.

For example:

• https://rubygems.org/gems/sass-embedded/versions/1.83.4 - platform ruby requires ruby 3.1

• https://rubygems.org/gems/sass-embedded/versions/1.83.4-aarch64-linux-gnu - native gem requires ruby 3.2

• https://rubygems.org/gems/nokogiri/versions/1.18.1 -platform ruby gem requires ruby 3.1

• https://rubygems.org/gems/nokogiri/versions/1.18.1-x86_64-linux-gnu - native gem requires ruby 3.1 AND rubygems 3.3.22

This allow older version of ruby and rubygems/bundler to install the gem by compiling from source. For google-protobuf, we can require 3.0 on the platform ruby gem, and require ruby 3.0 AND rubygems 3.3.22.

You can learn more here about in this issue: rake-compiler/rake-compiler-dock#117 and why ruby 3.2 instead of rubygems 3.3.22 on native gem might be a better idea in this comment rake-compiler/rake-compiler-dock#117 (comment)

[dazuma]

@ntkme It sounds like the above would force ALL users of Ruby 3.0 on Linux to install the gem from source. That's a regression we're trying to avoid.

(The news about Debian-sid and Ruby 3.1 is also disappointing. But I don't think we can wait another year for this.)

[ntkme]

Yes, I agree not able to use prebuilt is a “regression”. However, I’d argue it’s way better than segmentation fault on musl-libc, especially that 4.29.3 literally just regressed with segmentation fault on x86_64-linux-musl. It used to have segmentation fault only on aarch64-linux-musl, which has a very tiny user base, but now impacting many more users (about 10x more based on download stats on sass-embedded and nokogiri).

The 4.29.3 segmentation fault regression on all ruby versions on x86_64-linux-musl is much worse than asking ruby 3.0 on any linux to compile from source. I think it’s worth trading the compiling from source “regression” to fix the segmentation fault, especially given that the compiling from source “regression” will likely break much smaller number of installation on ruby 3.0 on Linux without a working compiler. In addition, the installation time error will be more trivial for users to deal with than a runtime segmentation fault.

After all, I don’t have statistics to proof which one has a border impact on the user base, but here are the two things we are trading:

• ruby 3.0-3.4 *-linux-musl crashing with segmentation fault
• ruby 3.0 *-linux-* without a working compiler failing to install, or with a working compiler to have slightly slower installation

I would recommend that we don’t wait.