document how bundler and rubygems behave wrt `linux`, `linux-gnu`, and `linux-musl` releases #117

flavorjones · 2024-03-03T21:55:04Z

We know that rubygems 3.33.22 / bundler 2.3.21 is required for basic support of linux-gnu and linux-musl platform names; but buggy bundler behavior prior to 2.5.6 may cause challenges for some users.

This issue is to record some research into how the default versions behave with different native platform gems, and to document recommendations for maintainers as to how and when to ship linux-gnu and linux-musl versions.

testing

To help test this, I published "linux + linux-musl" and "linux-gnu + linux-musl" combinations of the gem rcee_precompiled which you can find at https://rubygems.org/gems/rcee_precompiled/versions.

v0.5.0 and v0.5.0.1 are "linux + linux-musl"
v0.5.1 is `"linux-gnu + linux--musl"

Note that the versions in question being tested below are:

ruby	gem	bundler
3.0	3.2.33	2.2.33
3.0 (updated rubygems)	3.5.7	2.5.7
3.1	3.3.26	2.3.26
3.2	3.4.19	2.4.19
3.3	3.5.3	2.5.3

and the alpine containers have build-base installed to compile the source gem if necessary.

We test Ruby 3.0 with updated rubygems because its default version is below the known minimum needed for this to work correctly (3.3.22).

testing `gem install`

target	linux + linux-musl (v0.5.0.1)	linux-gnu + linux-musl (v0.5.1)
3.0 gnu	✔ installs `-linux`	🤷 installs source gem
3.0 musl	✖ installs `-linux`	🤷 installs source gem
3.0 gnu (updated rubygems)	✔ installs `-linux`	✔ installs `-linux-gnu`
3.0 musl (updated rubygems)	✔ installs `-linux-musl`	✔ installs `-linux-musl`
3.1 gnu	✔ installs `-linux`	✔ installs `-linux-gnu`
3.1 musl	✔ installs `-linux-musl`	✔ installs `-linux-musl`
3.2 gnu	✔ installs `-linux`	✔ installs `-linux-gnu`
3.2 musl	✔ installs `-linux-musl`	✔ installs `-linux-musl`
3.3 gnu	✔ installs `-linux`	✔ installs `-linux-gnu`
3.3 musl	✔ installs `-linux-musl`	✔ installs `-linux-musl`

testing `bundle install`

target	linux + linux-musl (v0.5.0.1)	linux-gnu + linux-musl (v0.5.1)
3.0 gnu	✔ installs `-linux`	🤷 installs source gem
3.0 musl	🤷 installs source gem	🤷 installs source gem
3.0 gnu (updated rubygems)	✖ fails to install [1]	✔ installs `-linux-gnu`
3.0 musl (updated rubygems)	✔ installs `-linux-musl`	✔ installs `-linux-musl`
3.1 gnu	✔ installs `-linux`	✔ installs `-linux-gnu`
3.1 musl	✔ installs `-linux-musl`	✔ installs `-linux-musl`
3.2 gnu	✔ installs `-linux`	✔ installs `-linux-gnu`
3.2 musl	✔ installs `-linux-musl`	✔ installs `-linux-musl`
3.3 gnu	✔ installs `-linux`	✔ installs `-linux-gnu`
3.3 musl	✖ installs `-linux`	✔ installs `-linux-musl`

Error [1]:

Could not find gems matching 'rcee_precompiled (= 0.5.0.1)' valid for all resolution platforms (aarch64-linux-musl,
aarch64-linux, arm-linux-musl, arm-linux, arm64-darwin, x86-linux-musl, x86-linux, x86_64-darwin, x86_64-linux-musl) in
rubygems repository https://rubygems.org/ or installed locally.

The source contains the following gems matching 'rcee_precompiled (= 0.5.0.1)':
  * rcee_precompiled-0.5.0.1-aarch64-linux
  * rcee_precompiled-0.5.0.1-aarch64-linux-musl
  * rcee_precompiled-0.5.0.1-arm-linux
  * rcee_precompiled-0.5.0.1-arm-linux-musl
  * rcee_precompiled-0.5.0.1-arm64-darwin
  * rcee_precompiled-0.5.0.1
  * rcee_precompiled-0.5.0.1-x64-mingw-ucrt
  * rcee_precompiled-0.5.0.1-x64-mingw32
  * rcee_precompiled-0.5.0.1-x86-linux
  * rcee_precompiled-0.5.0.1-x86-linux-musl
  * rcee_precompiled-0.5.0.1-x86_64-darwin
  * rcee_precompiled-0.5.0.1-x86_64-linux
  * rcee_precompiled-0.5.0.1-x86_64-linux-musl

Recommendations

Given the inconsisent behavior of gem and bundle when the gem in question ships linux and linux-musl platform gems, it seems pretty clear that it's not a good idea to do this.

So the recommendation has to be:

either:
- ship a -linux platform gem that works for both gnu and musl systems,
- or ship both -linux-gnu and -linux-musl platform gems.
do NOT ship -linux and -linux-musl platform gems for the same version.

The installation docs for users should contain some warnings:

end users on Ruby 3.0 must be using gem 3.3.22 or later / bundler 2.3.21 or later to get basic support for precompiled gems (will fall back to installing the vanilla "ruby" platform gem) set required_rubygems_version for native gems that specify the linux libc rake-compiler#236
end users on musl systems should be using bundler 2.5.6 or later to avoid on musl systems, bundler does not include the relevant musl native gem platform in the lock file rubygems/rubygems#7432

The text was updated successfully, but these errors were encountered:

flavorjones · 2024-03-24T19:41:27Z

OK, so understanding that bundler's behavior has changed in at least two unfortunate ways for the "linux + linux-musl" scenario:

at some version since 2.5.3, on gnu target systems, bundler stopped being able to resolve a gem
at some version between 2.4.19 and 2.5.3, on musl target systems, bundler stopped being able to resolve a gem

Let's dig in.

gnu target platform

On a gnu system (docker image ruby:3.2), running ruby 3.2.3, with the following Gemfile:

source "https://rubygems.org"

gem "rcee_precompiled", "0.5.0.1"

bundler	status
2.4.19	✔
2.4.22	✔
2.5.0	✔
2.5.3	✔
2.5.4	✔
2.5.5	✔
2.5.6	✖ [1]

Error [1]: git-bisected to rubygems/rubygems@66ee354

Could not find gems matching 'rcee_precompiled (= 0.5.0.1)' valid for all resolution platforms (aarch64-linux-musl,
aarch64-linux, arm-linux-musl, arm-linux, arm64-darwin, x86-linux-musl, x86-linux, x86_64-darwin, x86_64-linux-musl) in
rubygems repository https://rubygems.org/ or installed locally.

The source contains the following gems matching 'rcee_precompiled (= 0.5.0.1)':
  * rcee_precompiled-0.5.0.1-aarch64-linux
  * rcee_precompiled-0.5.0.1-aarch64-linux-musl
  * rcee_precompiled-0.5.0.1-arm-linux
  * rcee_precompiled-0.5.0.1-arm-linux-musl
  * rcee_precompiled-0.5.0.1-arm64-darwin
  * rcee_precompiled-0.5.0.1
  * rcee_precompiled-0.5.0.1-x64-mingw-ucrt
  * rcee_precompiled-0.5.0.1-x64-mingw32
  * rcee_precompiled-0.5.0.1-x86-linux
  * rcee_precompiled-0.5.0.1-x86-linux-musl
  * rcee_precompiled-0.5.0.1-x86_64-darwin
  * rcee_precompiled-0.5.0.1-x86_64-linux
  * rcee_precompiled-0.5.0.1-x86_64-linux-musl

musl target platform

On a musl system (docker image ruby:3.2-alpine), running ruby 3.2.3, with the following Gemfile:

source "https://rubygems.org"

gem "rcee_precompiled", "0.5.0.1"

bundler	status
2.4.19	✔
2.4.20	✔
2.4.21	✔
2.4.22	✔
2.5.0	✖ [1]
2.5.5	✖ [1]
2.5.6	✔ [2]
2.5.7	✔

Error [1]: incorrectly installs x86_64-linux instead of x86_64-linux-musl

# Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    rcee_precompiled (0.5.0.1)
      mini_portile2
    rcee_precompiled (0.5.0.1-aarch64-linux)
    rcee_precompiled (0.5.0.1-aarch64-linux-musl)
    rcee_precompiled (0.5.0.1-arm-linux)
    rcee_precompiled (0.5.0.1-arm-linux-musl)
    rcee_precompiled (0.5.0.1-arm64-darwin)
    rcee_precompiled (0.5.0.1-x86-linux)
    rcee_precompiled (0.5.0.1-x86-linux-musl)
    rcee_precompiled (0.5.0.1-x86_64-darwin)
    rcee_precompiled (0.5.0.1-x86_64-linux)

PLATFORMS
  aarch64-linux
  aarch64-linux-musl
  arm-linux
  arm-linux-musl
  arm64-darwin
  ruby
  x86-linux
  x86-linux-musl
  x86_64-darwin
  x86_64-linux

DEPENDENCIES
  rcee_precompiled (= 0.5.0.1)

BUNDLED WITH
   2.5.0

Note [2]: rubygems/rubygems#7432 was fixed in 2.5.6 by rubygems/rubygems#7441

flavorjones · 2024-03-24T20:00:38Z

I've updated the recommendations in the description. I'll be opening a bug report on rubygems shortly to report these findings.

flavorjones · 2024-04-15T17:30:30Z

I've created an issue upstream at test: demonstrate issue with installing linux-musl gems by flavorjones · Pull Request #7583 · rubygems/rubygems for the issue described above in bundler 2.5.6.

per the findings from #117

flavorjones · 2024-04-15T20:07:12Z

Recommendations added to the README. Closing.

described in rake-compiler/rake-compiler-dock#117

ntkme · 2024-05-08T20:05:59Z

Debian unstable really screwed it up: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069089

Debian unstable's ruby3.1 package currently bundles rubygems 3.4.20. However, Debian's maintainers patched it in a way that behavior like 3.3.15:

diff --git a/lib/rubygems/platform.rb b/lib/rubygems/platform.rb
index b721629b7..a16c6fca9 100644
--- a/lib/rubygems/platform.rb
+++ b/lib/rubygems/platform.rb
@@ -117,6 +117,9 @@ def initialize(arch)
       when /^(\w+_platform)(\d+)?/ then [ $1,          $2  ]
       else                              [ "unknown",   nil ]
       end
+
+      # emulate behavior from 3.3.15 on ruby 3.1
+      @version = nil if (RUBY_VERSION < "3.2" && @os == "linux")
     when Gem::Platform then
       @cpu = arch.cpu
       @os = arch.os

It's completely broken despite 3.4.20 > 3.3.22...

irb(main):001:0> Gem::Platform.new('aarch64-linux-gnu')
=> #<Gem::Platform:0x0000ffffa56fc8e0 @cpu="aarch64", @os="linux", @version=nil>
irb(main):002:0> Gem::Platform.new('aarch64-linux-musl')
=> #<Gem::Platform:0x0000ffffa56c8680 @cpu="aarch64", @os="linux", @version=nil>
irb(main):003:0> Gem::Platform.new('aarch64-linux-musl') == Gem::Platform.new('aarch64-linux-gnu')
=> true

I have already got a few bug reports from users on Debian unstable. What I don't want to do is to block rubygems <= 3.4.20, because ruby3.2 on Debian unstable also have same patched rubygems 3.4.20, but works properly (see the version check in the patch above).

I have tried a few ways to reduce the noises from users by checking if they have a broken rubygems and raise a runtime error, etc. and unfortunately it brought even more noise and confusion. The final solution I end up with is to conditionally block ruby <3.2 on linux native gems, as ruby >=3.2 seems to always have a good default rubygems version (assuming users won't downgrade default rubygems).

  spec.required_ruby_version = if spec.platform == Gem::Platform::RUBY || spec.platform.os != 'linux'
                                 '>= 3.1.0'
                               else
                                 '>= 3.2.0'
                               end

flavorjones · 2024-05-13T14:32:37Z

@ntkme Thanks for adding this context, that's really too bad about the downstream patches that get applied in the Sid distro. I hope it gets fixed.

flavorjones mentioned this issue Apr 15, 2024

Fix issue installing gems with linux-musl variant on non musl linux rubygems/rubygems#7583

Merged

flavorjones added a commit that referenced this issue Apr 15, 2024

doc: clarify gnu/musl recommendations for maintainers and end-users

67e05f1

per the findings from #117

flavorjones closed this as completed Apr 15, 2024

This was referenced Apr 15, 2024

introduce general musl support sparklemotion/sqlite3-ruby#442

Merged

feat: first-class musl support sparklemotion/nokogiri#3111

Closed

deivid-rodriguez pushed a commit to flavorjones/rubygems that referenced this issue Apr 16, 2024

test: demonstrate issue with installing linux-musl gems

faa7aba

described in rake-compiler/rake-compiler-dock#117

ntkme mentioned this issue Jan 15, 2025

Ruby google-protobuf linux precompiled gem is not compatible with linux-musl protocolbuffers/protobuf#16853

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

document how bundler and rubygems behave wrt `linux`, `linux-gnu`, and `linux-musl` releases #117

document how bundler and rubygems behave wrt `linux`, `linux-gnu`, and `linux-musl` releases #117

flavorjones commented Mar 3, 2024 •

edited

Loading

flavorjones commented Mar 24, 2024 •

edited

Loading

flavorjones commented Mar 24, 2024

flavorjones commented Apr 15, 2024

flavorjones commented Apr 15, 2024

ntkme commented May 8, 2024 •

edited

Loading

flavorjones commented May 13, 2024

document how bundler and rubygems behave wrt linux, linux-gnu, and linux-musl releases #117

document how bundler and rubygems behave wrt linux, linux-gnu, and linux-musl releases #117

Comments

flavorjones commented Mar 3, 2024 • edited Loading

testing

testing gem install

testing bundle install

Recommendations

flavorjones commented Mar 24, 2024 • edited Loading

gnu target platform

musl target platform

flavorjones commented Mar 24, 2024

flavorjones commented Apr 15, 2024

flavorjones commented Apr 15, 2024

ntkme commented May 8, 2024 • edited Loading

flavorjones commented May 13, 2024

document how bundler and rubygems behave wrt `linux`, `linux-gnu`, and `linux-musl` releases #117

document how bundler and rubygems behave wrt `linux`, `linux-gnu`, and `linux-musl` releases #117

flavorjones commented Mar 3, 2024 •

edited

Loading

testing `gem install`

testing `bundle install`

flavorjones commented Mar 24, 2024 •

edited

Loading

ntkme commented May 8, 2024 •

edited

Loading