[draft] ruby 3.2 support

[flavorjones]

This is a naive attempt (based on #81) to add Ruby 3.2.0 support using the recent rc1.

I've built the x86_64-linux image on my dev machine, and used it to build Nokogiri libraries that support 3.2 and pass all the tests.

The changes I made to Nokogiri are:

diff --git a/.cross_rubies b/.cross_rubies
index b585a0e..5c42e9c 100644
--- a/.cross_rubies
+++ b/.cross_rubies
@@ -30,3 +30,11 @@
 3.1.0:x86-mingw32
 3.1.0:x86_64-darwin
 3.1.0:x86_64-linux
+3.2.0:aarch64-linux
+3.2.0:arm-linux
+3.2.0:arm64-darwin
+3.2.0:x64-mingw-ucrt
+3.2.0:x86-linux
+3.2.0:x86-mingw32
+3.2.0:x86_64-darwin
+3.2.0:x86_64-linux
diff --git a/rakelib/extensions.rake b/rakelib/extensions.rake
index aabdb3e..ef67f9a 100644
--- a/rakelib/extensions.rake
+++ b/rakelib/extensions.rake
@@ -168,7 +168,7 @@ def allowed_dlls
         "libc.so.6",
         "libdl.so.2", # on old dists only - now in libc
       ].tap do |dlls|
-        dlls << "libpthread.so.0" if ver < "2.6.0"
+        dlls << "libpthread.so.0" if ver < "2.6.0" or ver >= "3.2.0"
       end
     when AARCH_LINUX_PLATFORM_REGEX
       [

I'm not sure why libpthread.so.0 shows up as a shared object dependency, when it hasn't been one since 2.5. @larskanis any ideas on that? It seems OK, the tests pass.

[larskanis]

Looks like a dependency issue in the makefiles on Macos. Everytime I added a new ruby version I got such dependency issues that appear only when cross compiling and often only with "make -j".

[larskanis]

libpthread was introduced as a dependency of C-exts due to requirements of the MJIT implementation. Now in Ruby-3.2 both MJIT as well as YJIT received a major refactoring, so I guess that's the root cause. I did a quick search in the ruby sources, but didn't find the corresponding commit. So I think the changes to nokogiri are OK.

That brings me back to the idea to move these checks from Nokogiri to RCD. I had the plan to introduce a class kind of RakeCompilerDock::ExtensionDll that knows how to check the DLL on Windows/Linux/Macos and that knows all the details of our cross ruby versions. It might be added to the Rakefile of Nokogiri and other projects to verify their output. Of course there must be a way to add additional DLLs to the check by subclassing or so.

[flavorjones]

I'll see if I can repro and diagnose it without the -j option on my local machine.

Re: the DLL checks, I agree it would be nice to have something in rake-compiler-dock (or some other, new gem?), but it's not a blocker. Maybe we can commit to working on that together in 2023, I would love to add checks like this to sqlite3-ruby, for example, but haven't gotten there yet.

If I get this to go green, and can build a Nokogiri on it that passes the tests, I'll merge this (unless you have objections). Then I'll tackle #71 so I can set up a Nokogiri CI pipeline that will use images generated from RCD head.

[flavorjones]

Isolated the issue to building 3.2.0-rc1. Here's the error:

make V=1 -f enc.mk V="1" UNICODE_HDR_DIR="/usr/local/rake-compiler/sources/ruby-3.2.0-rc1/enc/unicode/15.0.0" RUBY="/usr/local/rvm/rubies/ruby-3.1.0/bin/ruby --disable=gems " MINIRUBY="/usr/local/rvm/rubies/ruby-3.1.0/bin/ruby --disable=gems "  enc
make[1]: Entering directory '/usr/local/rake-compiler/builds/x86_64-apple-darwin/ruby-3.2.0-rc1'
x86_64-apple-darwin-clang -I. -I.ext/include/x86_64-darwin -I/usr/local/rake-compiler/sources/ruby-3.2.0-rc1/include -I/usr/local/rake-compiler/sources/ruby-3.2.0-rc1 -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT   -DONIG_ENC_REGISTER=rb_enc_register -fno-common -O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong  -fdeclspec -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wundef -pipe  -o enc/encdb.o -c /usr/local/rake-compiler/sources/ruby-3.2.0-rc1/enc/encdb.c
/usr/bin/mkdir -p .ext/x86_64-darwin/enc/trans
x86_64-apple-darwin-clang -dynamic -bundle -o .ext/x86_64-darwin/enc/encdb.bundle enc/encdb.o -L. -L. -L. -pipe  -fstack-protector-strong -pipe  -Wl,-multiply_defined,suppress -Wl,-undefined,dynamic_lookup -bundle_loader './miniruby'   -lpthread  
ld: file not found: ./miniruby
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I'm not sure where to start to debug this, so parking it here for bit until I've got more time.

[flavorjones]

Note: on a failing build, if I cd into /usr/local/rake-compiler/builds/x86_64-apple-darwin/ruby-3.2.0-rc1 and run make miniruby then a subsequent make will complete.

Is this a matter of making a change like this to rake-compiler?

diff --git a/tasks/bin/cross-ruby.rake b/tasks/bin/cross-ruby.rake
index 8317a2a..6c8a774 100644
--- a/tasks/bin/cross-ruby.rake
+++ b/tasks/bin/cross-ruby.rake
@@ -129,6 +129,7 @@
 
     # make
     file "#{build_dir}/ruby.exe" => ["#{build_dir}/Makefile"] do |t|
+      sh "#{MAKE} miniruby", chdir: File.dirname(t.prerequisites.first)
       sh MAKE, chdir: File.dirname(t.prerequisites.first)
     end
 

[flavorjones]

It looks like that patch does fix the build; and the need for it seems to come from ruby/ruby@50d81bf

@kateinoigakukun When building cross-rubies in this project, miniruby doesn't seem to be a makefile dependency, it's not getting built before it's needed to resolve symbols. Do you think this should be made a dependency upstream in Ruby's makefiles, or should we just make sure rake-compiler explicitly builds miniruby first?

Here's a dockerfile that reproduces the issue with the 3.2.0-rc1 tarball:

FROM larskanis/rake-compiler-dock-mri-x86_64-darwin:1.2.2

RUN wget https://cache.ruby-lang.org/pub/ruby/3.2/ruby-3.2.0-rc1.tar.gz
RUN tar -zxf ruby-3.2.0-rc1.tar.gz

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get -y update && \
    apt-get install -y curl git-core xz-utils build-essential zlib1g-dev libreadline-dev libssl-dev wget unzip sudo gnupg2 dirmngr cmake pkg-config autoconf libyaml-dev

ENV CC x86_64-apple-darwin-clang

RUN bash -c " \
      rvm use 3.1.0 && \
      cd ruby-3.2.0-rc1 && \
      ./configure \
        --host=x86_64-apple-darwin \
        --target=x86_64-apple-darwin \
        --build=x86_64-pc-linux-gnu \
        --disable-jit-support \
    "

RUN bash -c " \
      rvm use 3.1.0 && \
      cd ruby-3.2.0-rc1 && \
      make V=1 \
    "

but replacing the last command with this succeeds:

RUN bash -c " \
      rvm use 3.1.0 && \
      cd ruby-3.2.0-rc1 && \
      make miniruby V=1 && \
      make V=1 \
    "

[flavorjones]

I've opened an issue at https://bugs.ruby-lang.org/issues/19239

[flavorjones]

And I've suggested a fix at ruby/ruby#6944

[flavorjones]

It's not clear if that fix is going to be accepted upstream, so I'm going to leave the patch in this PR for now and merge it once it goes green again. We can remove it later if it becomes unnecessary.