Add support for Ruby 3.0.0-preview1

[JonRowe]

👋 We're in the process of switching RSpec over to using Github actions and it would be helpful to use to have have access to preview versions of Ruby 3.0.0.

(It'd also be helpful to have some legacy versions of Ruby such as 1.8.7. and 1.9.2 but I realise thats probably very out of scope, if anyone wants to help get those running on Github actions with us it would be very appreciated).

[hsbt]

FYI ruby/ruby-builder#4 (comment)

[JonRowe]

I disagree with that conclusion, preview builds are more useful to a gem library because they indicate a stable build that should work, head often contains things that are in progress and flux. We're safely able to get consistency with a preview build that we cannot with head, meaning we often have to allow failures for head for the stability of our builds, but we don't have that problem with preview.

[MSP-Greg]

@JonRowe

We're in the process of switching RSpec over to using Github actions

I did look thru some of the RSpec repos, and didn't find any GitHub Actions (GHA) workflow files. Anyway...

Two differences between GHA and popular CI services are:

1. A much higher number of concurrent jobs.
2. 'Allow failure' is still not as friendly as what people were used to when using Travis or AppVeyor.

So, ruby/ruby is now running a lot more CI jobs than were run using previous CI services.

Before GHA, I worked quite a bit with the Travis head build, and it was an automated build that was not tested.

All current Ruby MRI head builds in this action are fully tested. There are even a few CLI/bin tests run that check whether the 'install' works, as these may not be tested in ruby/ruby.

If tests do not pass, the previous passing head build is left as the build used for CI testing. Ubuntu & macOS builds are done once a day, Windows are run three times a day.

There are quite a few repos using the head builds available here, and many do not implement an 'Allow failure' setup for those jobs.

If you're basing your opinion on other CI services, maybe try the GHA builds first. If you are using the builds, and are having issues with failures, any feedback (either here or ruby/ruby) would be appreciated.

Thanks for RSpec...

[JonRowe]

It's on a branch and an issue as we experiment, its early days, you can see the latest "most rubies" setup here:

https://github.com/rspec/rspec-core/actions/runs/343419327

We're moving off Travis because Travis is limiting support for open source.

Currently we are missing support for 3.0.0-preview1 which would have to remain on Travis, or build a custom image like we may have to do for older Rubies.

[JonRowe]

We find people actually use released versions of Ruby, like preview or rc1, rc 2 etc so we need to support them by testing those specific versions.

[MSP-Greg]

Thanks for the GHA run link. I may not have been clear enough, but everything I stated was very specific to MRI builds.

We find people actually use released versions of Ruby, like preview or rc1, rc 2 etc

I can understand that, but it can also be stated that adoption of head/master builds for CI has been good, so why should the (probably smaller set of) people using rc builds be accounted for? As mentioned in the link @hsbt provided, rc builds really don't have a lot of significance. There's a good chance if someone reported a bug, the response would be 'have you tried it with current master?'

like we may have to do for older Rubies.

Old compilers, unsecure OpenSSL etc. It's odd when a group states that they don't have the resources to update their app, so they'd like other people to continue to support old tools used in their infrastructure, whether that be apps, gems, or CI tools.

It's a matter of opinion. From my perspective, given finite resources, there are other things to work on other than supporting rc builds...

Edit: I'd rather encourage people to use head/master builds all year long rather than use rc builds for a few months.

[JonRowe]

I can understand that, but it can also be stated that adoption of head/master builds for CI has been good, so why should the (probably smaller set of) people using rc builds be accounted for? As mentioned in the link @hsbt provided, rc builds really don't have a lot of significance. There's a good chance if someone reported a bug, the response would be 'have you tried it with current master?'

We don't get people reporting issues from head, where as we do with the official preview versions. They report bugs to us from RSpec on those versions, we do occasionally refer issues back to Ruby but generally its something a new version is introducing for us to handle that doesn't change.

From our perspective we use our finite resources to support fixed versions, especially as preview and release candidates lead to the support of full versions.

[eregon]

We're in the process of switching RSpec over to using Github actions

Great to hear :)

I disagree with that conclusion, preview builds are more useful to a gem library because they indicate a stable build that should work, head often contains things that are in progress and flux. We're safely able to get consistency with a preview build that we cannot with head, meaning we often have to allow failures for head for the stability of our builds, but we don't have that problem with preview.

That's kind of my point, previews are not stable builds, consider them basically a random commit from ruby/ruby master.
Of course, they are a fixed build, so they don't change and remain as good or broken as they are.

The purpose of previews (AFAIK) is to make it easier to build a development version of the upcoming version and gather early feedback. Given Ruby installers support installing ruby-head/ruby-dev I'm not sure what's the extra value nowadays, except announcing "this is a good time to try the upcoming features".

Following the idea "make it easier to try", it might make sense to have it for ruby/ruby-builder too, but OTOH ruby/ruby-builder builds are not meant for end users (only to run in GitHub Actions).
Concretely, there are several practical problems, highlighted in ruby/ruby-builder#4 (comment). Most of them apply to RCs too:

• No Windows builds of previews. I don't know anyone making reusable builds of Ruby previews on Windows (yet). Having inconsistent platform support is a source of confusion.
• It's quite a few extra builds, and these builds don't make sense to be used after the real release. Can we drop those builds after the release? Probably not so easy, we would first need to deprecate them before dropping them.
• Previews are not more stable than ruby-head in my experience. I'm happy to see counter examples, using setup-ruby (as @MSP-Greg mentioned, the builds here are significantly more stable than TravisCI head builds).
• There can be many breaking changes between a preview and the next release (e.g. Fiber scheduler & Ractor API is changing recently), so it often doesn't make too much sense to test in CI, at least not for new APIs that have not stabilized yet.

head often contains things that are in progress and flux

So do previews, several APIs have already been renamed since preview1 and many bugs fixed.

We find people actually use released versions of Ruby, like preview or rc1, rc2 etc so we need to support them by testing those specific versions.

rc1/rc2 etc do aim for progressively more stability.
Nobody should use previews or RCs in production, I think that should be considered unsupported.

I agree with @MSP-Greg. Testing ruby-head seems enough and very similar for stability.
This is not the same as TravisCI ruby-head builds.

I'll reconsider if there are several concrete cases where using ruby-head is less stable than a preview, and the error is not something that would be valuable to report to Ruby core.

like we may have to do for older Rubies.

Yes, that definitely needs Docker or a VM with a old OS. I don't think it's possible to build 1.8 or 1.9 on anything remotely recent.
My personal opinion is 1.8/1.9 support should be dropped, unless a company sponsors it and also sponsors the needed CI resources & builds for it.

[JonRowe]

3.0.0.preview1 is a fixed in time release, even if its marked as unstable, it's even on the official download page:
https://www.ruby-lang.org/en/downloads/

So do previews, several APIs have already been renamed since preview1 and many bugs fixed.

Those changes are not in preview1 though, they will be in preview2 or rc1 etc.

[eregon]

We don't get people reporting issues from head, where as we do with the official preview versions.

I would treat them exactly the same as reports from people using ruby-head.
To validate the bug, I would check it's not been fixed on ruby-head. If it is fixed on ruby-head, there is IMHO no point to change anything. That illustrates for me that it's always better to test against ruby-head than previews.
Maybe we need to educate users that previews are not stable and bugs are expected.

@JonRowe Could you try ruby-head and if that causes extra failures link them here?

To really consider this, we would need to solve the 4 items above. It seems a lot more work than what it's worth.
It's useful to test against ruby-head all year round (even if allow failures), but there are only a couple previews per year.

[eregon]

FWIW there are Docker builds for each commit of ruby/ruby at https://hub.docker.com/r/rubylang/rubyfarm
Maybe that's a good solution if someone wants to test a fixed development version.
ruby/setup-ruby cannot support a persisted build for each commit, that would be way too much.

[MSP-Greg]

@JonRowe Could you try ruby-head and if that causes extra failures link them here?

From the link above (https://github.com/rspec/rspec-core/actions/runs/343419327), it appears to have passed?

See: https://github.com/rspec/rspec-core/runs/1346985289

[eregon]

I'm thinking having RCs in ruby/setup-ruby might be nice, to catch extra bugs before the real release.
Although, testing ruby-head is likely even better, and doesn't need any CI adjustment to pick up RC-like builds.
And previews are similar from a building point of view.

Interestingly, TravisCI doesn't seem to have 3.0.0-preview1, but it has other previews and RCs: https://rubies.travis-ci.org/

No Windows builds of previews. I don't know anyone making reusable builds of Ruby previews on Windows (yet). Having inconsistent platform support is a source of confusion.

I think this is the main issue, and I have no idea how to solve it.
Maybe we just don't have them on Windows, and if someone asks "there are no Windows preview builds provided anywhere, and we don't have the resources and time to figure out how to build them on Windows".
Building previews on Ubuntu & macOS should be easy, if they do build without any patch.

It's quite a few extra builds, and these builds don't make sense to be used after the real release. Can we drop those builds after the release? Probably not so easy, we would first need to deprecate them before dropping them.

I guess we can just keep them, it's often not so many previews.

There can be many breaking changes between a preview and the next release (e.g. Fiber scheduler & Ractor API is changing recently), so it often doesn't make too much sense to test in CI, at least not for new APIs that have not stabilized yet.

Code in CI is probably unlikely to use new experiemental APIs. But in any case, it's not ruby/setup-ruby's problem, so I think it's fair to ignore this point.
This point is about communicating what a preview is, and it seems there is some confusion about them.

Previews are not more stable than ruby-head in my experience. I'm happy to see counter examples, using setup-ruby (as @MSP-Greg mentioned, the builds here are significantly more stable than TravisCI head builds).

This point still stands IMHO. But it seems in some cases people want to test against both ruby-head and previews like in the run above.

I think @JonRowe wants a good way to answer these issues reported with 3.0.0-preview1. Testing it in CI is one way to ensure it works of course. Other possibilities to address such issues:

• If it fails on preview1 but passes on head: "Try ruby-head instead, previews are unstable builds and that is a bug of the preview."
• If it fails on both preview1 and head, the question is "does RSpec support ruby-head?". If yes, then there is no point to test preview1. If no, or best effort, it still seems far more useful to check against ruby-head.
  Does that make sense?

[JonRowe]

Interestingly, TravisCI doesn't seem to have 3.0.0-preview1, but it has other previews and RCs: https://rubies.travis-ci.org/

Travis uses RVM, any version on RVM is available automatically but they don't autogenerate their docs.

[MSP-Greg]

Re Windows, the head/master build system builds from the repo, so builds could be done based on tags. Given how few there are, I'd probably trigger builds manually.

But, the other items needed to interact with this action aren't in place (location/releases, available versions, etc).

I suppose we could patch https://github.com/ruby/setup-ruby/blob/master/windows-versions.js manually...

And, all the issues you mentioned, like when to remove, etc...

I guess I still don't see good reason for them. Someone runs CI on a two week old preview/rc, it fails, but head/master passes. What has anyone learned?

[JonRowe]

I guess I still don't see good reason for them. Someone runs CI on a two week old preview/rc, it fails, but head/master passes. What has anyone learned?

In our case, that something in RSpec doesn't work with a released version of Ruby, and needs fixing or otherwise handling.

[eregon]

In our case, that something in RSpec doesn't work with a released version of Ruby

My point of view, and I'm confident this is the point of view of Ruby core too is that previews are not released versions of CRuby.
They are a development snapshot at a fixed commit to let people try new/upcoming features, but they don't have improved stability or really much of a release process and are therefore very different than releases.

In other words, I think if RSpec officially supports only released versions (seems reasonable for most gems), then it would be completely fair to say it does not support previews (and RCs for that matter), because they are not releases (or "released versions").

Of course, it is still interesting to track ruby-head in RSpec's CI, just it isn't officially supported.

[JonRowe]

It's released as in it's on the official downloads page as a preview version, and gets built and used as a binary ruby by developers and is a named version.

[MSP-Greg]

I might be willing to call them 'prereleases', but certainly not 'releases'. What's their lifetime? A few weeks?

Speaking of Rubygems, they test on ruby-head and truffleruby daily. As of right now, the first page run list shows only one failure, and that's from truffleruby:

https://github.com/rubygems/rubygems/actions?query=workflow%3Adaily-rubygems

I suspect you can find plenty of other gems/repos with similar results. Until you've tried it...

[robotdana]

as a person who uses rspec, and before i'd brought up the issue on ruby-builder, i would expect that trying out a ruby prerelease/rc means adding it to the matrix of rubies running my gems (on travis or GitHub actions or just running it on my computer), fixing any incompatibilities that are documented/expected and therefore my fault, and reporting bugs/PRs to authors of gems i'm using for any incompatibilities that seem to be issues with their gems, and to ruby anything that seems to run counter to the documented changes for that prerelease/rc. i wouldn't however feel the same responsibility to try it out and report issues the same way for daily/head builds. maybe my interpretation/expectation is my own confusion and not a real thing ruby team is expecting to be different between rc/prerelease and head/daily

[MSP-Greg]

@robotdana

Assuming you've read the rest of the comments...

For me, this is messy, and involves past experiences. I'm not a part of 'Ruby Core', but I've been involved with Windows Ruby for several years, and interacted with them due to that. I've heard people's impressions of the Core team (often negative), and this started when Travis and AppVeyor were the standards for CI.

I've concluded that repos/gems testing with ruby head/master provides valuable feedback to the Core team, and they will react to it.

Also, even though this is a Ruby repo, the number of people maintaining it is much smaller than the Core team. The people here are involved in other repos, and many of them are running CI on head/master.

Hence, 'confusion' really isn't the case, but also the Core team really hasn't addressed the issue, nor is there a communications channel for them to do so.

So, my initial reaction to a request for preview/rc builds is, if you're not testing with head/master, come back after you've done that for long enough to determine if it works. There are plenty of repos where it does.

<rant off>

[JonRowe]

We test ruby head as well. It is not sufficient as we don't get people complaining about issues from it, because most people don't compile Ruby from source to develop with. They do rvm install 3.0.0-preview or equivalent, they don't use commits or head because thats not a consistent commonly shared version. They do come and say "hey this doesn't work on 3.0.0-preview" and mostly thats us, and sometimes it's a Ruby bug, we can compare the build that fails on 3.0.0-preview1 to head and say "hey its fixed in head" and decide wether to worry about it, or we can fix it.

It, running CI for preview releases, is part of our workflow and has been for a long time, we're investigating options to replace our Travis setup and this repo is part of that option currently because it supports the most Rubies so far. We cannot drop supported Ruby versions until the next major, RSpec 4, which we are planning for release around Ruby 3.0.0. until then we are probably going to be supplement Github actions with travis to provide the older builds, but they are currently a major bottleneck spending hours queued. So I may end up forking this / rolling my own docker images to support our full matrix on github actions.

So if you do wish to reject this request I understand, but respectfully disagree that preview and rc Rubies are not important enough to test against, I apologise for dragging this off track a bit, that wasn't my intent.

[MSP-Greg]

No need for an apology. Have you thought about doing a CRON build once a day against ruby-head? RubyGems/Bundler does that. I don't know how often you have commits...

One issue is whether the problem is your master with ruby master, or your release with ruby master.

I've been around for new Ruby releases, both major/minor and patch releases, and people expect them immediately. Previews and rc's would probably be the same, and their lifetime is so short...

[eregon]

@robotdana This is the difference between previews and RCs. Previews are not meant to test compatibility, RCs are.
So I think having RCs makes a lot of sense, but less sure about previews.
If we have RC builds, we could as well have builds for previews though.
Both of these would be to "try it out in CI", so probably not having Windows builds is not a huge deal (ruby-head is available on Windows though, including mingw & mswin variants).
I guess we should deprecate/warn previews as soon as there is a RC, and deprecate RCs as soon as there is a release, so to make it clear it's not worthwhile to keep testing those after the release. Extra work though.

Previews are only to try out new features, and they might have many incompatible changes, so trying to be compatible with a preview is often not such a good idea. If people want to track compatibility with ruby development, they should use ruby-head, not previews (we said that too many times in this thread :D).
As an example, I believe there is absolutely no value to be compatible with a preview once a RC or the release is out. It's the same "it's compatible with ruby-head N months ago", which by itself has very little value.

@JonRowe

So I may end up forking this / rolling my own docker images to support our full matrix on github actions.

I think you will need Docker images for 1.8 & 1.9 anyway.
BTW, this action is meant to be reusable with other builds, so there would be no need to fork this action, but rather to provide a location with the extra builds. We would then have some syntax to use builds from another repository than ruby/ruby-builder.
So if someone somehow managed to build Ruby 1.8/1.9 on GitHub runners, that would also be a possibility (but it's probably not worth the effort, Docker seems much easier there).

Anyway, I am thinking for RCs at least it seems worth supporting in ruby/ruby-builder.

[eregon]

I included 3.0.0-preview1 (on Ubuntu & macOS) for the new the set of builds that will be used for #98.
https://github.com/ruby/ruby-builder/releases/tag/toolcache

[eregon]

Released as https://github.com/ruby/setup-ruby/releases/tag/v1.50.0

[JonRowe]

👏 Thanks, it works for us

[dentarg]

I have a (private) build that successfully used 3.0.0-preview1 ~16 days ago (14 Nov 2020 23:40 CET):

Run ruby/setup-ruby@v1
  with:
    bundler-cache: true
    ruby-version: default
    bundler: default
    working-directory: .
Using 3.0.0 as input from file .ruby-version
Downloading Ruby
  https://github.com/ruby/ruby-builder/releases/download/toolcache/ruby-3.0.0-preview1-ubuntu-18.04.tar.gz
  Took   0.55 seconds
Extracting Ruby
  /bin/tar -xz -C /opt/hostedtoolcache/Ruby/3.0.0-preview1 -f /home/runner/work/_temp/b0a24f40-b79a-4a20-bdc6-743baf32dc9b
  Took   0.40 seconds
Installing Bundler
  Using Bundler 2 from Gemfile.lock BUNDLED WITH 2.2.0.rc.1
  /opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/gem install bundler -v ~> 2 --no-document
  Successfully installed bundler-2.1.4
  1 gem installed
  Took   0.58 seconds
bundle install
  /opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/bundle config --local path vendor/bundle
  /opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/bundle config --local deployment true
  Cache key: setup-ruby-bundler-cache-v2-ubuntu-18.04-ruby-3.0.0-preview1-Gemfile.lock-f3baecba40e3a8b5622afdca11282aa1ededc7dc12895472af03c0685797b471
  /opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/bundle install --jobs 4
  Fetching gem metadata from https://rubygems.org/....
  Using bundler 2.2.0.rc.1
  Fetching ruby2_keywords 0.0.2
  Fetching nio4r 2.5.4
  Fetching rack 2.2.3
  Fetching tilt 2.0.10
  Installing ruby2_keywords 0.0.2
  Installing tilt 2.0.10
  Installing nio4r 2.5.4 with native extensions
  Installing rack 2.2.3
  Fetching mustermann 1.1.1
  Installing mustermann 1.1.1
  Fetching rack-protection 2.1.0
  Fetching puma 5.0.4
  Installing rack-protection 2.1.0
  Fetching sinatra 2.1.0
  Installing puma 5.0.4 with native extensions
  Installing sinatra 2.1.0
  Bundle complete! 2 Gemfile dependencies, 9 gems now installed.
  Bundled gems are installed into `./vendor/bundle`
  Saving cache
  /bin/tar --posix --use-compress-program zstd -T0 -cf cache.tzst -P -C /home/runner/work/swiss/swiss --files-from manifest.txt
  Cache saved successfully
  Took   9.36 seconds

Now (~7 hours ago) it fails:

Run ruby/setup-ruby@v1
  with:
    bundler-cache: true
    ruby-version: default
    bundler: default
    working-directory: .
Using 3.0.0 as input from file .ruby-version
Error: Unknown version 3.0.0 for ruby on ubuntu-18.04
        available versions for ruby on ubuntu-18.04: 2.1.9, 2.2.10, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.9, 2.4.10, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.5.6, 2.5.7, 2.5.8, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.6.6, 2.7.0, 2.7.1, 2.7.2, 3.0.0-preview1, head, debug
        File an issue at https://github.com/ruby/setup-ruby/issues if would like support for a new version

Was it a mistake that 3.0.0 resolved to 3.0.0-preview1?

I changed the contents of .ruby-version from 3.0.0 to 3.0.0-preview1 and ran the build again:

Run ruby/setup-ruby@v1
  with:
    bundler-cache: true
    ruby-version: default
    bundler: default
    working-directory: .
Using 3.0.0-preview1 as input from file .ruby-version
Downloading Ruby
  https://github.com/ruby/ruby-builder/releases/download/toolcache/ruby-3.0.0-preview1-ubuntu-18.04.tar.gz
  Took   0.67 seconds
Extracting Ruby
  /bin/tar -xz -C /opt/hostedtoolcache/Ruby/3.0.0-preview1 -f /home/runner/work/_temp/91dc075f-6e80-44cf-80fc-692dc13162b9
  Took   0.38 seconds
Installing Bundler
  Using Bundler 2 from Gemfile.lock BUNDLED WITH 2.2.0.rc.1
  /opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/gem install bundler -v ~> 2 --no-document
  Successfully installed bundler-2.1.4
  1 gem installed
  Took   0.53 seconds
bundle install
  /opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/bundle config --local path vendor/bundle
  /opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/bundle config --local deployment true
  Cache key: setup-ruby-bundler-cache-v2-ubuntu-18.04-ruby-3.0.0-preview1-Gemfile.lock-f3baecba40e3a8b5622afdca11282aa1ededc7dc12895472af03c0685797b471
  /opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/bundle install --jobs 4
  Your Ruby version is 3.0.0, but your Gemfile specified 3.0.0.pre.preview1
  Took   0.84 seconds
Error: The process '/opt/hostedtoolcache/Ruby/3.0.0-preview1/x64/bin/bundle' failed with exit code 18

From Gemfile.lock:

RUBY VERSION
   ruby 3.0.0p-1

BUNDLED WITH
   2.2.0.rc.1

[dentarg]

I can't reproduce the last failure above, Using 3.0.0-preview1 as input from file .ruby-version in another repo works: https://github.com/dentarg/actions-test/runs/1478587439?check_suite_focus=true#step:3:1 – I guess there's some issue with the cache saved from the successful run 16 days ago? (I updated my comment above to include all the output from setup-ruby from the successful run)

[eregon]

I recently fixed version parsing so it would work for ruby-3.0.0-preview1:
https://github.com/ruby/setup-ruby/releases/tag/v1.51.3

3.0.0 should work, but of course it will pick the future previews/RCs/3.0.0 release.
3.0 probably makes more sense if you want that.

If you want 3.0.0-preview1 specifically, then of course 3.0.0-preview1 should be used.

I can't reproduce the last failure above, Using 3.0.0-preview1 as input from file .ruby-version in another repo works:

Does it fail if you use 3.0.0 in .ruby-version? That would probably be a bug.
Please report as a separate issue though, I'm not tracking closed issues and it notifies other people needlessly.

[eregon]

I plan to remove preview/RC builds which already have a full release, see #367