-
-
Notifications
You must be signed in to change notification settings - Fork 263
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pre-cache official Ruby binaries on GitHub Actions images instead of custom binaries #98
Comments
Thanks for opening this issue, that makes sense to me. So the logic would be like:
Only Ruby >= 2.4 currently (2.4, 2.5, 2.6, 2.7), right? Is It might be possible to make the Ruby builds work in any directory. I'm not sure how feasible/reliable is that, but it would be convenient. Is it OK to write to the hostedtoolcache, so even downloaded versions would be placed there for consistency? Ruby gems will get installed under the Ruby prefix by default, and I would very much like to avoid messing with GEM_HOME/GEM_PATH variables, as that can lead to various non-trivial issues. What about Windows? On Windows, we noticed the D: drive seems faster than C:, probably because it seems to be a SSD (#14). Alternative Ruby implementations like JRuby & TruffleRuby would always be downloaded, unless you consider adding them to the hostedtoolcache too? |
Yes, exactly. If version is not found, download it and do
Yes, these paths won't be changed in future because too many things depend on it.
Yes, we have to find balance between disk space on image and customers' build speed. We don't have telemetry on GitHub Actions but in our understanding, 4 latest versions are most popular and caching them will cover ~90% use-cases. If customers need any specific old version and freeze patch version, they will have minor latency to install in runtime.
On Windows we use very tricky scheme:
I could be wrong, but in my understanding, if Ruby was built with --enable-shared flag, it will work only in the same directory where it was built. |
JFYI, all Windows builds used here are self contained, as they contain the dll's needed at runtime, and all the dll's are 'manifested' (their location is linked in the exe), so the 'builds' can be placed anywhere. None use the internal MSYS2 layout. |
At present, the naming/folder structure for Windows Ruby versions in the toolcache and this action are different. Is that a concern? Most scripts would rely on the Ruby version that 'Path' activates, but there could be scripts that are expecting the current layout? EDIT: I knew I had it somewhere in Actions workflows. This action uses Also, Windows Rubies are currently available in both 64 & 32 bit versions. I'm somewhat active in repos/gems testing & building with Windows, and I don't recall any requests for 32 bit versions. Have you had requests for them in the toolcache? |
@MSP-Greg , Yes, naming/structure could be concern.
actions/toolkit contains two methods: As I mentioned above, It could be something like:
As for the 32 bit versions, we have never seen requests for it so I think we should put x64 versions to the image cache and x32 can be downloaded in runtime with latency as currenty. |
Sorry I wasn't clearer. At present ruby/setup-ruby does not support 32 bit Windows versions. Or, It won't download them, and there's no way to tag the actions inputs that they're 'requested'. |
We have never seen requests for 32 bit Ruby on Windows. |
Is there a way to get the toolcache path, so we could extract directly to the right location? I think it would be better if we always have the Ruby prefix consistently under toolcache, whether or not we had to download, and whether or not it's self-hosted. Regarding having a build independent of the build directory, I found these configure options: |
I tried building Ruby with those flags, results at https://github.com/ruby/ruby-builder/releases/tag/load-relative One issue is that
Because the existing #!/bin/sh
# -*- ruby -*-
_=_\
=begin
bindir="${0%/*}"
exec "$bindir/ruby" "-x" "$0" "$@"
=end
#!/usr/bin/env ruby
#
# This file was generated by RubyGems.
#
# The application 'bundler' is installed as part of a gem, and
# this file is here to facilitate running it.
#
require 'rubygems' instead of #!/home/eregon/.rubies/ruby-2.7.2/bin/ruby
#
# This file was generated by RubyGems.
#
# The application 'bundler' is installed as part of a gem, and
# this file is here to facilitate running it.
#
require 'rubygems' And RubyGems doesn't seem to notice it's safe to override unfortunately. |
Interestingly #!/bin/sh
# -*- ruby -*-
# This file was generated by RubyGems.
#
# The application 'rake' is installed as part of a gem, and
# this file is here to facilitate running it.
#
_=_\
=begin
bindir="${0%/*}"
exec "$bindir/ruby" "-x" "$0" "$@"
=end
#!/usr/bin/env ruby
require 'rubygems'
|
Removing the |
There are various issues with having builds built in a separate directory than the runtime directory:
So I think building with Also, this seems unrelated, 2.2.10 fails the compile on macOS because |
@eregon thank you for investigation.
P.S. We will be overload next week due to a few upcoming releases. After that, we will try to rework Windows installation as MSP-Greg mentioned above and test binaries |
More of a note, but
It works on RubyInstaller 2.3 & 2.4 interestingly. I guess it doesn't matter too much since we did not receive a bug report about this, and |
…ble as safe to override" * This reverts commit 35f0af3. * It fails for some cases like Ruby <= 2.2 and on some RubyInstaller builds, see #98 (comment) * Most users will use Bundler to install rake, and that already works fine.
Builds with the hostedtoolcache prefix are at https://github.com/ruby/ruby-builder/releases/tag/toolcache The branch to test in this repo is https://github.com/ruby/setup-ruby/compare/toolcache @MSP-Greg the |
@MSP-Greg Actually, I can easily work around that, my initial condition for "use the tool cache for prefix" was |
I hate this (working on something else). Back when I started building mswin, I started on modifying the scripts I used for mingw. Never finished. The scripts for mingw do a bit of processing of the bin *.cmd scripts. The mswin *.cmd scripts currently have some hard coded paths in them that match the install folders, which I think match the previous location used here. I'm looking at it... I was writing while you were... I'll still take a look at it. It would be good to be consistent with the install location? |
@MSP-Greg I think it's fine as-is, it's actually not a good idea to have head builds under the hostedtoolcache as that can be persisted for self-hosted runners. Caching head builds would defeat the purpose of head builds. So, everything is fine now :) |
Cool. I think the hard coded paths are coming from the 'make' scripts, it's not my code. There is a script I can (easily) modify to remove them. But, mingw runs all it's test suites from install, I haven't done that yet for mswin. The hard coded paths may be needed since RbConfig is a bit strange when running make tests. I'll see. My memory isn't the best for these things... |
All builds are ready now at https://github.com/ruby/ruby-builder/releases/tag/toolcache. Test run at https://github.com/ruby/setup-ruby/actions/runs/352578630, I expect everything will be passing. |
I made a new release of It doesn't use the |
@eregon is correct. The standard install folder for MSYS2 is C:/msys64, and all Ruby versions >= 2.4 will find it when using gem & bundler. Off topic note: at present, compiling (outside of gem or bundler, typically via |
Hi there,
|
A couple of questions I wanted to discuss about communication process:
|
|
* Based on ruby/setup-ruby#98 (comment)
README updated: https://github.com/ruby/ruby-builder#readme |
Well phrased. 😊
They shouldn't, but sometimes security issues force changes that could be considered 'breaking', although often narrow in scope. Note that many repos test against major.minor Ruby versions (not pinned to 'patch'), so there will be CI testing run in the interval between new Ruby patch releases here and the update to them in the image toolcache. I think that made sense... |
That would be the responsibility of these users/customers and/or CRuby branch maintainers then. |
Sorry. My point was in reference to new Ruby patch versions having 'critical changes', and that because many repos do not pin to patch versions for CI, the new versions will have been tested on those repos before Actions adds them to the toolcache. Besides, and as you mentioned, Actions isn't responsible for changes in third party software, their only responsibility is to install new versions in a timely manner. |
Since I have you here, do you know who I could ping on actions/toolkit#632 / who are the maintainers of |
I think you can file an issue in https://github.com/actions/cache, since it is an open-source repo |
Thank you for updating README! |
@eregon - I pinged the team that owns that and they'll look as soon as they can. thanks! |
Hello @eregon , we have tested Ruby binaries from https://github.com/ruby/ruby-builder/releases/tag/toolcache on Ubuntu and MacOS images and can confirm that it works correctly 🚀 If you would like to review pull-requests:
We are planning to deploy the new images with this change in 1-2 weeks. After that you will be able to modify |
Note:
If you are going to find pre-cached versions manually via custom logic, you don't need to take care about this flag.
During image-generation, we just create this flag after unpacking to make sure that both GitHub Actions task and [Azure DevOps task] will be able to resolve version via actions/toolkit: https://github.com/actions/virtual-environments/pull/2084/files#diff-cd3a602dc8e0d7432bae1f80a68dd14d6397606db6474777a1b78f4129f8b345R42 |
Sorry for the late reply, I will take a deeper look at this soon.
Yes.
I'll do that, and I might try tc.find() and see how well it plays with the hardcoded paths (needed because building Ruby captures the paths where it will be installed ( The reason for not using cacheDir() is that would create an extra copy, I would need to first extract, then call cacheDir() which would mean having 2 copies on disk. Doing it manually, I can extract directly to the right directory. |
I did these changes, so now Changes: v1.56.0...v1.57.0 For self-hosted the toolcache directory is always persisted, is that correct? So that creates some difference. For instance both gem Gems installed via |
I think we can close this issue now, thanks for proposing it, the feedback and making quick progress on it. Since this action is now a strict superset (in that it also uses Rubies already in the toolcache, and even adds new Rubies there) of |
@eregon , thank you for changes on your side. As I understand, currently, your action should work pretty fast for pre-cached versions. Is it correct? Just want to make sure that customers will get the best experience with your action. @AlenaSviridenko and me will take care about plan for deprecation actions/setup-ruby. Thank you for your help! |
Thanks for confirming, indeed I missed I should wait up to 2 weeks from #98 (comment). I was actually hesitating to add an input (or a variable in the code) to control whether the toolcache is used (on by default), in case there is any issue. It might be useful in general.
Correct, yes, the There are extra steps (unless disabled by inputs): installing Bundler can take a couple seconds, and then (Linux example: https://github.com/ruby/setup-ruby/runs/1503878808?check_suite_focus=true#step:3:11) |
Was running CI elsewhere. All Rubies (on all three OS's) reported '0s' as the step time. No one can complain about that... |
We are the maintainers of actions/virtual-environments repo and owns the Hosted images for GitHub Actions and Azure DevOps. We are looking for some collaboration and improving UX of our customers in Ruby area.
Currently, we support two tasks related to Ruby: actions/setup-ruby (for GitHub Actions) and UseRuby (for Azure DevOps). Both tasks require pre-built binaries of Ruby for Linux and MacOS to be placed on images. We build them by ourselves from source code and then put to the images during image generation.
Although this approach works, we believe that a more proper way is to use official Ruby binaries through the https://github.com/ruby/setup-ruby task that is provided by you. You have more expertise in Ruby building and supporting. Actually, ruby/setup-ruby is already mentioned in starter workflow as a default way to use Ruby for GitHub Actions https://github.com/actions/starter-workflows/blob/main/ci/ruby.yml#L27.
The only benefit of actions/setup-ruby is that it understands when Ruby version is already pre-cached on image and doesn't download it again. Action ruby/setup-ruby always downloads specified version.
We think that it would be great if we can collaborate more and pre-cache binaries provided by ruby-builder to our images:
We have done investigation in order to investigate differences between ruby-builder and how we build Ruby from our side.
The main difference is that you build with shared libs support that means that binaries are bound to location where they were built and can’t be run from different location.
NOTE:
hostedtoolcache
is recommended approach to store cached tools on images.actions/toolkit
works with it by default. Also both actions/setup-ruby (GitHub Actions) and UseRuby (Azure DevOps) tasks are expecting to consume binaries from hostedtoolcache.Is it possible to change the ruby-builder directory and build binaries under
hostedtoolcache
? If so, we would be able to achieve better customers UX and get benefits that were mentioned above.Thank you,
Alyona.
cc: @maxim-lobanov, @alepauly, @sergey-akhalkov
The text was updated successfully, but these errors were encountered: