Merge branch 'master' into add-array-methods-return-array-support

.gitattributes

@@ -1,11 +1,7 @@
 # Merge options
 
-# The changelog should use union merge on master, but not on release branches
 /CHANGELOG.md merge=union
 
-# spec/truffleruby.mspec should use union merge to make it easier to add specs to :next
-/spec/truffleruby.mspec merge=union
-
 # Rules for GitHub's Linguist language-classification system. We're abusing the
 # 'vendored' attribute to exclude files as a lot of this isn't really vendored,
 # and a whole lot of actually vendored code isn't listed! What we want to do is

.rubocop.yml

@@ -1,5 +1,5 @@
 AllCops:
-  TargetRubyVersion: 2.7
+  TargetRubyVersion: 3.0
   DisabledByDefault: true
   DisplayCopNames: true
   ExtraDetails: true

.ruby-version

@@ -1 +1 @@
-2.7.4
+3.0.2

3rd_party_licenses.txt

@@ -96,7 +96,7 @@ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF   ADVISED OF THE
 POSSIBILITY OF SUCH DAMAGE.
 ================================================================================
 
-Ruby 2.7.x
+Ruby 3.0.x
 
 According to Ruby's license terms, Ruby is licensed under a dual license.  
 Oracle elects the 2-clause BSD license.
@@ -111,10 +111,10 @@ Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions
 are met:
 1. Redistributions of source code must retain the above copyright
-notice, this list of conditions and the following disclaimer.
+   notice, this list of conditions and the following disclaimer.
 2. Redistributions in binary form must reproduce the above copyright
-notice, this list of conditions and the following disclaimer in the
-documentation and/or other materials provided with the distribution.
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
 
 THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 New features:
 
+* Updated to Ruby 3.0.2 (#2453).
 
 Bug fixes:
 
@@ -17,12 +18,25 @@ Compatibility:
 * Use compensated summation for `{Array,Enumerable}#sum` when floating point values are included.
 * `Module#attr_*` methods now return an array of method names (#2498, @gogainda).
 * Fixed `Socket#(local|remote)_address` to retrieve family and type from the file descriptor (#2444, @larskanis).
+* Add `Thread.ignore_deadlock` accessor (#2453).
+* Allow `Hash#transform_keys` to take a hash argument (@ccocchi, #2464).
+* Add `Enumerable#grep{_v}` optimization for `Regexp` (#2453).
+* Update `IO#write` to accept multiple arguments (#2501).
+* Do not warn when uninitialized instance variable is accessed (#2502, @andrykonchin).
+* Remove `TRUE`, `FALSE`, and `NIL` constants like CRuby 3.0 (#2505, @andrykonchin).
+* `Symbol#to_proc` now returns a lambda like in Ruby 3 (#2508, @andrykonchin).
+* `Kernel#lambda` now warns if called without a literal block (#2500, @andrykonchin).
+* Implement Hash#except (#2463, @wildmaples).
+* Remove special `$SAFE` global and related C API methods (#2453).
+* Assigning to a numbered parameter raises `SyntaxError` (#2506, @andrykonchin).
 
 Performance:
 
 * Regexp objects are now interned in a similar way to symbols.
-* Improve performance of regexps using POSIX bracket expressions (e.g., `[[:lower:]]`) matching against ASCII-only strings (#2447).
+* Improve performance of regexps using POSIX bracket expressions (e.g., `[[:lower:]]`) matching against ASCII-only strings (#2447, @nirvdrum).
 * `String#sub`, `sub!`, `gsub`, and `gsub!` have been refactored for better performance.
+* Don't allocate a `MatchData` object when `Regexp#match?` or `String#match?` is used (#2509, @nirvdrum).
+* Add `ENV.except` (#2507, @Strech).
 
 Changes:
 

README.md

@@ -117,7 +117,7 @@ See the [contributor workflow](doc/contributor/workflow.md) document if you wish
 We recommend that people trying TruffleRuby on their gems and applications [get in touch with us](#contact) for help.
 
 TruffleRuby can run Rails and is compatible with many gems, including C extensions.
-However, TruffleRuby is not 100% compatible with MRI 2.7 yet. Please report any compatibility issues you might find.
+However, TruffleRuby is not 100% compatible with MRI 3.0 yet. Please report any compatibility issues you might find.
 TruffleRuby [passes around 97% of ruby/spec](https://eregon.me/blog/2020/06/27/ruby-spec-compatibility-report.html),
 more than any other alternative Ruby implementation.
 

bench/chunky_png/chunky-decode-png-image-pass.rb

@@ -6,12 +6,6 @@
 # GNU General Public License version 2, or
 # GNU Lesser General Public License version 2.1.
 
-class Set
-end
-
-class SortedSet < Set
-end
-
 $LOAD_PATH << File.join(File.dirname(__FILE__), 'chunky_png', 'lib')
 
 if ENV['USE_CEXTS']

bench/chunky_png/chunky-encode-png-image-pass-to-stream.rb

@@ -6,12 +6,6 @@
 # GNU General Public License version 2, or
 # GNU Lesser General Public License version 2.1.
 
-class Set
-end
-
-class SortedSet < Set
-end
-
 $LOAD_PATH << File.join(File.dirname(__FILE__), 'chunky_png', 'lib')
 
 if ENV['USE_CEXTS']

bench/chunky_png/chunky_png/lib/chunky_png/palette.rb

@@ -11,7 +11,7 @@ module ChunkyPNG
   # to an explicit palette (stores as PLTE & tRNS chunks in a PNG file).
   #
   # @see ChunkyPNG::Color
-  class Palette < SortedSet
+  class Palette # Workaround SortedSet no longer being stdlib in Ruby 3, was `class Palette < SortedSet`
 
     # Builds a new palette given a set (Enumerable instance) of colors.
     #

ci.jsonnet

@@ -30,7 +30,7 @@ local common = (import "common.json");
 local part_definitions = {
   local jt = function(args) [["bin/jt"] + args],
   local mri_path = function(version) "/cm/shared/apps/ruby/" + version + "/bin/ruby",
-  local mri_version = "2.7.2",
+  local mri_version = "3.0.2",
 
   use: {
     common: {
@@ -318,6 +318,7 @@ local part_definitions = {
         "CHECK_LEAKS": "true",
       },
       run+: jt(["-u", mri_path(mri_version), "mspec", "spec/ruby"]) +
+            jt(["-u", mri_path("2.7.2"), "mspec", "spec/ruby"]) +
             jt(["-u", mri_path("2.6.6"), "mspec", "spec/ruby"]),
     },
 
@@ -367,20 +368,23 @@ local part_definitions = {
         ["mx", "sforceimports"], # clone the graal repo
         ["mx", "-p", "../graal/sulong", "build"],
         ["set-export", "TOOLCHAIN_PATH", ["mx", "-p", "../graal/sulong", "lli", "--print-toolchain-path"]],
-        ["set-export", "CC", "$TOOLCHAIN_PATH/clang"],
-        ["set-export", "CXX", "$TOOLCHAIN_PATH/clang++"],
-        # for finding libc++
-        ["set-export", "LD_LIBRARY_PATH", "$BUILD_DIR/graal/sulong/mxbuild/" + self.platform + "-" + self.arch + "/SULONG_HOME/native/lib:$LD_LIBRARY_PATH"],
       ],
       run+: [
-        ["ruby", "tool/generate-native-config.rb"],
+        ["env",
+          "LD_LIBRARY_PATH=$BUILD_DIR/graal/sulong/mxbuild/" + self.platform + "-" + self.arch + "/SULONG_HOME/native/lib:$LD_LIBRARY_PATH", # for finding libc++
+          "PATH=$TOOLCHAIN_PATH:$PATH",
+          "ruby", "tool/generate-native-config.rb"],
         ["cat", "src/main/java/org/truffleruby/platform/" + self.platform_name + "NativeConfiguration.java"],
+
+        # Uses the system compiler as using the toolchain for this does not work on macOS
+        ["tool/generate-config-header.sh"],
+        ["cat", "lib/cext/include/truffleruby/config_" + self.platform + "_" + self.arch + ".h"],
       ],
     },
 
     check_native_config: {
       is_after+:: ["$.run.generate_native_config"],
-      run+: jt(["check_native_configuration"]),
+      run+: jt(["check_native_configuration"]) + jt(["check_config_header"]),
     },
   },
 
@@ -706,9 +710,9 @@ local composition_environment = utils.add_inclusion_tracking(part_definitions, "
   manual_builds: {
     local shared = $.use.common + $.cap.manual + { timelimit: "15:00" },
 
-    "ruby-generate-native-config-linux": $.platform.linux + $.jdk.v11 + shared + $.run.generate_native_config,
-    "ruby-generate-native-config-linux-arm64": $.platform.linux_arm64 + $.jdk.v11 + shared + $.run.generate_native_config,
-    "ruby-generate-native-config-darwin": $.platform.darwin + $.jdk.v11 + shared + $.run.generate_native_config,
+    "ruby-generate-native-config-linux-amd64": $.platform.linux + $.jdk.v11 + shared + $.run.generate_native_config,
+    "ruby-generate-native-config-linux-aarch64": $.platform.linux_arm64 + $.jdk.v11 + shared + $.run.generate_native_config,
+    "ruby-generate-native-config-darwin-amd64": $.platform.darwin + $.jdk.v11 + shared + $.run.generate_native_config,
   },
 
   builds:

doc/contributor/updating-ruby.md

@@ -35,7 +35,7 @@ ruby-install ruby $VERSION
 # OR
 rm -rf ~/.rubies/ruby-$VERSION
 ruby-build $VERSION ~/.rubies/ruby-$VERSION
-ruby-install -r ~/tmp ruby $VERSION
+ruby-install --no-install-deps -r ~/tmp ruby $VERSION
 ```
 
 `ruby-build` does not keep the build directory
@@ -90,36 +90,9 @@ but not for files under `test/mri/tests/cext-ruby`.
 
 ## Update config_*.h files
 
-Configuration files must be regenerated from ruby for Linux and macOS
-and copied into `lib/cext/include/truffleruby`. In the MRI repository
-do the following:
-
-```
-ruby-build truffleruby-dev ~/.rubies/truffleruby-dev
-chruby truffleruby-dev
-
-graalvm_clang=$(ruby -e 'puts RbConfig::CONFIG["CC"]')
-
-autoconf
-CC=$graalvm_clang ./configure
-```
-
-The output of configure should report that it has created or updated a
-config.h file. For example
-
-```
-.ext/include/x86_64-linux/ruby/config.h updated
-```
-
-You will need to copy that file to
-`lib/cext/include/truffleruby/config_linux.h` or
-`lib/cext/include/truffleruby/config_darwin.h`.
-
-After that you should clean your MRI source repository with:
-
-```bash
-git clean -Xdf
-```
+Configuration files must be regenerated for each platform with `tool/generate-config-header.sh`.
+Adapt the Ruby versions in that script.
+Then run all the `ruby-generate-native-config-*` jobs in CI and copy their output.
 
 ## Update libraries from third-party repos
 
@@ -131,6 +104,8 @@ rm -rf lib/json/lib
 cp -R ../../json/lib lib/json
 ```
 
+Also reapply our changes to json files, by looking with `git log -p lib/json`.
+
 ## Updating default and bundled gems
 
 You need a clean install (e.g., no extra gems installed) of MRI for this
@@ -149,7 +124,7 @@ cd $TRUFFLERUBY
 ruby tool/patch-default-gemspecs.rb
 ```
 
-## Updating bin/ executables
+## Updating exe/ executables
 
 ```
 rm -rf exe
@@ -158,6 +133,8 @@ rm -f exe/ruby
 ruby tool/patch_launchers.rb
 ```
 
+Also update the list of `provided_executables` in `mx_truffleruby.py` if some launchers were added or removed.
+
 ## Make other changes
 
 In a separate commit, update all of these:

doc/contributor/workflow.md

@@ -288,17 +288,17 @@ exception.
 When the spec is fixed the `untag` command will remove the tag and you can
 commit the fix and the removal of the tag.
 
-## Running specs for Ruby 3.0 features
+## Running specs for Ruby 3.1 features
 
-TruffleRuby currently targets Ruby 2.7. However, we welcome pull requests for
-Ruby 3.0 features as long as they don't conflict significantly with
-Ruby 2.7 semantics.
+TruffleRuby currently targets Ruby 3.0. However, we welcome pull requests for
+Ruby 3.1 features as long as they don't conflict significantly with
+Ruby 3.0 semantics.
 
-It is possible to run specs for Ruby 3.0 features by setting
+It is possible to run specs for Ruby 3.1 features by setting
 `PRETEND_RUBY_VERSION`:
 
 ```bash
-PRETEND_RUBY_VERSION=3.0.2 jt test spec/ruby/.../some_spec.rb
+PRETEND_RUBY_VERSION=3.1.0 jt test spec/ruby/.../some_spec.rb
 ```
 
 This also works for `jt tag`/`jt untag`.

doc/legal/legal.md

@@ -19,7 +19,7 @@ See `epl-2.0.txt`, `gpl-2.txt`, `lgpl-2.1.txt`.
 ## MRI
 
 The standard implementation of Ruby is MRI. TruffleRuby contains code from MRI
-version 2.7.4, including:
+version 3.0.2, including:
 
 * the standard library in `lib/mri`, 
 * Ruby C extension API in `lib/cext/include` and `src/main/c/cext`, 

doc/legal/ruby-bsdl.txt

@@ -4,10 +4,10 @@ Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions
 are met:
 1. Redistributions of source code must retain the above copyright
-notice, this list of conditions and the following disclaimer.
+   notice, this list of conditions and the following disclaimer.
 2. Redistributions in binary form must reproduce the above copyright
-notice, this list of conditions and the following disclaimer in the
-documentation and/or other materials provided with the distribution.
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
 
 THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE

doc/user/compatibility.md

@@ -7,7 +7,7 @@ permalink: /reference-manual/ruby/Compatibility/
 # Compatibility
 
 TruffleRuby aims to be fully compatible with the standard implementation of
-Ruby, MRI, version 2.7.4, [including C extensions](#c-extension-compatibility).
+Ruby, MRI, version 3.0.2, [including C extensions](#c-extension-compatibility).
 TruffleRuby is still in development, so it is not 100% compatible yet.
 
 TruffleRuby can run Rails and is compatible with many gems, including C extensions.
@@ -70,13 +70,6 @@ TruffleRuby provides its own backend implementation for the `ffi` gem, similar t
 This should be completely transparent and behave the same as on MRI.
 The implementation should be fairly complete and passes all the specs of the `ffi` gem except for some rarely-used corner cases.
 
-### Safe levels
-
-`$SAFE` and `Thread#safe_level` are `0` and no other levels are implemented.
-Trying to use level `1` will raise a `SecurityError`.
-Other levels will raise `ArgumentError` as in standard Ruby.
-See the [security notes](known-cves.md) for more explanation on this.
-
 ### Internal MRI functionality
 
 `RubyVM` is not intended for users and is not implemented.

doc/user/known-cves.md

@@ -9,20 +9,6 @@ permalink: /reference-manual/ruby/KnownCVEs/
 Please report security vulnerabilities via the process outlined in the [reporting vulnerabilities guide](https://www.oracle.com/corporate/security-practices/assurance/vulnerability/reporting.html).
 Specific guidelines for reporting security issues of the GraalVM project, including TruffleRuby, can be found in the [SECURITY file](https://github.com/oracle/truffleruby/blob/master/SECURITY.md).
 
-## Unimplemented Security Features
-
-Ruby's `$SAFE` feature adds additional checks regarding how tainted data is used, but they are not always correct.
-The checks for tainted data are likewise inconsistent and their implementation has been the subject of many vulnerabilities,
-including regressions of previously fixed vulnerabilities, as detailed below.
-Consensus in the Ruby community is that `$SAFE` is a broken security feature that does not provide genuine safety and it will eventually be removed.
-
-For these reasons, TruffleRuby will not let you enable the `$SAFE` feature.
-This does not disable a security feature that would normally be enabled - it prevents you from using a broken security feature.
-
-This has the effect that `$SAFE` and `Thread#safe_level` are `0` and no other levels are implemented.
-Trying to use level `1` will raise a `SecurityError`.
-Other levels will raise an `ArgumentError` as in standard Ruby.
-
 ## MRI Vulnerabilities
 
 Vulnerabilities reported against MRI may apply to the design of Ruby or to code that we share with MRI.

doc/user/options.md

@@ -40,6 +40,8 @@ Usage: truffleruby [switches] [--] [programfile] [arguments]
   --verbose       turn on verbose mode and disable script from stdin
   --version       print the version number, then exit
   --help          show this message, -h for short message
+  --backtrace-limit=num
+                  limit the maximum length of backtrace
 
 Features:
   gems            rubygems (default: enabled)