Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve testing infra #442

Merged
merged 3 commits into from
Nov 15, 2022
Merged

Improve testing infra #442

merged 3 commits into from
Nov 15, 2022

Conversation

st0012
Copy link
Member

@st0012 st0012 commented Nov 15, 2022

There's a huge difference between how tests are run in this repo and after being synced to Ruby CI. And that causes Ruby CI failure like this not being detected in the original PR #441. I think the rule of thumb is to have every test file self-contained, which is hard to verify with the current rake test.

So in this PR I added test_in_isolation task, which runs each test file in isolation. With this task, we would be able to catch the above failure before the PR is merged.

With this new task, I also caught another undiscovered error:

=========================================================================================================================================================================================================================================================================================================================================================================================================================================
Error: test_raise_exception_with_different_encoding_containing_invalid_byte_sequence(TestIRB::TestRaiseNoBacktraceException):
  NoMethodError: undefined method `mktmpdir' for Dir:Class

        Dir.mktmpdir("test_irb_raise_no_backtrace_exception_#{$$}") do |tmpdir|
           ^^^^^^^^^
  Did you mean?  mkdir
/Users/hung-wulo/src/github.com/ruby/irb/test/irb/test_raise_no_backtrace_exception.rb:28:in `test_raise_exception_with_different_encoding_containing_invalid_byte_sequence'
     25:
     26:     def test_raise_exception_with_different_encoding_containing_invalid_byte_sequence
     27:       backup_home = ENV["HOME"]
  => 28:       Dir.mktmpdir("test_irb_raise_no_backtrace_exception_#{$$}") do |tmpdir|
     29:         ENV["HOME"] = tmpdir
     30:
     31:         bundle_exec = ENV.key?('BUNDLE_GEMFILE') ? ['-rbundler/setup'] : []
=========================================================================================================================================================================================================================================================================================================================================================================================================================================
.
Finished in 0.477298 seconds.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3 tests, 6 assertions, 0 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
66.6667% passed
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6.29 tests/s, 12.57 assertions/s
=====================================================================================================================================================================================
Test 'test/irb/test_raise_no_backtrace_exception.rb' failed when being executed in isolation. Please make sure 'rake test TEST=test/irb/test_raise_no_backtrace_exception.rb' passes.
=====================================================================================================================================================================================

Which is now fixed in the last commit.

.github/workflows/test.yml Outdated Show resolved Hide resolved
Rakefile Outdated Show resolved Hide resolved
This simulates how tests are run in Ruby CI and can detect some issues
before they're merged and break Ruby CI later.
@st0012 st0012 force-pushed the improve-testing-infra branch from 8381bef to 195fa67 Compare November 15, 2022 17:18
@st0012 st0012 requested a review from k0kubun November 15, 2022 17:25
Copy link
Member

@k0kubun k0kubun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@k0kubun k0kubun merged commit 51f23c5 into ruby:master Nov 15, 2022
@st0012 st0012 deleted the improve-testing-infra branch November 15, 2022 17:26
matzbot pushed a commit to ruby/ruby that referenced this pull request Nov 15, 2022
(ruby/irb#442)

* Add test_in_isolation task to run tests in isolation

This simulates how tests are run in Ruby CI and can detect some issues
before they're merged and break Ruby CI later.

* Run test_in_isolation task on CI

* Fix TestRaiseNoBacktraceException's setup

ruby/irb@51f23c58b0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants