Add flag -testify.shuffle for suites #1413
Conversation
suite/suite.go
Outdated
| } else { | ||
| n, err = strconv.ParseInt(*shuffle, 10, 64) | ||
| if err != nil { | ||
| fmt.Fprintln(os.Stderr, `testing: -shuffle should be "off", "on", or a valid integer:`, err) |
There was a problem hiding this comment.
Better use t.Fatal or t.Skip.
suite/suite.go
Outdated
| return | ||
| } | ||
| } | ||
| fmt.Println("-testify.shuffle", n) |
suite/suite.go
Outdated
| @@ -17,6 +19,7 @@ import ( | |||
|
|
|||
| var allTestsFilter = func(_, _ string) (bool, error) { return true, nil } | |||
| var matchMethod = flag.String("testify.m", "", "regular expression to select tests of the testify suite to run") | |||
| var shuffle = flag.String("testify.shuffle", "off", "randomize the execution order of tests") | |||
There was a problem hiding this comment.
Add a reference to -test.shuffle:
randomize the execution order of tests in testify suites, like -test.shuffle
suite/suite_test.go
Outdated
|
|
||
| func TestShuffleOrderSuiteShuffleOn(t *testing.T) { | ||
| // To test this with testify.shuffle on we launch it in its own process | ||
| cmd := exec.Command("go", "test", "-v", "-race", "-run", "TestSuiteShuffleOrder", "-testify.shuffle", "2") |
There was a problem hiding this comment.
Add a comment mentionning that the test reliability relies on the algorithm used by package math/rand. If that algorithm changes the seed value may have to be adapted.
suite/suite_test.go
Outdated
|
|
||
| func TestShuffleOrderSuiteShuffleOn(t *testing.T) { | ||
| // To test this with testify.shuffle on we launch it in its own process | ||
| cmd := exec.Command("go", "test", "-v", "-race", "-run", "TestSuiteShuffleOrder", "-testify.shuffle", "2") |
There was a problem hiding this comment.
The test doesn't have to run in -race mode which is slower.
|
This seems a good feature to add. |
|
@dolmen thanks for review! I have fixed all the issues, if you have time, could you review it again? |
|
@dolmen I'm sorry to bother you, but any chance to be reviewed again? |
There was a problem hiding this comment.
This implementation requires users of package suite to use a specific kind of Suite. This is inconsistent with the shuffle flag of go test.
I think it would be better to instead change the default behavior of Suite. It would be appreciated if you could submit that alternative implementation as a separate PRs to let us choose the one to merge.
|
This patch conflicts with #1471 which will soon be merged. |
Summary
Add flag
-testify.shuffle=(off|on|N)to randomize execution of a suite's testsChanges
Add flag
-testify.shuffle=(off|on|N)Motivation
If go 1.17 was added ability to run test in random order.
I'd like to randomize tests inside suite.
Example usage (if applicable)
go test -testify.shuffle=on ./...