-
-
Notifications
You must be signed in to change notification settings - Fork 650
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduction of 'MustPassRepeatedly' decorator #1051
Changes from 2 commits
053369f
d83abd9
f8d214e
5356dc4
d9b6aec
ee9b1cf
705d8c4
6abd43e
c729308
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2485,7 +2485,21 @@ One quick note on `--repeat`: when you invoke `ginkgo --repeat=N` Ginkgo will ru | |
|
||
Both `--until-it-fails` and `--repeat` help you identify flaky specs early. Doing so will help you debug flaky specs while the context that introduced them is fresh. | ||
|
||
However. There are times when the cost of preventing and/or debugging flaky specs simply is simply too high and specs simply need to be retried. While this should never be the primary way of dealing with flaky specs, Ginkgo is pragmatic about this reality and provides a mechanism for retrying specs. | ||
A more granular approach to repeating tests is by decorating individual subject or container nodes with the RepeatAttempts(N) decorator: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fantastic! thanks for taking the time to integrate this into the docs so well :) |
||
|
||
```go | ||
Describe("Storing books", func() { | ||
It("can save books to the central library", RepeatAttempts(3), func() { | ||
// this spec has been marked and will be retried up to 3 times | ||
}) | ||
|
||
It("can save books locally", func() { | ||
// this spec has not been marked and will not be retired | ||
}) | ||
}) | ||
``` | ||
|
||
However, There are times when the cost of preventing and/or debugging flaky specs simply is simply too high and specs simply need to be retried. While this should never be the primary way of dealing with flaky specs, Ginkgo is pragmatic about this reality and provides a mechanism for retrying specs. | ||
|
||
You can retry all specs in a suite via: | ||
|
||
|
@@ -2495,7 +2509,7 @@ ginkgo --flake-attempts=N | |
|
||
Now, when a spec fails Ginkgo will not automatically mark the suite as failed. Instead it will attempt to rerun the spec up to `N` times. If the spec succeeds during a retry, Ginkgo moves on and marks the suite as successful but reports that the spec needed to be retried. | ||
|
||
You can take a more granular approach by decorating individual subject nodes or container nodes as potentially flaky with the `FlakeAttempts(N)` decorator: | ||
A more granular approach is also provided for this functionality with the use of the `FlakeAttempts(N)` decorator: | ||
|
||
```go | ||
Describe("Storing books", func() { | ||
|
@@ -4797,7 +4811,7 @@ In addition to `Offset`, users can decorate nodes with a `types.CodeLocation`. | |
Passing a `types.CodeLocation` decorator in has the same semantics as passing `Offset` in: it only applies to the node in question. | ||
|
||
#### The FlakeAttempts Decorator | ||
The `FlakeAttempts(uint)` decorator applies container and subject nodes. It is an error to apply `FlakeAttempts` to a setup node. | ||
The `FlakeAttempts(uint)` decorator applies to container and subject nodes. It is an error to apply `FlakeAttempts` to a setup node. | ||
|
||
`FlakeAttempts` allows the user to flag specific tests or groups of tests as potentially flaky. Ginkgo will run tests up to the number of times specified in `FlakeAttempts` until they pass. For example: | ||
|
||
|
@@ -4825,6 +4839,35 @@ With this setup, `"is flaky"` and `"is also flaky"` will run up to 3 times. `"i | |
|
||
If `ginkgo --flake-attempts=N` is set the value passed in by the CLI will override all the decorated values. Every test will now run up to `N` times. | ||
|
||
#### The RepeatAttempts Decorator | ||
The `RepeatAttempts(uint)` decorator applies to container and subject nodes. It is an error to apply `RepeatAttempts` to a setup node. | ||
|
||
`RepeatAttempts` allows the user to flag specific tests or groups of tests for debbuging flaky tests. Ginkgo will run tests up to the number of times specified in `RepeatAttempts` until they fail. For example: | ||
|
||
```go | ||
Describe("repeated tests", RepeatAttempts(3), func() { | ||
It("is retried", func() { | ||
... | ||
}) | ||
|
||
It("is also retried", func() { | ||
... | ||
}) | ||
|
||
It("is retried even more", RepeatAttempts(5) func() { | ||
... | ||
}) | ||
|
||
It("is retried less", RepeatAttempts(1), func() { | ||
... | ||
}) | ||
}) | ||
``` | ||
|
||
With this setup, `"is retried"` and `"is also retried"` will run up to 3 times. `"is retried even more"` will run up to 5 times. `"is retried less"` will run only once. Note that if multiple `RepeatAttempts` appear in a spec's hierarchy, the most deeply nested `RepeatAttempts` wins. If multiple `RepeatAttempts` are passed into a given node, the last one wins. | ||
|
||
The `ginkgo --repeat=N` value passed in by the CLI has no relation with the `RepeatAttempts` decorator. If the `--repeat` CLI flag is used and a container or subject node also contains the `RepeatAttempts` decorator, then the test will run up to `N*R` times, where `N` is the values passed to the `--repeat` CLI flag and `R` is the value passed to the RepeatAttempts decorator. | ||
|
||
|
||
#### The SuppressProgressOutput Decorator | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -12,7 +12,8 @@ func TestDecorationsFixture(t *testing.T) { | |
RunSpecs(t, "DecorationsFixture Suite") | ||
} | ||
|
||
var count = 0 | ||
var countFlake = 0 | ||
// var countRepeat = 0 | ||
|
||
var _ = Describe("some decorated tests", func() { | ||
Describe("focused", Focus, func() { | ||
|
@@ -23,13 +24,21 @@ var _ = Describe("some decorated tests", func() { | |
|
||
}) | ||
|
||
It("passes eventually", func() { | ||
count += 1 | ||
if count < 3 { | ||
FIt("passes eventually", func() { | ||
countFlake += 1 | ||
if countFlake < 3 { | ||
Fail("fail") | ||
} | ||
}, FlakeAttempts(3)) | ||
|
||
// how to/should we test negative test cases? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Commented on an approach to this on the issue |
||
// FIt("fails eventually", func() { | ||
// countRepeat += 1 | ||
// if countRepeat >=3 { | ||
// Fail("fail") | ||
// } | ||
// }, RepeatAttempts(3)) | ||
|
||
It("focused it", Focus, func() { | ||
Ω(true).Should(BeTrue()) | ||
}) | ||
|
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
package invalid_decorations_flakeattempts_repeatattempts_test | ||
|
||
import ( | ||
"testing" | ||
|
||
. "github.com/onsi/ginkgo/v2" | ||
. "github.com/onsi/gomega" | ||
) | ||
|
||
func TestInvalidDecorations(t *testing.T) { | ||
RegisterFailHandler(Fail) | ||
RunSpecs(t, "InvalidDecorations Suite - RepeatAttempts and FlakeAttempts") | ||
} | ||
|
||
var _ = Describe("invalid decorators: repeatattempts and flakeattempts", FlakeAttempts(3), RepeatAttempts(3), func() { | ||
It("never runs", func() { | ||
|
||
}) | ||
}) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
package invalid_decorations_focused_pending_test | ||
|
||
import ( | ||
"testing" | ||
|
||
. "github.com/onsi/ginkgo/v2" | ||
. "github.com/onsi/gomega" | ||
) | ||
|
||
func TestInvalidDecorations(t *testing.T) { | ||
RegisterFailHandler(Fail) | ||
RunSpecs(t, "InvalidDecorations Suite - Focused and Pending") | ||
} | ||
|
||
var _ = Describe("invalid decorators: focused and pending", Focus, Pending, func() { | ||
It("never runs", func() { | ||
|
||
}) | ||
}) |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,6 +2,7 @@ package internal | |
|
||
import ( | ||
"fmt" | ||
"reflect" | ||
"time" | ||
|
||
"github.com/onsi/ginkgo/v2/types" | ||
|
@@ -118,6 +119,8 @@ func (g *group) initialReportForSpec(spec Spec) types.SpecReport { | |
ParallelProcess: g.suite.config.ParallelProcess, | ||
IsSerial: spec.Nodes.HasNodeMarkedSerial(), | ||
IsInOrderedContainer: !spec.Nodes.FirstNodeMarkedOrdered().IsZero(), | ||
IsFlakeAttempts: spec.Nodes.HasSetFlakeAttempts(), | ||
IsRepeatAttempts: spec.Nodes.HasSetRepeatAttempts(), | ||
} | ||
} | ||
|
||
|
@@ -299,16 +302,39 @@ func (g *group) run(specs Specs) { | |
|
||
g.suite.currentSpecReport.StartTime = time.Now() | ||
if !skip { | ||
maxAttempts := max(1, spec.FlakeAttempts()) | ||
|
||
var maxAttempts = 1 | ||
var multipleExecutionDecorator interface{} | ||
|
||
if g.suite.currentSpecReport.IsFlakeAttempts { | ||
multipleExecutionDecorator = reflect.TypeOf(FlakeAttempts(0)) | ||
maxAttempts = max(1, spec.FlakeAttempts()) | ||
} | ||
if g.suite.config.FlakeAttempts > 0 { | ||
multipleExecutionDecorator = reflect.TypeOf(FlakeAttempts(0)) | ||
maxAttempts = g.suite.config.FlakeAttempts | ||
g.suite.currentSpecReport.IsFlakeAttempts = true | ||
} | ||
|
||
if g.suite.currentSpecReport.IsRepeatAttempts { | ||
multipleExecutionDecorator = reflect.TypeOf(RepeatAttempts(0)) | ||
maxAttempts = max(1, spec.RepeatAttempts()) | ||
} else if g.suite.config.FlakeAttempts > 0 { | ||
//What should be the behavior when --flakeattempts is defined in CLI and the RepeatAttemps decorator is used? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. as mentioned in the issue - it should be ignored and only the explicit value set by the user in the decorator should be used There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Answered with suggestions in the comments |
||
} | ||
|
||
maxAttemptForLoop: | ||
for attempt := 0; attempt < maxAttempts; attempt++ { | ||
g.suite.currentSpecReport.NumAttempts = attempt + 1 | ||
g.suite.writer.Truncate() | ||
g.suite.outputInterceptor.StartInterceptingOutput() | ||
if attempt > 0 { | ||
fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Failed. Retrying...\n", attempt) | ||
switch t := multipleExecutionDecorator; { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. rather than tracking |
||
case t == reflect.TypeOf(FlakeAttempts(0)): | ||
fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Failed. Retrying...\n", attempt) | ||
case t == reflect.TypeOf(RepeatAttempts(0)): | ||
fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Passed. Retrying...\n", attempt) | ||
} | ||
} | ||
|
||
g.attemptSpec(attempt == maxAttempts-1, spec) | ||
|
@@ -318,9 +344,17 @@ func (g *group) run(specs Specs) { | |
g.suite.currentSpecReport.CapturedGinkgoWriterOutput += string(g.suite.writer.Bytes()) | ||
g.suite.currentSpecReport.CapturedStdOutErr += g.suite.outputInterceptor.StopInterceptingAndReturnOutput() | ||
|
||
if g.suite.currentSpecReport.State.Is(types.SpecStatePassed | types.SpecStateSkipped | types.SpecStateAborted | types.SpecStateInterrupted) { | ||
break | ||
switch t := multipleExecutionDecorator; { | ||
case t == reflect.TypeOf(FlakeAttempts(0)): | ||
if g.suite.currentSpecReport.State.Is(types.SpecStatePassed | types.SpecStateSkipped | types.SpecStateAborted | types.SpecStateInterrupted) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this looks right to me. |
||
break maxAttemptForLoop | ||
} | ||
case t == reflect.TypeOf(RepeatAttempts(0)): | ||
if g.suite.currentSpecReport.State.Is(types.SpecStateFailed | types.SpecStateSkipped | types.SpecStateAborted | types.SpecStateInterrupted) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we need There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What about There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes. in fact... I think you want There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In that case There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. sure that works too :) |
||
break maxAttemptForLoop | ||
} | ||
} | ||
|
||
} | ||
} | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned on the issue - I'd suggest
MustPassRepeatedly