Testing: Skip flaky e2e tests #30344
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
youknowriad
added
the
[Type] Automated Testing
youknowriad
commented
Mar 29, 2021
| @@ -23,7 +23,7 @@ class CustomEnvironment extends JSDOMEnvironment { | |||
| } | |||
|
|
|||
| async teardown() { | |||
| await this.global.editorPage.stopDriver(); | |||
| await this.global.editorPage?.stopDriver(); | |||
There was a problem hiding this comment.
I'm not sure this is a good fix, mainly a hunch to see if it fixes the instability.
There was a problem hiding this comment.
stopDriver being undefined is usually the result of a failure with something else. But I understand that it causes confusion to see this error. I think it should be fine to have this change.
|
Size Change: 0 B Total Size: 1.42 MB ℹ️ View Unchanged
|
|
The first iOS one seems to be related to GitHub servers returning a 500 error during an artifact upload. We did a few fixes (1, 2, 3) recently with iOS tests and they should be quite stable. The second Android one is indeed flakiness, but We're currently working on adding a different kind of e2e test for mobile here, some changes we introduce there should get rid of this Android flakiness. And this time we're making sure it's very robust before merging it by running them consecutively multiple times. |
6 tasks
|
Thanks for looking at this. Flaky tests lead to apathy in regarding failures. It's better to quarantine tests that have hiccups and revisit them |
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To have more confidence in our test suite, we need to maintain a stable suite and avoid flaky tests as much as we can. So we should fix them ideally or skip them until they're made stable.
This PR takes a look at the latest commits in master and skips the tests that are flaky.