Disable parallelism on BootResourceCachingTest #30395
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
javiercn
reviewed
Feb 23, 2021
| @@ -18,6 +18,9 @@ | |||
|
|
|||
| namespace Microsoft.AspNetCore.Components.E2ETest.Tests | |||
| { | |||
| // Disabling parallelism for these tests because of flakiness | |||
| [CollectionDefinition(nameof(BootResourceCachingTest), DisableParallelization = true)] | |||
| [Collection(nameof(BootResourceCachingTest))] | |||
There was a problem hiding this comment.
Does this mean that this collection runs sequentially from any other collection and tests that are not in a collection?
There was a problem hiding this comment.
Yes, that is my understanding based on the Xunit docs.
That might be an option, although I'm not sure if the request will be emitted in that case anyways even if it's served from the cache. |
javiercn
approved these changes
Feb 23, 2021
SteveSandersonMS
deleted the
stevesa/disable-parallelism-on-bootresourcecachingtest
branch
This was referenced Feb 23, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tests in
BootResourceCachingTesthave been quarantined for a long time due to flakiness. Even after several attempts to fix this (#20154 and #27374), they continue to fail about 1 in 10 times. Somehow, the browser's cache occasionally doesn't include files we expect to have been cached, or it does include a file that should not yet be cached (suggesting that the test browser isn't as isolated as it should be). Maybe the browser doesn't always commit the cache updates to disk in a synchronous and blocking manner, so sometimes the expected cache updates haven't completed by the time we try to observe them.The nuclear solutions might include:
As one final attempt to avert those, this PR disables parallelism for those tests. It's quite possible that if the test environment isn't under heavy load at the time it runs these tests, then the browser's cache storage might behave predictably. We can see if this is reliable enough to justify unquarantining the tests.