-
Notifications
You must be signed in to change notification settings - Fork 10k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mark tests flaky instead of skipped #8727
Conversation
@@ -37,7 +37,8 @@ private RedisHubLifetimeManager<MyHub> CreateLifetimeManager(TestRedisServer ser | |||
}, NullLogger<DefaultHubProtocolResolver>.Instance)); | |||
} | |||
|
|||
[Fact(Skip = "https://github.com/aspnet/SignalR/issues/3088")] | |||
[Fact] | |||
[Flaky("https://github.com/aspnet/SignalR/issues/3088")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one isn't flaky, it's just broken. Do we want to keep it as skip or no?
@@ -77,7 +77,8 @@ public async Task AppOfflineDroppedWhileSiteFailedToStartInShim_AppOfflineServed | |||
DeletePublishOutput(deploymentResult); | |||
} | |||
|
|||
[ConditionalFact(Skip = "https://github.com/aspnet/IISIntegration/issues/933")] | |||
[ConditionalFact] | |||
[Flaky("https://github.com/aspnet/IISIntegration/issues/933")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Delete this test. There has been no action on fixing https://github.com/dotnet/core-setup/issues/4646 and it isn't our problem.
@@ -37,7 +37,8 @@ public static IEnumerable<object[]> InvalidTestVariants | |||
from s in new string[] { (_minPort - 1).ToString(), (_maxPort + 1).ToString(), "noninteger" } | |||
select new object[] { v, s }; | |||
|
|||
[ConditionalTheory(Skip = "https://github.com/aspnet/AspNetCore/issues/8329")] | |||
[ConditionalTheory] | |||
[Flaky("https://github.com/aspnet/AspNetCore/issues/8329")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't enable these tests. We need an SDK upgrade to enable these
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -26,7 +26,8 @@ public static TestMatrix TestVariants | |||
.WithTfms(Tfm.NetCoreApp30) | |||
.WithAllApplicationTypes(); | |||
|
|||
[ConditionalTheory(Skip = "https://github.com/aspnet/AspNetCore/issues/8329")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These too.
@@ -173,7 +173,7 @@ public async Task WriterThrowsCancelledException() | |||
} | |||
} | |||
|
|||
[ConditionalFact] | |||
[Fact] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These need to be ConditionalFacts still. There are conditions applied on the entire assembly to check whether we are running on a machine that supports this test.
@@ -29,7 +29,8 @@ public static TestMatrix TestVariants | |||
=> TestMatrix.ForServers(DeployerSelector.ServerType) | |||
.WithTfms(Tfm.NetCoreApp30); | |||
|
|||
[ConditionalTheory(Skip = "https://github.com/aspnet/AspNetCore/issues/8329")] | |||
[ConditionalTheory] | |||
[Flaky("https://github.com/aspnet/AspNetCore/issues/8329")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These three. At this point I think you can just undo all changes to the src/Servers/IIS subfolder.
How many of these fail every time? Not much point in marking them as flaky. |
I think we should review all the ones we know about and keep the “consistently broken” ones as Skipped (or delete them). There are bound to be ones we don’t know/remember well though. Might be worth doing a survey of the flaky tests in, say, a week and for any of them that have never passed in that time, switch them back to fully skipped. Thoughts? Sent with GitHawk |
I was going to go with "unskip everything and let it make noise" but I'm fine with leaving consistently failing ones out. |
🆙📅 I reviewed the issues and left the ones that claim to fail every time as skipped. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good from SignalR and dotnet-watch side
a36102c
to
d203f4c
Compare
091e2ba
to
a5ba79e
Compare
src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2TimeoutTests.cs
Show resolved
Hide resolved
4fd7e88
to
00b8554
Compare
This should make sure there's plenty to hear about once we finish https://github.com/aspnet/AspNetCore-Internal/issues/1948.