MAUI tests fail frequently and aren't easily diagnosable

[JeremyKuhne]

These are an attempt to shim in the old internal tests (note that MAUI here is an old internal test framework). The old tests worked quite differently and don't fit in well with the current test infrastructure. We're getting frequent intermittent failures like so:

System.Windows.Forms.Maui.IntegrationTests.WinformsMauiButtonTests.MauiButtonTest(scenarioName: \"Hotkey_Fires_OnClick\")

Assert.Equal() Failure\r\n          ↓ (pos 0)\r\nExpected: Pass\r\nActual:   Fail\r\n          ↑ (pos 0)

Stack trace
   at System.Windows.Forms.Maui.IntegrationTests.MauiTestHelper.ValidateScenarioPassed(String projectName, String scenarioName) in /_/src/System.Windows.Forms/tests/IntegrationTests/System.Windows.Forms.Maui.IntegrationTests/MauiTestHelper.cs:line 90
   at System.Windows.Forms.Maui.IntegrationTests.WinformsMauiButtonTests.MauiButtonTest(String scenarioName) in /_/src/System.Windows.Forms/tests/IntegrationTests/System.Windows.Forms.Maui.IntegrationTests/WinformsMauiButtonTests.cs:line 35

This doesn't really help you much and it isn't obvious as to what you can do to move forward. In addition, the MAUI infrastructure sources aren't available so it is difficult to understand how the whole system works.

My recommendation (and current POR):

1. Disable the MAUI tests for now
2. Make them more xUnit-like and have any shim code in the repo
3. If we can make the tests plug in as a "normal" xUnit test then re-enable them for the CI, otherwise leave them for manual runs

cc: @AdamYoblick, @RussKie

Failed with #3600

[merriemcgaw]

We ought to rethink our approach to those old tests. I think now, thanks to our amazing contributors, we've got a lot more coverage and don't necessarily need the old school style tests. I'll work with @dreddy-work , @Olina-Zhang and a few others to make sure we have the scenarios covered in our new infrastructure, or rewritten from scratch. It just seems like the right approach here. Go ahead and disable them for the time being, I don't think we're losing much. @Olina-Zhang just to be safe, can you run the MAUI tests once or twice per-preview internally?

[AdamYoblick]

I agree, these tests are pretty fragile. Merrie asked me a while back to see if I could add the manual MAUI runs to our test automation, so that's what all the shims are. 😄

[Olina-Zhang]

@merriemcgaw, So far we run all migrated Maui cases in our every regular .net 5.0 SDK testing, once a week, these cases can ensure no big regression issue entering.

[weltkante]

I don't think the xunit tests are covering anything regarding input infrastructure at all? The problem is tests doing input coverage can't be multithreaded (input/focus is a global resource) so xunit is not really a good fit. We've actively been moving away some of the flaky tests from xunit too, either to RemoteExecutor or MAUI (RemoteExecutor currently being even more broken than MAUI)

[merriemcgaw]

Given these scenarios are part of the manual suite of tests, I think we can disable the most flaky ones. We should have a separate effort to refactor and improve the integration test bed. In the interim we don't want them in CI.

[RussKie]

@SergeySmirnov-Akvelon as discussed please disable the integration tests for #3600

diff --git a/eng/ci.yml b/eng/ci.yml
index 829f86639..d21577582 100644
--- a/eng/ci.yml
+++ b/eng/ci.yml
@@ -125,13 +125,14 @@ stages:
         # Run Integration Tests
         # Tests are run with /m:1 to avoid parallelism across different assemblies which can lead to
         # UI race conditions
-        - script: eng\cibuild.cmd
-            -integrationTest
-            -configuration $(_BuildConfig)
-            $(_OfficialBuildIdArgs)
-            /bl:$(BUILD.SOURCESDIRECTORY)\artifacts\log\$(_BuildConfig)\IntegrationTest.binlog
-            /m:1
-          displayName: Run Integration Tests
+        # Flaky tests, see: https://github.com/dotnet/winforms/issues/3617
+        # - script: eng\cibuild.cmd
+        #     -integrationTest
+        #     -configuration $(_BuildConfig)
+        #     $(_OfficialBuildIdArgs)
+        #     /bl:$(BUILD.SOURCESDIRECTORY)\artifacts\log\$(_BuildConfig)\IntegrationTest.binlog
+        #     /m:1
+        #   displayName: Run Integration Tests
 
         # Create Nuget package, sign, and publish
         - script: eng\cibuild.cmd

[JeremyKuhne]

Got the same failure again:

System.Windows.Forms.Maui.IntegrationTests.WinformsMauiButtonTests.MauiButtonTest(scenarioName: \"Hotkey_Fires_OnClick\")

Error message
Assert.Equal() Failure\r\n          ↓ (pos 0)\r\nExpected: Pass\r\nActual:   Fail\r\n          ↑ (pos 0)

Stack trace
   at System.Windows.Forms.Maui.IntegrationTests.MauiTestHelper.ValidateScenarioPassed(String projectName, String scenarioName) in F:\workspace\_work\1\s\src\System.Windows.Forms\tests\IntegrationTests\System.Windows.Forms.Maui.IntegrationTests\MauiTestHelper.cs:line 89
   at System.Windows.Forms.Maui.IntegrationTests.WinformsMauiButtonTests.MauiButtonTest(String scenarioName) in F:\workspace\_work\1\s\src\System.Windows.Forms\tests\IntegrationTests\System.Windows.Forms.Maui.IntegrationTests\WinformsMauiButtonTests.cs:line 34

[RussKie]

Sergey was looking into this on Fri.

[RussKie]

Keeping it open until we migrate all MAUI tests into the new project.