System.Diagnostics.Tests.DiagnosticSourceTest.AllSubscriberStress blocking test runs on ARM64

[wfurt]

We get catastrophic failure in CI.

~/dotnetbuild/work/6271876f-6ed7-4469-828e-f8c62abc33df/Work/4b9b88e6-1a8f-468f-8386-88f45204d46c/Exec ~/dotnetbuild/work/6271876f-6ed7-4469-828e-f8c62abc33df/Work/4b9b88e6-1a8f-468f-8386-88f45204d46c/Exec
�[37m  Discovering: System.Diagnostics.DiagnosticSource.Tests (method display = ClassAndMethod, method display options = None)
�[m�[37m  Discovered:  System.Diagnostics.DiagnosticSource.Tests (found 50 test cases)
�[m�[37m  Starting:    System.Diagnostics.DiagnosticSource.Tests (parallel test collections = on, max threads = 2)
�[m�[33;1m   System.Diagnostics.DiagnosticSource.Tests: [Long Running Test] 'System.Diagnostics.Tests.DiagnosticSourceTest.AllSubscriberStress', Elapsed: 00:02:36
�[m�[33;1m   System.Diagnostics.DiagnosticSource.Tests: [Long Running Test] 'System.Diagnostics.Tests.DiagnosticSourceTest.AllSubscriberStress', Elapsed: 00:04:38
�[m�[33;1m   System.Diagnostics.DiagnosticSource.Tests: [Long Running Test] 'System.Diagnostics.Tests.DiagnosticSourceTest.AllSubscriberStress', Elapsed: 00:06:38
�[m�[33;1m   System.Diagnostics.DiagnosticSource.Tests: [Long Running Test] 'System.Diagnostics.Tests.DiagnosticSourceTest.AllSubscriberStress', Elapsed: 00:08:38
�[mKilled

[stephentoub]

Most likely hitting the 10 minute timeout.

[wfurt]

I'll try to run it without limit and see how long does it take.
For now, I was primarily trying to add tracking for tests failing daily in CI.

[wfurt]

I did run on container similar to CI and it took 34 minutes for this test set alone to finish. (outerloop)

=== TEST EXECUTION SUMMARY ===
   System.Diagnostics.DiagnosticSource.Tests  Total: 54, Errors: 0, Failed: 0, Skipped: 0, Time: 2050.368s

cc: @safern @danmosemsft

I'm not sure what is best strategy here: either disabling biggest offenders (or moving them to stress category) or increasing timeout quite a bit. Second option will of course have impact on PR as it will take much more time to get feedback.

[safern]

Note that the engineering team just let me know last week that Ubuntu ARM tests are now going to run within a docker container limited to 2 cores only just as official builds do. I guess we should also, ask them to provide more cores, since the servers have 48 cores, specially as this cores are slower than intel ones -- at least get 4 cores for each container would be nice.

[stephentoub]

Note that the engineering team just let me know last week that Ubuntu ARM tests are now going to run within a docker container limited to 2 cores only just as official builds do.

Do you know when this change is taking effect?

[safern]

Do you know when this change is taking effect?

Just asked and it happened some time ago today. I've been looking at some random PR failures due to this. Working with the engineering team on getting them fixed.

cc: @ulisesh

[wfurt]

Did we end-up using 4 cores or did we get more? I can re-run tests to get new timing

[safern]

Did we end-up using 4 cores or did we get more? I can re-run tests to get new timing

Not yet. I just spoke with @Chrisboh and he said that they can probably do that, but first they want to get ARM stuff stable and afterwards do this kind of changes.

[josalem]

Is this still an issue? The last comments on this are from over 2 years ago at this point.

[hoyosjs]

Test is still disabled in outerloop

[noahfalk]

Trial by fire, I re-enabled it to see what happens.