-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unstable duration of tests running in ADO #12392
Comments
cc @echesakov @dotnet/jit-contrib |
@sandreenko Just FYI, in Helix we are also using -parallel collections |
I have checked some simple tests without threads/GC and they show the same instability. I think this should be in 3.0. |
regressions\beta2\437017 frequently times out on Ubuntu.1404.Arm32 TestResults
| where Method == '_regressions_beta2_437017_437017_sh' and Result == 'Fail' and Duration > 600
| join kind=inner (Jobs) on JobId
| extend Configuration=extract('"configuration":"([^"]+)"', 1, Properties) shows 24 records. 23 of them are Checked-no_tiered_compilation and one is Checked-jitstress1 |
_tailcall_Desktop__il_relthread_race__il_relthread_race_sh frequently times out in multiple configurations (Checked, Checked-no_tiered_compilation, Checked-gcstress0xc, and Checked-jitstress2_jitstressregs2 |
beta2_437044 is just an unstable threading test. I will look at other failures soon. |
More data about tests\src\JIT\Methodical\tailcall\Desktop_il_relthread-race arm32 Linux shows min = 2.2, max = 7, factor 3; x64 Windows min = 2, max = 256, factor 128; x64 OSX min = 2, max = 32, factor 16; the test uses 4 threads and one mutex. |
If you ask Kusto a query like this:
it will show you data like this:
that means that we have a problem not with a particular set of tests or platform or test mode but it is a general issue that affects the whole testing. |
That query indicates that a very few set of "huge tests" have widely varying executions times. _regress_ddb |
:mips-interest |
I am going to reenable the disabled tests and see if the issue still reproduces. There were many infrastructure changes since it was open, so it could have been fixed already. In the past, I was trying to make a change to allocate 2 cores per running test in helix but it did not work well (https://github.com/dotnet/coreclr/pull/24749/files). On some machines I was getting an incorrect number of cores (like 2 cores VM on 64 cores physical arm machine was giving me 32, instead of 1), on others, it still was hitting timeouts, but less frequently, because the tests that hit them did not allocate 2 cores, they were allocating dozen each. The big downside of that change was that on average a test run was taking longer (like ~20% slower). |
As I see from the recent GCStress runs it is still an issue: JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_r/AdvSimd_r.sh |
@sandreenko I am working on splitting those up #37282 |
Could you please try to reenabled tests from https://github.com/dotnet/runtime/pull/37530/files and close this issue in your PR? |
Sure |
Addressed |
We hit timeouts with GCStress on tests like
baseservices/threading/regressions/beta2/437017
orJIT/HardwareIntrinsics/X86/Avx2/Avx2_ro
from time to time. The timeout for GCStress is set to 1 hour per test. Kusto results show that sometimes these tests execution takes 20 minutes, sometimes 40, some fail with 1 hour timeout.Duratation for beta2_437044_437044_sh Ubuntu.1604 gcstress0x3:
The similar picture is for other tests, for example
beta2_437044
onx64 WIndows.10 minopts
:So max is 35, min is 3, that doesn't look right.
(queue for this:
`)
We have seen a similar problem with Jenkins but it was because we used XUnit
ParallelRun=all
and it was creating more threads than CPU cores and it was fixed by changing it to XUnitParallelRun=collections
(#22746). With ADO our understanding was that we run only one assembly on one machine so there were no difference betweenall
andcollections
.The text was updated successfully, but these errors were encountered: