Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Formats.Nrbf.Tests timeouts #110285

Closed
EgorBo opened this issue Nov 30, 2024 · 5 comments · Fixed by #110550
Closed

System.Formats.Nrbf.Tests timeouts #110285

EgorBo opened this issue Nov 30, 2024 · 5 comments · Fixed by #110550
Assignees
@adamsitnik
Labels
arch-x86 area-System.Formats.Nrbf blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' in-pr There is an active PR which will close this issue when it is merged Known Build Error Use this to report build issues in the .NET Helix tab os-windows
Milestone
10.0.0

Comments

@EgorBo
Copy link
Member

EgorBo commented Nov 30, 2024 •
edited by build-analysis bot
Loading

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=882740
Build error leg or test failing:

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "'System.Formats.Nrbf.Tests' END OF WORK ITEM LOG: Command timed out, and was killed",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=882740
Error message validated: ['System.Formats.Nrbf.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/30/2024 4:39:23 PM UTC

Report

Build Definition Test Pull Request
892948 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110554
892884 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110342
892786 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #109826
892767 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110198
892724 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110589
892643 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110619
891862 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110322
892590 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110617
892519 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110330
892495 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110613
892403 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110477
892370 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110612
892361 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110610
890559 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110545
892224 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110477
892221 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110567
892128 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110477
892120 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110589
892104 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110602
891876 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110477
891834 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110554
891774 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110342
891768 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110548
891637 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110589
891613 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110574
891601 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110586
891157 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110566
891492 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110579
891484 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110330
891469 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110576
891445 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #108750
891442 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110197
891430 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110574
891042 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110396
890210 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110531
890941 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110353
890938 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110105
890782 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110557
890774 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #109136
890349 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110494
890662 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110465
890658 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110105
890588 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110540
890581 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110542
890256 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110481
890562 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110551
890471 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110548
890465 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110520
890436 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110518
890418 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110322
890379 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110516
890306 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #109378
890289 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110501
890246 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110536
890226 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110535
890192 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110471
889958 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110528
889951 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110308
889230 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110484
889766 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110522
889597 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110518
889586 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110516
889581 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110501
889212 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110506
889532 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110513
889527 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110481
889444 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110465
889401 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110501
889383 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110501
889250 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110342
889192 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110342
889140 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110033
889095 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110503
888986 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #104906
2597163 dotnet-runtime System.Formats.Nrbf.Tests.WorkItemExecution
888810 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110485
888893 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110342
888474 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110449
888803 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110481
888785 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110486
888660 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110494
888709 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110033
888706 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110465
888617 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110448
888589 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110491
888566 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110471
888563 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110484
888561 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110476
888553 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110486
888537 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110064
888494 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110477
888424 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #104906
888417 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110484
888408 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110483
888363 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110481
888292 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #108941
888265 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #104906
888192 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110119
888153 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110472
888145 dotnet/runtime System.Formats.Nrbf.Tests.WorkItemExecution #110471
Displaying 100 of 211 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
19 124 211
@EgorBo EgorBo added the Known Build Error Use this to report build issues in the .NET Helix tab label Nov 30, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Nov 30, 2024
@EgorBo
Copy link
Member Author

EgorBo commented Nov 30, 2024

Could be #110194 ? cc @adamsitnik

@am11 am11 mentioned this issue Dec 1, 2024
Closed
@am11
Copy link
Member

am11 commented Dec 1, 2024

All these failures seem to be exclusive to win-x86. e.g. https://helix.dot.net/api/2019-06-17/jobs/6ec31758-1d44-4d2d-a257-0af23b46ac9a/workitems/System.Formats.Nrbf.Tests/console

@adamsitnik adamsitnik self-assigned this Dec 3, 2024
@adamsitnik
Copy link
Member

adamsitnik commented Dec 3, 2024 •
edited
Loading

Of course I can't repro it locally..

.\build.cmd -c Release -subset clr+libs+libs.tests -arch x86
.\dotnet.cmd build -c Release .\src\libraries\System.Formats.Nrbf\tests\System.Formats.Nrbf.Tests.csproj /t:Test /p:TargetArchitecture=x86
  ----- start Tue 12/03/2024 16:45:01.66 ===============  To repro directly: =====================================================
  pushd D:\projects\runtime\artifacts\bin\System.Formats.Nrbf.Tests\Release\net10.0\
  "D:\projects\runtime\artifacts\bin\testhost\net10.0-windows-Release-x86\dotnet.exe" exec --runtimeconfig System.Formats.Nrbf.Tests.runtimeconfig.json --depsfile System.Formats.Nrbf.Tests.deps.json C:\Users\adsitnik\.nuget\packages\mi
  crosoft.dotnet.xunitconsolerunner\2.9.2-beta.24570.4\build\..\tools\net\xunit.console.dll System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Nrbf.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Nrbf.Tests (found 125 test cases)
    Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [24 threads], stop on fail = off)
      System.Formats.Nrbf.Tests.EdgeCaseTests.CanReadArrayOfAnySize [SKIP]
        Condition(s) not met: "Is64BitProcess"
    Finished:    System.Formats.Nrbf.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Nrbf.Tests  Total: 365, Errors: 0, Failed: 0, Skipped: 1, Time: 12.073s
  ----- end Tue 12/03/2024 16:45:14.71 ----- exit code 0 ----------------------------------------------------------
  ----- start Tue 12/03/2024 16:45:36.03 ===============  To repro directly: =====================================================
  pushd D:\projects\runtime\artifacts\bin\System.Formats.Nrbf.Tests\Release\net462\
  C:\Users\adsitnik\.nuget\packages\xunit.runner.console\2.9.2\build\..\tools\net472\xunit.console.exe System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Nrbf.Tests (app domain = on [no shadow copy], method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Nrbf.Tests (found 121 of 123 test cases)
    Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [24 threads], stop on fail = off)
    Finished:    System.Formats.Nrbf.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Nrbf.Tests  Total: 363, Errors: 0, Failed: 0, Skipped: 0, Time: 9.918s
  ----- end Tue 12/03/2024 16:45:46.64 ----- exit code 0 ----------------------------------------------------------

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:52.31

.\build.cmd -rc Release -subset clr+libs+libs.tests -arch x86
.\dotnet.cmd build .\src\libraries\System.Formats.Nrbf\tests\System.Formats.Nrbf.Tests.csproj /t:Test /p:TargetArchitecture=x86
  ========================== End custom configuration settings ===============================
  ----- start Tue 12/03/2024 14:05:44.36 ===============  To repro directly: =====================================================
  pushd D:\projects\runtime\artifacts\bin\System.Formats.Nrbf.Tests\Debug\net10.0\
  "D:\projects\runtime\artifacts\bin\testhost\net10.0-windows-Debug-x86\dotnet.exe" exec --runtimeconfig System.Formats.Nrbf.Tests.runtimeconfig.json --depsfile System.Formats.Nrbf.Tests.deps.json C:\Users\adsitnik\.nuget\packages\micr
  osoft.dotnet.xunitconsolerunner\2.9.2-beta.24570.4\build\..\tools\net\xunit.console.dll System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Nrbf.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Nrbf.Tests (found 125 test cases)
    Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [24 threads], stop on fail = off)
      System.Formats.Nrbf.Tests.EdgeCaseTests.CanReadArrayOfAnySize [SKIP]
        Condition(s) not met: "Is64BitProcess"
    Finished:    System.Formats.Nrbf.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Nrbf.Tests  Total: 365, Errors: 0, Failed: 0, Skipped: 1, Time: 21.922s
  ----- end Tue 12/03/2024 14:06:07.86 ----- exit code 0 ----------------------------------------------------------
  ========================== End custom configuration settings ===============================
  ----- start Tue 12/03/2024 14:06:28.71 ===============  To repro directly: =====================================================
  pushd D:\projects\runtime\artifacts\bin\System.Formats.Nrbf.Tests\Debug\net462\
  C:\Users\adsitnik\.nuget\packages\xunit.runner.console\2.9.2\build\..\tools\net472\xunit.console.exe System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Nrbf.Tests (app domain = on [no shadow copy], method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Nrbf.Tests (found 121 of 123 test cases)
    Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [24 threads], stop on fail = off)
    Finished:    System.Formats.Nrbf.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Nrbf.Tests  Total: 363, Errors: 0, Failed: 0, Skipped: 0, Time: 10.364s
  ----- end Tue 12/03/2024 14:06:40.41 ----- exit code 0 ----------------------------------------------------------

I'll try to reproduce it on Win 10

@jeffhandley jeffhandley added this to the 10.0.0 milestone Dec 3, 2024
@jeffhandley jeffhandley removed the untriaged New issue has not been triaged by the area owner label Dec 3, 2024
@jkotas
Copy link
Member

jkotas commented Dec 3, 2024

build.cmd -rc Release

The timeouts occur with x86-coreclr_checked. Try with -rc checked.

parallel test collections = on [24 threads]

The CI machines have much smaller number of cores. This message on CI machines says 4 threads. You may want to restrict the test parallelization to better much the CI conditions.

@thaystg thaystg mentioned this issue Dec 9, 2024
Merged
@jakobbotsch
Copy link
Member

This has a ton of hits, any progress on this @adamsitnik? Can the test be disabled while the investigation happens?

@jakobbotsch jakobbotsch added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Dec 10, 2024
@adamsitnik adamsitnik mentioned this issue Dec 10, 2024
Merged
@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Dec 10, 2024
@max-charlamb max-charlamb mentioned this issue Dec 11, 2024
Merged
@VSadov VSadov mentioned this issue Dec 11, 2024
Merged
@elinor-fung elinor-fung mentioned this issue Dec 13, 2024
Merged
@akoeplinger akoeplinger mentioned this issue Dec 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adamsitnik adamsitnik

Labels
arch-x86 area-System.Formats.Nrbf blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' in-pr There is an active PR which will close this issue when it is merged Known Build Error Use this to report build issues in the .NET Helix tab os-windows
Projects
None yet
Milestone
10.0.0
6 participants
@EgorBo @jeffhandley @am11 @adamsitnik @jkotas @jakobbotsch