Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GC/Regressions/v2.0-beta2/452950 failed in CI #103494

Closed
stephentoub opened this issue Jun 14, 2024 · 6 comments · Fixed by #103898
Closed

GC/Regressions/v2.0-beta2/452950 failed in CI #103494

stephentoub opened this issue Jun 14, 2024 · 6 comments · Fixed by #103898
Labels
area-GC-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab

Comments

@stephentoub
Copy link
Member

stephentoub commented Jun 14, 2024 •
edited by build-analysis bot
Loading

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=708252
Build error leg or test failing: GC/Regressions/v2.0-beta2/452950/452950/452950.sh
Pull request: #103308

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "452950",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}
❌ GC/Regressions/v2.0-beta2/452950/452950/452950.sh [[Console]](https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-103308-merge-646fbe4a4e9e472bad/PayloadGroup0/1/console.be1394e5.log?helixlogtype=result) [[Details]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=708252&view=ms.vss-test-web.build-test-results-tab&runId=17681016&resultId=110930&paneView=debug) [[Artifacts]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=708252&view=ms.vss-test-web.build-test-results-tab&runId=17681016&resultId=110930&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab) [[0.29% failure rate]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=708252&view=ms.vss-test-web.build-test-results-tab&runId=17681016&resultId=110930&paneView=history)
[[ 🚧 Report test infrastructure issue]](https://helix.dot.net/ki/new/repository=dotnet%2Fdnceng&build=https%3A%2F%2Fdev.azure.com%2Fdnceng-public%2Fcbb18261-c48f-4abb-8651-8cdcb5474649%2F_build%2Fresults%3FbuildId%3D708252&template=z-build-break-infrastructure-issue-template.yml&pr=https%3A%2F%2Fgit.luolix.top%2Fdotnet%2Fruntime%2Fpull%2F103308&labels=Known%20Build%20Error%2CFirst%20Responder%2CDetected%20By%20-%20Customer&build-leg=GC%2FRegressions%2Fv2.0-beta2%2F452950%2F452950%2F452950.sh) [[ 📄 Report test repository issue]](https://helix.dot.net/ki/new/repository=dotnet%2Fruntime&build=https%3A%2F%2Fdev.azure.com%2Fdnceng-public%2Fcbb18261-c48f-4abb-8651-8cdcb5474649%2F_build%2Fresults%3FbuildId%3D708252&pr=https%3A%2F%2Fgit.luolix.top%2Fdotnet%2Fruntime%2Fpull%2F103308&labels=Known%20Build%20Error%2Cblocking-clean-ci&build-leg=GC%2FRegressions%2Fv2.0-beta2%2F452950%2F452950%2F452950.sh)
Failing Configuration
[mono osx x64 Release @ OSX.1200.Amd64.Open](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=708252&view=ms.vss-test-web.build-test-results-tab&runId=17681016&resultId=110930)
Exception Message
Return code:      1
Raw output file:      /tmp/helix/working/B9D70A12/w/B066094B/uploads/Reports/GC.Regressions/v2.0-beta2/452950/452950/452950.output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/B9D70A12/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true 452950.dll ''
53
Failed
Expected: 100
Actual: 1
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /cores since 6/14/2024 12:11:43 PM
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/tmp/helix/working/B9D70A12/p
> /private/tmp/helix/working/B9D70A12/w/B066094B/e/GC/Regressions/v2.0-beta2/452950/452950/452950.sh
CallStack
   at GC_Regressions._v2_0_beta2_452950_452950_452950_._v2_0_beta2_452950_452950_452950_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)

at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=708252
Error message validated: [452950]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/14/2024 10:08:00 PM UTC

Report

Build Definition Step Name Console log Pull Request
717846 dotnet/runtime Build product Log #103583
716665 dotnet/runtime Build Tests Log #103795
716470 dotnet/runtime Build Tests Log #103795
2479502 dotnet-runtime Build product Log
715298 dotnet/runtime Build Tests Log #103681
714548 dotnet/runtime Build Tests Log #103184
714434 dotnet/runtime Build product Log #103583
714256 dotnet/runtime Build product Log #103228
714140 dotnet/runtime Build Log #103726
713697 dotnet/runtime Send to Helix Log #102739
713671 dotnet/runtime Build product Log #103726
713672 dotnet/runtime Build product Log #103726
713619 dotnet/runtime Build product Log #103732
713506 dotnet/runtime Build product Log #103726
713111 dotnet/runtime Build Tests Log #103673
713090 dotnet/runtime Build product Log #103709
2477353 dotnet-runtime Build product Log
711934 dotnet/runtime Build Log #103594
711700 dotnet/runtime Build product Log #103647
711056 dotnet/runtime Build product Log #103361
710997 dotnet/runtime Build product Log #103364
710836 dotnet/runtime Build Tests Log #103427
710820 dotnet/runtime Build product Log #103502
710611 dotnet/runtime Build product Log #103593
710476 dotnet/runtime Build product Log #103585
709734 dotnet/runtime Build product Log #103274
709124 dotnet/runtime Build Tests Log #103427
709030 dotnet/runtime Build product Log #103503
708732 dotnet/runtime Build product Log #103503
708604 dotnet/runtime Build product Log #103184
Build Definition Test Pull Request
717806 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103889
717754 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103884
717720 dotnet/runtime GC\Scenarios\RanCollect\rancollect\rancollect.cmd #103799
717693 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103883
717668 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103701
717653 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103852
717621 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103876
717620 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103876
717617 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103510
717542 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103846
717524 dotnet/runtime GC/Regressions/v2.0-beta2/452950/452950/452950.sh #103738
717491 dotnet/runtime PayloadGroup0.WorkItemExecution
717472 dotnet/runtime PayloadGroup0.WorkItemExecution #103697
717493 dotnet/runtime PayloadGroup0.WorkItemExecution #100030
717449 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103869
2480457 dotnet-runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd
2480433 dotnet-runtime release.Partition14.WorkItemExecution
717274 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103412
717200 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd
2480201 dotnet-runtime x64.micro_mono.net9.0.Partition5.WorkItemExecution
716966 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103216
716928 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103837
716905 dotnet/runtime PayloadGroup0.WorkItemExecution #103852
716753 dotnet/runtime PayloadGroup0.WorkItemExecution #103701
716689 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103832
716673 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd
2479625 dotnet-runtime release.Partition2.WorkItemExecution
716602 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103849
716510 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103361
716194 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd
716155 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103827
716143 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103818
715858 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103788
715882 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103735
715785 dotnet/runtime GC\Regressions\v2.0-beta2\452950\452950\452950.cmd #101796
715812 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103701
715736 dotnet/runtime PayloadGroup0.WorkItemExecution #103765
715613 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103813
715604 dotnet/runtime PayloadGroup0.WorkItemExecution #103697
715524 dotnet/runtime GC\Regressions\v2.0-beta2\452950\452950\452950.cmd #101796
715414 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103803
715411 dotnet/runtime PayloadGroup0.WorkItemExecution #103701
715312 dotnet/runtime PayloadGroup0.WorkItemExecution
715289 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103766
715226 dotnet/runtime PayloadGroup0.WorkItemExecution #103795
715154 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103792
2478804 dotnet-runtime release.Partition14.WorkItemExecution
715076 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103790
715103 dotnet/runtime PayloadGroup0.WorkItemExecution #103673
715065 dotnet/runtime PayloadGroup0.WorkItemExecution #103739
714919 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103594
714936 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103361
714627 dotnet/runtime PayloadGroup0.WorkItemExecution #103646
714480 dotnet/runtime PayloadGroup0.WorkItemExecution #103668
714212 dotnet/runtime GC/API/NoGCRegion/Callback_Svr/Callback_Svr.sh #103738
2478327 dotnet-runtime release.Partition14.WorkItemExecution
714391 dotnet/runtime PayloadGroup0.WorkItemExecution #103766
714331 dotnet/runtime PayloadGroup0.WorkItemExecution #103706
714215 dotnet/runtime PayloadGroup0.WorkItemExecution #103555
714025 dotnet/runtime PayloadGroup0.WorkItemExecution #103757
713739 dotnet/runtime System.Net.Http.Functional.Tests.WorkItemExecution #103646
713974 dotnet/runtime PayloadGroup0.WorkItemExecution #102739
713967 dotnet/runtime PayloadGroup0.WorkItemExecution #103076
2477968 dotnet-runtime wasm.x64.micro.net9.0.Partition14.WorkItemExecution
2477814 dotnet-runtime release.Partition26.WorkItemExecution
713550 dotnet/runtime PayloadGroup0.WorkItemExecution #103735
713594 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103737
713553 dotnet/runtime PayloadGroup0.WorkItemExecution #103673
713456 dotnet/runtime PayloadGroup0.WorkItemExecution #103104
713315 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103701
713368 dotnet/runtime PayloadGroup0.WorkItemExecution #103725
713124 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103184
713049 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103689
713033 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103361
713026 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103705
2477548 dotnet-runtime release.Partition14.WorkItemExecution
712969 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103704
712887 dotnet/runtime PayloadGroup0.WorkItemExecution #103701
712828 dotnet/runtime PayloadGroup0.WorkItemExecution #103574
712763 dotnet/runtime PayloadGroup0.WorkItemExecution #103635
712742 dotnet/runtime PayloadGroup0.WorkItemExecution #103697
712593 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103689
711814 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103559
712486 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #102403
2477175 dotnet-runtime release.Partition11.WorkItemExecution
712446 dotnet/runtime PayloadGroup0.WorkItemExecution #103361
712260 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103675
712223 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103540
712239 dotnet/runtime PayloadGroup0.WorkItemExecution #103638
712203 dotnet/runtime PayloadGroup0.WorkItemExecution #103144
712047 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103444
711823 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103502
711832 dotnet/runtime PayloadGroup0.WorkItemExecution #103620
711695 dotnet/runtime System.Net.Http.Functional.Tests.WorkItemExecution #103646
711611 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103564
711619 dotnet/runtime GC/Regressions/v2.0-beta2/452950/452950/452950.sh #101796
711489 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103634
2476348 dotnet-runtime release.Partition8.WorkItemExecution
711257 dotnet/runtime GC\Scenarios\FinalizeTimeout\FinalizeTimeout\FinalizeTimeout.cmd #103620
711283 dotnet/runtime PayloadGroup0.WorkItemExecution #103570
Displaying 100 of 123 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
19 131 153
@stephentoub stephentoub added area-GC-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Jun 14, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Jun 14, 2024
@dotnet-policy-service Dotnet Policy Service
Copy link
Contributor

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

@steveisok
Copy link
Member

@stephentoub I don't have the right answer now, but 452950 is too generic. For example, https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/2477353/logs/2590 matches because there's a timestamp with the same value. The problem with that build is it timed out.

@stephentoub
Copy link
Member Author

What would you recommend? Trying to be a good citizen and opening such issues according to the merge on green effort, but we don't make it easy to do the "right thing".

@steveisok
Copy link
Member

Maybe this as the errormessage?

GC/Regressions/v2.0-beta2/452950/452950/452950.sh [FAIL]

@jkotas
Copy link
Member

jkotas commented Jun 24, 2024

This accumulated too much noise due to too generic pattern. I am going to close it. We should open new issues with more specific pattern match as we hit them.

What would you recommend?

Use more specific pattern like what @steveisok suggested. Our docs at https://github.com/dotnet/runtime/blob/main/docs/workflow/ci/failure-analysis.md#bad-usage-examples mention that the test name alone is not a good pattern.

@jkotas jkotas closed this as completed Jun 24, 2024
@dotnet-policy-service dotnet-policy-service bot removed the untriaged New issue has not been triaged by the area owner label Jun 24, 2024
@jkotas
Copy link
Member

jkotas commented Jun 24, 2024

Updated #103874 to tracks some of the failures that this matched.

jkotas added a commit to jkotas/runtime that referenced this issue Jun 24, 2024
@jkotas
Delete flaky test
52cd0d1 
This test is tried to verify that the full GCs were performed less than 5% of time for particular allocation pattern. The number of full GCs performed can be influenced by timing and other activity on the machine, that makes this test unreliable.

Fixes dotnet#100174
Fixes dotnet#99729
Fixes dotnet#103494
jkotas added a commit to jkotas/runtime that referenced this issue Jun 24, 2024
@jkotas
Delete flaky test
db4da98 
This test tried to verify that the full GCs were performed less than 5% of time for particular allocation pattern. The number of full GCs performed can be influenced by timing and other activity on the machine, that makes this test unreliable.

Related dotnet#103494
Fixes dotnet#100174
Fixes dotnet#99729
jkotas added a commit to jkotas/runtime that referenced this issue Jun 24, 2024
@jkotas
Relax threshold in GC Regressions test
3f61f61 
Fixes dotnet#103494
@jkotas jkotas mentioned this issue Jun 24, 2024
Merged
jkotas added a commit that referenced this issue Jun 24, 2024
@jkotas
Relax threshold in GC Regressions test (#103898)
8d6e07d 
Fixes #103494
@github-actions github-actions bot locked and limited conversation to collaborators Jul 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-GC-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Relax threshold in GC Regressions test jkotas/runtime
3 participants
@steveisok @stephentoub @jkotas