Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure: AggregateException: Not all processes in process tree could be terminated. #93321

Open
v-wenyuxu opened this issue Oct 11, 2023 · 21 comments
Open

Test failure: AggregateException: Not all processes in process tree could be terminated. #93321

v-wenyuxu opened this issue Oct 11, 2023 · 21 comments
Labels
area-System.Diagnostics.Process Known Build Error Use this to report build issues in the .NET Helix tab
Milestone
Future

Comments

@v-wenyuxu
Copy link

v-wenyuxu commented Oct 11, 2023 •
edited by build-analysis bot
Loading

Failed in: runtime-coreclr jitstress 20231009.2

Failed tests:

coreclr osx x64 Checked jitstress2 @ OSX.1200.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh
    - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
    - baseservices/exceptions/unhandled/unhandled/unhandled.sh

Error message:

Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
Expected: True
Actual:   False

Stack trace:

   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2023-10-10T05:50:17.995Z runtime-coreclr jitstress Checked-jitstress2 osx.1200.amd64.open x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-10T05:50:17.995Z runtime-coreclr jitstress Checked-jitstress2 osx.1200.amd64.open x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-10T05:50:17.995Z runtime-coreclr jitstress Checked-jitstress2 osx.1200.amd64.open x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-10T05:50:17.995Z runtime-coreclr jitstress Checked-jitstress2 osx.1200.amd64.open x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
More failures
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
### Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": [ "Not all processes in process tree could be terminated.", "Operation not permitted" ],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎
Result validation: ⚠️ Provided build not found. Provide a valid build in the "Build: 🔎" line.
Validation performed at: 4/2/2024 6:55:27 AM UTC

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0
@v-wenyuxu v-wenyuxu added os-mac-os-x macOS aka OSX JitStress CLR JIT issues involving JIT internal stress modes arch-x64 labels Oct 11, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 11, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 11, 2023
@ghost
Copy link

ghost commented Oct 12, 2023

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

Issue Details

Failed in: runtime-coreclr jitstress 20231009.2

Failed tests:

coreclr osx x64 Checked jitstress2 @ OSX.1200.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh
    - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
    - baseservices/exceptions/unhandled/unhandled/unhandled.sh

Error message:

Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
Expected: True
Actual:   False

Stack trace:

   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2023-10-10T05:50:17.995Z runtime-coreclr jitstress Checked-jitstress2 osx.1200.amd64.open x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-10T05:50:17.995Z runtime-coreclr jitstress Checked-jitstress2 osx.1200.amd64.open x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-10T05:50:17.995Z runtime-coreclr jitstress Checked-jitstress2 osx.1200.amd64.open x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-10T05:50:17.995Z runtime-coreclr jitstress Checked-jitstress2 osx.1200.amd64.open x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
More failures
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:17.734Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-08T14:21:16.532Z runtime-coreclr runincontext Checked ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
2023-10-07T14:20:57.576Z runtime-coreclr runincontext Checked-no_tiered_compilation ubuntu.2204.amd64.open.rt x64 baseservices/exceptions/unhandled/unhandled/unhandled.sh Failed {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} Schedule
Author: v-wenyuxu
Assignees: -
Labels:

area-System.Diagnostics.Process, os-mac-os-x, JitStress, arch-x64, untriaged, needs-area-label
Milestone: -

@vcsjones vcsjones removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 16, 2023
@AndyAyersMS AndyAyersMS mentioned this issue Nov 1, 2023
Merged
@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r-extra 20240224.1

Failed tests:

R2R-CG2 linux arm Checked jitstressregs4 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor251|25_252(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r-extra 20240316.1

Failed tests:

R2R-CG2 linux arm Checked jitstress1 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor251|25_252(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)

@mikelle-rogers
Copy link
Member

@mangod9, what is the status of this failure?

@mangod9
Copy link
Member

mangod9 commented Mar 18, 2024

seems to be only affecting jitstress. Did you notice this failure on a PR run?

@mikelle-rogers
Copy link
Member

One of the vendors noticed it here, as seen in the above comment.

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r-extra 20240323.1

Failed tests:

R2R-CG2 linux arm Checked jitstress2_tiered @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor251|25_252(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)

@jkotas jkotas mentioned this issue Apr 2, 2024
Closed
@jkotas jkotas changed the title Test failure baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh Test failure: AggregateException: Not all processes in process tree could be terminated. Apr 2, 2024
@jkotas jkotas added area-System.Diagnostics.Process Known Build Error Use this to report build issues in the .NET Helix tab blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' and removed os-mac-os-x macOS aka OSX JitStress CLR JIT issues involving JIT internal stress modes arch-x64 area-VM-coreclr labels Apr 2, 2024
@jkotas
Copy link
Member

jkotas commented Apr 2, 2024

This is hit intermittently on both macOS and Linux, different architectures, for different tests. I have just resolved #99885 as a duplicate.

The problem seems to be a race condition in the implementation of System.Diagnostic.Process.Kill(entireProcessTree: true).

The processes in the tree can be starting and exiting while Kill(entireProcessTree: true) is trying to kill them all. There seems to be some specific pattern where the process is found, but it exits on its own as it is getting killed, that results into Operation not permitted error and exception.

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r 20240401.1

Failed tests:

R2R-CG2 linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

    at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor251|25_252(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r-extra 20240407.1

Failed tests:

R2R-CG2 linux arm Checked jitstressregs0x80 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs8 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs0x2000 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstress2 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked forcerelocs @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs3 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitminopts @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstress2_tiered @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule

@jakobbotsch jakobbotsch mentioned this issue Apr 11, 2024
Merged
@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r-extra 20240413.1

Failed tests:

R2R-CG2 linux arm Checked jitstressregs0x1000 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs3 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstress1_tiered @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs0x10 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs8 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstress2 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule

@mikelle-rogers
Copy link
Member

cc @dotnet/area-system-diagnostics-process

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r 20240415.2

Failed tests:

R2R-CG2 linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r 20240416.1

Failed tests:

R2R-CG2 linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r 20240417.1

Failed tests:

R2R-CG2 linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr r2r 20240419.1

Failed tests:

R2R-CG2 linux arm Checked @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule

@mikelle-rogers
Copy link
Member

@jeffhandley, what is the status of this test failure?

@jozkee
Copy link
Member

jozkee commented Apr 29, 2024

@jkotas would it be OK to ignore "operation not permitted" errors on Kill(entireProcessTree: true)?

@jozkee jozkee added this to the 9.0.0 milestone Apr 29, 2024
@jozkee jozkee removed the untriaged New issue has not been triaged by the area owner label Apr 29, 2024
@jozkee
Copy link
Member

jozkee commented Apr 29, 2024

@mikelle-rogers we currently don't have a fix nor a consistent repro, we could spend more time looking into it but not sure if we could get a repro at all.

@jkotas
Copy link
Member

jkotas commented May 6, 2024

would it be OK to ignore "operation not permitted" errors on Kill(entireProcessTree: true)?

I do not know. We need to understand what leads to the "operation not permitted" errors.

jkotas added a commit to jkotas/runtime that referenced this issue Jun 1, 2024
@jkotas
Add retry loop around Process.Kill(entireProcessTree: true)
a7993bb 
Workaround dotnet#93321
@jkotas jkotas mentioned this issue Jun 1, 2024
Merged
jkotas added a commit that referenced this issue Jun 1, 2024
@jkotas
Add retry loop around Process.Kill(entireProcessTree: true) (#102958)
cb71b60 
Workaround #93321
@jkotas jkotas removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Jun 1, 2024
@jkotas
Copy link
Member

jkotas commented Jun 2, 2024

Delete workaround added in #102958 once this gets fixed.

@jeffhandley jeffhandley modified the milestones: 9.0.0, Future Aug 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-System.Diagnostics.Process Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Milestone
Future
Development

No branches or pull requests
9 participants
@vcsjones @jeffhandley @adamsitnik @jkotas @jeffschwMSFT @jozkee @mikelle-rogers @mangod9 @v-wenyuxu