Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mono_os_mutex_destroy: pthread_mutex_destroy failed with "Device or resource busy" #99609

Closed
jkotas opened this issue Mar 12, 2024 · 3 comments · Fixed by #100938
Closed

mono_os_mutex_destroy: pthread_mutex_destroy failed with "Device or resource busy" #99609

jkotas opened this issue Mar 12, 2024 · 3 comments · Fixed by #100938
Assignees
@davmason
Labels
area-Diagnostics-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Milestone
9.0.0

Comments

@jkotas
Copy link
Member

jkotas commented Mar 12, 2024 •
edited by build-analysis bot
Loading

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=599402
Build error leg or test failing: System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
Pull request: #99595

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "mono_os_mutex_destroy: pthread_mutex_destroy failed",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=599402
Error message validated: [mono_os_mutex_destroy: pthread_mutex_destroy failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 3/12/2024 3:23:03 PM UTC

Report

Build Definition Test Pull Request
641780 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100386
641646 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100993
640958 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100765
640844 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100951
640688 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #93014
640353 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100743
639824 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #100767
639593 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100669
639437 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100520
638524 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100767
637807 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100489
637015 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100828
636603 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100802
636443 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100821
635603 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #100743
635054 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100778
633229 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #100731
631865 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100676
631246 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99659
631044 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100497
630711 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100650
630272 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #100280
629948 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100525
629121 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100489
629043 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100589
628975 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100141
628043 dotnet/runtime System.Transactions.Tests.AsyncTransactionScopeTests.AsyncTSTest
627578 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100550
626716 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
626182 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #92057
625897 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100379
625012 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99908
624382 dotnet/runtime System.Diagnostics.Tests.ProcessWaitingTests.WaitAsyncForPeerProcess
623236 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100176
623242 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100386
623223 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100283
622469 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #100141
622371 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100362
621284 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100352
621302 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100266
620319 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99433
619690 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99596
619142 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #99840
618646 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100227
618638 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100272
618562 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100004
618336 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100004
617941 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100265
617442 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100252
616081 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99860
614850 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #97589
614368 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #100152
614460 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100156
613929 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #100134
614036 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #100146
614003 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99760
613807 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100038
611976 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #100087
611769 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #100056
611393 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #99956
611360 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
610653 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99973
610384 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #96332
609332 dotnet/runtime System.Threading.Tests.EtwTests.WaitHandleWaitEventTest #99970
608721 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #99956
607317 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99924
607174 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99920
606180 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
605861 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #97759
603435 dotnet/runtime System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest #99766
603223 dotnet/runtime tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd #99761

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
5 17 71
@jkotas jkotas added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Mar 12, 2024
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Mar 12, 2024
@jkotas jkotas added area-Diagnostics-mono and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Mar 12, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Mar 12, 2024
@jkotas
Copy link
Member Author

jkotas commented Mar 12, 2024

This is same as #97565 . The fix does not seem to work.

@jkotas
Copy link
Member Author

jkotas commented Mar 12, 2024

The eventpipe mutex is being destroyed here:

#17 0x0000ffff93701468 in mono_os_mutex_destroy (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:97
#18 mono_coop_mutex_destroy (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:45
#19 ep_rt_spin_lock_free (spin_lock=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:1213
#20 ep_rt_mono_fini () at /__w/1/s/src/mono/mono/eventpipe/ep-rt-mono.c:892
#21 0x0000ffff935b4d68 in mini_cleanup (domain=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:5259
#22 0x0000ffff937814ac in ves_icall_System_Environment_Exit (result=42) at /__w/1/s/src/mono/mono/metadata/icall.c:6151

While eventpipe is still running on a different thread here:

#4  0x0000ffff93643a38 in mono_handle_native_crash (signal=0xffff934e04f3 "SIGSEGV", mctx=0xffff90255670, info=0xffff902559d0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2967
#5  0x0000ffff935b0774 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffff902559d0, context=0xffff90255a50, debug_fault_addr=0x78) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3888
#6  <signal handler called>
#7  mono_atomic_cas_ptr (dest=0x78, exch=0x0, comp=0x0) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/atomic.h:585
#8  mono_atomic_load_ptr (src=0x78) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/atomic.h:783
#9  update_callback (location=0x78, new_=<optimized out>, counter=<optimized out>) at /__w/1/s/src/mono/mono/metadata/profiler.c:805
#10 mono_profiler_set_jit_begin_callback (handle=0x0, cb=0x0, cb@entry=0xffff90256c30) at /__w/1/s/src/native/public/./mono/metadata/profiler-events.h:47
#11 0x0000ffff93702300 in EventPipeEtwCallbackDotNETRuntime (source_id=<optimized out>, is_enabled=is_enabled@entry=0, level=level@entry=4 '\004', match_any_keywords=match_any_keywords@entry=65536, match_all_keywords=<optimized out>, filter_data=<optimized out>, callback_data=callback_data@entry=0x0) at /__w/1/s/src/mono/mono/eventpipe/ep-rt-mono-runtime-provider.c:4396
#12 0x0000ffff936f3524 in ep_rt_provider_invoke_callback (callback_func=0xffff9370226c <EventPipeEtwCallbackDotNETRuntime>, source_id=0x0, is_enabled=0, level=4 '\004', match_any_keywords=65536, match_all_keywords=0, filter_data=0x0, callback_data=0x0) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:520
#13 provider_invoke_callback (provider_callback_data=<optimized out>, provider_callback_data@entry=0xffff90256cf0) at /__w/1/s/src/native/eventpipe/ep-provider.c:419
#14 0x0000ffff936f3cdc in disable_helper (id=<optimized out>) at /__w/1/s/src/native/eventpipe/ep.c:665

@tommcdon tommcdon added this to the 9.0.0 milestone Mar 18, 2024
@tommcdon
Copy link
Member

This is same as #97565 . The fix does not seem to work.

@davmason please take a look

@tommcdon tommcdon removed the untriaged New issue has not been triaged by the area owner label Mar 18, 2024
@pavelsavara pavelsavara mentioned this issue Apr 9, 2024
Merged
@davmason davmason mentioned this issue Apr 11, 2024
Merged
@github-actions github-actions bot locked and limited conversation to collaborators May 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@davmason davmason

Labels
area-Diagnostics-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Milestone
9.0.0
Development

Successfully merging a pull request may close this issue.

Skip cleanup of eventpipe for mono to prevent crashes on shutdown davmason/runtime
3 participants
@jkotas @davmason @tommcdon