Possible deadlock in System.Diagnostics.Tracing.EtwEventProvider.System.Diagnostics.Tracing.IEventProvider.EventUnregister #48342
Comments
dotnet-issue-labeler
bot
added
area-Tracing-coreclr
untriaged
sywhang
added
bug
and removed
untriaged
|
Thanks for reporting this @n777ty. I've identified the issue to be with the way we hold the EventListener lock inside EventSource. I'll follow up with you on additional details offline. |
|
@sywhang I have a similar deadlock issue intermittently in a Azure Webjob process - during shutdown, Finalizer is blocked on System_Private_CoreLib!System.Diagnostics.Tracing.EtwEventProvider.System.Diagnostics.Tracing.IEventProvider.EventUnregister, which seems to be waiting on another thread that's waiting on the finalizer itself. Its using .Net core 3.1. Is there any workaround to fix the issue (other than killing the process) ? Also, is there a possibility of fixing it in .Net core 3.1 ? |
|
@gasridha There isn't any workaround. If you're the one starting the ETW session, then you might be able to replace it with other tracing alternatives like Re: backport to 3.1, I'm no longer at .NET team so I can't answer that but @josalem should be able to. |
|
The bug appears to be that we are holding the lock while calling Dispose() here. We've already resolved a similar issue in the past and went out of our way in EventProvider.Dispose() to avoid having the lock held. However grabbing the lock outside of Dispose() prevented us from calling into ETW without the lock held as intended. |
Fixes dotnet#48342 A deadlock was occuring because we held the EventListenersLock while calling into EventUnregister which will take ETW's own native lock. In the case of ETW sending an enable/disable notification these locks are taken in reverse order which triggers a deadlock. The fix is to ensure that we always order the locks so that any code path taking both locks always takes the ETW lock first. In this case it meant accumulating the list of event sources to dispose under the lock but then exiting the lock prior to calling Dispose() which will eventually call EventUnregister.
ghost
added
the
in-pr
Fixes #48342 A deadlock was occuring because we held the EventListenersLock while calling into EventUnregister which will take ETW's own native lock. In the case of ETW sending an enable/disable notification these locks are taken in reverse order which triggers a deadlock. The fix is to ensure that we always order the locks so that any code path taking both locks always takes the ETW lock first. In this case it meant accumulating the list of event sources to dispose under the lock but then exiting the lock prior to calling Dispose() which will eventually call EventUnregister.
ghost
removed
the
in-pr
|
Fixed in #56453 |
ghost
locked as resolved and limited conversation to collaborators
We are observing a possible deadlock (at minimum it's an infinite hang) as follows:
Finalizer thread is blocked trying to acquire SRWLock
Finalizer thread owns another lock itself
Both of these threads are dealing with System.Threading.Tasks.TplEventSource event provider.
Shutdown never completes due to this lock-up (sequence of dumps confirms)
The text was updated successfully, but these errors were encountered: