Using CreateSnapshotAndAttach leaves a process running on Linux using ClrMD 3.1.456101 and .NET 8

[5teveb]

I'm using CreateSnapshotAndAttach to get stack dumps for all threads, calling it on my own process. Despite calling dispose on the DataTarget and ClrRuntime objects a process is left behind.

A small sample program shows the problem:

using System;
using System.Linq;
using Microsoft.Diagnostics.Runtime;

namespace ClrMDTest
{
    internal class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Press any key to stack dump all threads.");
            Console.ReadLine();

            PrintStackDump();

            Console.WriteLine("Stack dump complete... press any key to exit.");

            Console.ReadLine();

            Console.WriteLine("Exiting...");
        }

        private static void PrintStackDump()
        {
            Console.WriteLine("Attempt to stack dump threads:");

            DataTarget target = null;
            ClrRuntime runtime = null;
            try
            {
                target = DataTarget.CreateSnapshotAndAttach(Environment.ProcessId);
                runtime = target.ClrVersions.First().CreateRuntime();


                foreach (ClrThread thread in runtime.Threads)
                {
                    Console.WriteLine($"ID:{thread.ManagedThreadId} OSID:{thread.OSThreadId} Alive:{thread.IsAlive} Finalizer:{thread.IsFinalizer} GC:{thread.IsGc}");

                    var stackFrames = thread.EnumerateStackTrace();
                    foreach (var clrStackFrame in stackFrames)
                    {
                        if (clrStackFrame.Method != null)
                        {
                            Console.WriteLine("  " + clrStackFrame.Method?.Signature);
                        }
                    }

                    Console.WriteLine();
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Failed with exception: {ex}");
            }
            finally
            {
                runtime?.Dispose();
                target?.Dispose();
            }
        }
    }
}

I'm using the latest version or ClrMD and .NET 8:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Diagnostics.Runtime" Version="3.1.456101" />
  </ItemGroup>

</Project>

And publishing for Linux arm64:

<?xml version="1.0" encoding="utf-8"?>
<!--
https://go.microsoft.com/fwlink/?LinkID=208121. 
-->
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <Configuration>Release</Configuration>
    <Platform>Any CPU</Platform>
    <PublishDir>bin\Release\net8.0\publish\linux-arm64\</PublishDir>
    <PublishProtocol>FileSystem</PublishProtocol>
    <TargetFramework>net8.0</TargetFramework>
    <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
    <SelfContained>true</SelfContained>
    <PublishSingleFile>true</PublishSingleFile>
    <IncludeAllContentForSelfExtract>True</IncludeAllContentForSelfExtract>
    <PublishTrimmed>false</PublishTrimmed>
  </PropertyGroup>
</Project>

Cross compiling on Windows for arm64 Linux using VS2022 17.8.1.

Running the program:

rdp@rpi4:~/Software $ ./ClrMDTest
Press any key to stack dump all threads.

I check the process list:

rdp@rpi4:~ $ ps -au | grep ClrMDTest
rdp        24328  0.3  0.3 271588904 27044 pts/4 Sl+  11:36   0:00 ./ClrMDTest
rdp        24338  0.0  0.0   5908   636 pts/5    S+   11:37   0:00 grep --color=auto ClrMDTest

Then press any key (I chose space, in case you were wondering):

rdp@rpi4:~/Software $ ./ClrMDTest
Press any key to stack dump all threads.

Attempt to stack dump threads:
[createdump] Gathering state for process 24328 ClrMDTest
[createdump] Writing full dump to file /tmp/tmpw9ZIYK.tmp
[createdump] Written 163643392 bytes (39952 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 2891ms
ID:1 OSID:24328 Alive:True Finalizer:False GC:False
  Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(System.IO.Stream)
  Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(System.IO.Stream)
  Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(Microsoft.Diagnostics.NETCore.Client.IpcEndpoint, Microsoft.Diagnostics.NETCore.Client.IpcMessage)
  Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Microsoft.Diagnostics.NETCore.Client.IpcEndpoint, Microsoft.Diagnostics.NETCore.Client.IpcMessage)
  Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDump(Microsoft.Diagnostics.NETCore.Client.DumpType, System.String, Microsoft.Diagnostics.NETCore.Client.WriteDumpFlags)
  Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDump(Microsoft.Diagnostics.NETCore.Client.DumpType, System.String, Boolean)
  Microsoft.Diagnostics.Runtime.LinuxSnapshotTarget.CreateSnapshotFromProcess(Int32)
  Microsoft.Diagnostics.Runtime.DataTarget.CreateSnapshotAndAttach(Int32)
  ClrMDTest.Program.PrintStackDump()
  ClrMDTest.Program.Main(System.String[])

ID:2 OSID:24333 Alive:True Finalizer:True GC:False

ID:3 OSID:24340 Alive:True Finalizer:False GC:False

ID:4 OSID:24341 Alive:True Finalizer:False GC:False
  Interop+Sys.WaitForSocketEvents(IntPtr, SocketEvent*, Int32*)
  System.Net.Sockets.SocketAsyncEngine.EventLoop()

ID:5 OSID:24335 Alive:True Finalizer:False GC:False

Stack dump complete... press any key to exit.

Now the process list again:

rdp@rpi4:~ $ ps -au | grep ClrMDTest
rdp        24328  2.2  1.4 272186884 114616 pts/4 Sl+ 11:36   0:02 ./ClrMDTest
rdp        24342  5.1  0.2 271667512 17708 pts/4 S+   11:38   0:01 ./ClrMDTest
rdp        24354  0.0  0.0   5908   636 pts/5    S+   11:38   0:00 grep --color=auto ClrMDTest

A new process with the same name has showed up. Press space again:

rdp@rpi4:~/Software $ ./ClrMDTest
Press any key to stack dump all threads.

Attempt to stack dump threads:
[createdump] Gathering state for process 24328 ClrMDTest
[createdump] Writing full dump to file /tmp/tmpw9ZIYK.tmp
[createdump] Written 163643392 bytes (39952 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 2891ms
ID:1 OSID:24328 Alive:True Finalizer:False GC:False
  Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(System.IO.Stream)
  Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(System.IO.Stream)
  Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(Microsoft.Diagnostics.NETCore.Client.IpcEndpoint, Microsoft.Diagnostics.NETCore.Client.IpcMessage)
  Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Microsoft.Diagnostics.NETCore.Client.IpcEndpoint, Microsoft.Diagnostics.NETCore.Client.IpcMessage)
  Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDump(Microsoft.Diagnostics.NETCore.Client.DumpType, System.String, Microsoft.Diagnostics.NETCore.Client.WriteDumpFlags)
  Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDump(Microsoft.Diagnostics.NETCore.Client.DumpType, System.String, Boolean)
  Microsoft.Diagnostics.Runtime.LinuxSnapshotTarget.CreateSnapshotFromProcess(Int32)
  Microsoft.Diagnostics.Runtime.DataTarget.CreateSnapshotAndAttach(Int32)
  ClrMDTest.Program.PrintStackDump()
  ClrMDTest.Program.Main(System.String[])

ID:2 OSID:24333 Alive:True Finalizer:True GC:False

ID:3 OSID:24340 Alive:True Finalizer:False GC:False

ID:4 OSID:24341 Alive:True Finalizer:False GC:False
  Interop+Sys.WaitForSocketEvents(IntPtr, SocketEvent*, Int32*)
  System.Net.Sockets.SocketAsyncEngine.EventLoop()

ID:5 OSID:24335 Alive:True Finalizer:False GC:False

Stack dump complete... press any key to exit.

Exiting...
rdp@rpi4:~/Software $

And check the process list one more time:

rdp@rpi4:~ $ ps -au | grep ClrMDTest
rdp        24342  2.0  0.2 271667512 17708 pts/4 S    11:38   0:01 ./ClrMDTest
rdp        24357  0.0  0.0   5908   652 pts/5    S+   11:39   0:00 grep --color=auto ClrMDTest

Is there something I'm missing here? I checked the /tmp directory and the file tmpw9ZIYK.tmp was not there so must have been cleaned up successfully.

[5teveb]

Forgot to mention the OS version:

rdp@rpi4:/tmp $ uname -a
Linux rpi4 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

[leculver]

This is likely a problem with the diagnostics IPC channel, owned by the dotnet/diagnostics repo as my usage of it hasn't changed in many versions. If so, I will transfer this to their repo. I'll take a look next week.