Question : IIS dotnet.exe memory management

[Tratcher]

From @onesandzeros415 on December 8, 2016 0:9

I apologize if this is a duplicate question however I've yet to find any concrete information on IIS dotnet.exe memory/cpu management. According to @DamianEdwards from September 17, 2015 via issue 364 IIS is still managing the process. The AppPool settings that control recycling, memory limits, etc. still apply.

Does this apply to limiting memory/cpu consumption of the dotnet.exe process itself? I'm curious as I've set a memory limit of 524 megabytes on an application pool and dotnet.exe is able to exceed this set limit. Setting process affinity also does not appear to work on the dotnet.exe process.

Any clarification on memory/cpu management of the dotnet.exe process via IIS would be helpful.

Copied from original issue: aspnet/Hosting#900

[shirhatti]

AFAIK, the child process spawned by w3wp.exe respects memory/cpu consumptions set on the AppPool.
@jhkimnew Could you verify?

[jhkimnew]

@shirhatti

I have verified that it works.

An IIS worker process (w3wp.exe) and all the dotnet processes which are created as a child process of the w3wp.exe process are grouped in the same job object group and the Private Memory Limit and the CPU time limit is used to set the Job object limit.

So, if you set the Private Memory Limit(KB) correctly, that should work. If it does not work immediately, maybe you need to recycle the worker process after setting the configuration to apply the configuration change in case the apppool is configured not to be recycled automatically when there is a configuration change.

If the recycle happens, you can check the System Event viewer log and you will see this event log supposing you are using the DefaultAppPool worker process.

<><><>
EventSource: "WAS"
..
A worker process serving application pool 'DefaultAppPool' has requested a recycle because it reached its private bytes memory limit.
<><><>

Thanks,
Jeong Hwan Kim

[onesandzeros415]

@jhkimnew, @shirhatti

Thanks for looking into this and confirming that the child process dotnet.exe should be following whatever value is specified for Private Memory Limit and Process Affinity Mask. I've verified after recycling that enabling and setting a value for Process Affinity Mask does work however it appears that any value set for Private Memory Limit is still being ignored.

I've attached a screen shot similar to my original post showing I've set a Private Memory Limit of 524277 however the dotnet.exe process is consuming far more than this even after an application pool recycle, IIS Reset, and reboot.

Is there a proposed fix or workaround for this?

[jhkimnew]

@onesandzeros415
Hi,

I have already verified that the Private Memory Limit (KB) works for the dotnet.exe process on my two test machines, one with Wondows 10 OS amd64 and the other Windows 8 OS. So, there is no fix nor workaround required. You would need to check if there is any mis-configuration.

First, I'd like to inform that you should make sure that the dotnet.exe process shown in the task manager is really created by worker (w3wp.exe) process.

Another thing is the actual private memory size should be obtained with a different way such as using powershell cmdlets, not Task manager because the working set memory is not matched to the private memory size which is used by the configuration.

Here is how I tested this thing on my test machine with powershell cmdlets for your information.

1. First get the process id of w3wp.exe process. In this example, 472 is the process id value.

NOTE: I assumed you are using a single AppPool such as DefaultAppPool.
<><>
PS C:\WINDOWS\system32> get-process -Name w3wp

Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
    674      72    39180      33236       0.42    472   0 w3wp
<><>

2. Get the child process process id of the dotnet.exe with the worker process id (472) obtained in the previous step. In this example, 12128 is the process id of the child process of donet.exe.

<><>
PS C:\WINDOWS\system32> Get-WmiObject win32_process | where {$_.ParentProcessId
-eq 472} | Select-Object Handle

Handle
------
12128
<><>

3. Now get the current process memory used by the dotnet.exe. In this example, the private memory size value of dotnet.exe is 13356 KB.

<><>
PS C:\WINDOWS\system32> get-process -Id 12128

Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
    418      86    13356      26788       0.84  12128   0 dotnet
<><>

4. Get the total value of private memory of w3wp.exe (39180) and the private memory (13356) of dotnet.exe. In this example, the total value is 52536 KB (=> 39180 + 13356).

In that case, if we set the private memory value 50000, which is less than 52536, for the DefaultAppPool, the w3wp.exe will be recycled (old process is terminated and new process for w3wp.exe is created) when the total private memory usage is bigger than 50000 KB after some time.
You should know the recycling w3wp.exe does not happen immediately. It will take some time and you would need to wait until the recycling happens.
When the recycling happens, the event viewer log is immediately created.

And when the w3wp.exe is recycled, the child process (dotnet.exe) is gracefully shutdown together and the dotnet.exe process will be gone. When there is a new request to the corresponding aspnetcore app, you will see a new process of dotnet.exe shown.

Thanks,
Jeong Hwan Kim

[jhkimnew]

@onesandzeros415, Hi, is there any update? Can I close this?

[onesandzeros415]

@jhkimnew I am sorry for the delay, it's been busy with the holidays.

1. The w3wp process itself remains consistently low :

PS C:\WINDOWS\system32> get-process -id 4388

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName

---

328      26     4988        416    45     0.16   4388 w3wp

2. Upon closer investigation I'm seeing some varying results with the dotnet process. Below are three samples a few seconds apart :
`
PS C:\WINDOWS\system32> get-process -Id 3780

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName

---

439     746  1185420    1826460   690    91.44   3780 dotnet

`

PS C:\WINDOWS\system32> get-process -Id 3780

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName

---

439     746  1636556     -86704   690    92.98   3780 dotnet

PS C:\WINDOWS\system32> get-process -Id 3780

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName

---

439     746 -1866292    1010644   690    96.30   3780 dotnet

Please keep in mind I've set 524277 KB as the Private Memory Limit. It appears that while not as large of a value as Task Manager/Resource Monitor get-process is showing the dotnet.exe process exceeding the Private Memory Limit set via IIS. Please let me know if you need any additional information and or if I need to make any changes to my configuration.

Regards,
Matthew

[jhkimnew]

@onesandzeros415

Thanks for the reply. Unless you did any mistake, that can't happen.

To narrow down your isse, can you check if you can recycle the w3wp.exe first?
I think you can set 5000 for the Private Memory Limit and after that send several requests to make the memory usage of w3wp.exe is over the 5000 KB and check if the w3wp.exe is recycled.

You should monitor the process id of w3wp.exe. If the process id is changed, you can say that the process is recycled.
Another check-point is that you have to check if the dotnet.exe process is also recycled together when the w3wp.exe process is recycled. If only w3wp.exe is recycled and the dotnet.exe is not recycled, you really did some mistake. FYI, the dotnet.exe process is created as a child process of corresponding w3wp.exe.

Thanks,
Jeong Hwan Kim

[jhkimnew]

I found that this issue can be reproducible only on down-level OS versions such as Win7 OS or Windows 2008 R2.

The IIS feature of limiting private memory for Job object, in which the processes of w3wp.exe and dotnet.exe are combined, was implemented after Win7 OS version.
So, if you want to use the “Private Memory Limit (KB)” for ANCM backend process (dotnet.exe), please make sure you use the latest version OS or above of Win7 OS.

[muratg]

What @jhkimnew said above.

[Joebeazelman]

So, if you want to use the “Private Memory Limit (KB)” for ANCM backend process (dotnet.exe), please make sure you use the latest version OS or above of Win7 OS.

Sorry, but this answer is unacceptable. The documentation specifically states that it supports Window 7, 2008 R2:

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/aspnet-core-module?view=aspnetcore-2.1

Yesterday, the dotnet.exe used so much RAM, it caused my entire server to become unresponsive, forcing me to cold reboot the server through my ISP. Is there a workaround for this? Interestingly, the dotnet.exe process starts out with over 500,000K allocated to it. As the process runs, it gradually exceeds over 1,000,000K of RAM. I was under the impression that .NET core was lighter than .NET classic.

[jhkimnew]

@Joebeazelman This repo is not used anymore. Please use
https://github.com/aspnet/IISIntegration/issues instead to report an issue on ANCM.

Please notice that aspnetcore.dll is used by w3wp.exe, not by dotnet.exe. So, memory usage of dotnet.exe is not caused by aspnetcore.dll.
Anyway, if your symptom is related to memory leak issue, you would need to narrow down the issue first to see which DLL is actually allocating memory but not releasing.