How to tell how much memory a .NET 6 application is really using?

[loop-evgeny]

Following on from #52592 (which is locked now)... We tried upgrading our application from .NET Core 3.1 to .NET 6 again and, as @PeterSolMS wrote in #52592 (comment), memory is not released to the OS until the server is low on RAM. (This is #37894, right?)

The problem this creates for us is that we do not know how much RAM a server really has available to run more applications! We previously used Resident Set Size for this (on Ubuntu 18.04). The managed memory size reported by GC.GetTotalMemory() gives us some idea, but it's not enough - there is always a significant amount of unmanaged memory use, too, and that may increase and become more uncertain if we start using something like SQLite, for example.

How can we tell how much RAM is actually needed by a .NET 6 application and how much would be available to other applications if they tried to use it?

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Following on from #52592 (which is locked now)... We tried upgrading our application from .NET Core 3.1 to .NET 6 again and, as @PeterSolMS wrote in #52592 (comment), memory is not released to the OS until the server is low on RAM. (This is #37894, right?)

The problem this creates for us is that we do not know how much RAM a server really has available to run more applications! We previously used Resident Set Size for this (on Ubuntu 18.04). The managed memory size reported by GC.GetTotalMemory() gives us some idea, but it's not enough - there is always a significant amount of unmanaged memory use, too, and that may increase and become more uncertain if we start using something like SQLite, for example.

How can we tell how much RAM is actually needed by a .NET 6 application and how much would be available to other applications if they tried to use it?

Author:	loop-evgeny
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[cshung]

From the GC's perspective, The GC doesn't know how much memory is used by SQLite either, so it cannot report how much memory is needed to run the application. But it knows two things.

• The memory the application is using, through the GC.GetTotalMemory() API.
• The memory the GC acquired from the operating system, through the TotalCommittedBytes property on GCMemoryInfo.

Suppose:

• You can also know the resident size, and
• Let's assume the native code does not have much committed-but-not-used memory. (*)

The (*) condition is highly questionable - I don't know SQLite, but databases usually come with a buffer manager that is memory hungry and commit memory just for eventual query use - you might want to figure out that and subtract that as well.

Then you could compute the currently in use memory by:

currently in use = resident set - total committed bytes + get total memory.

Also, note that the memory currently in use is not the same thing as "how much memory is needed?". These days, many libraries adjust their memory usage based on available memory (and thus trade off other performance characteristics, such as latency or throughput), so you might be trying to hit a moving target.

Once the memory is committed for one application, the OS cannot use that memory for another application, even when it is considered unused by the application. GC has no way to know that another application wanted to use it and therefore relinquish memory for other applications' use, we need someone to tell us that, and here is how:

• Aggressive GC #69695 is a feature available in .NET 7 that allows you to decommit as much memory as possible from the GC. This is meant for applications going idle. It is a bad idea to decommit all the memory back to the operating system just to get it back again. But once you decommit, other applications can use it.

• [API Proposal]: GC.RefreshMemoryLimit #70601 is a proposal that is not implemented yet (prototype available) that allows you to tell the GC how much you would allow the GC to use dynamically. Once you tell the GC your intent, the GC can act the way you need it to.

[loop-evgeny]

Thanks for such a prompt and detailed response @cshung!

currently in use = resident set - total committed bytes + get total memory

I will try out your formula as soon as I can, but just to make sure I understand it correctly: this is equivalent to resident set - (total committed bytes - get total memory)where(total committed bytes - get total memory)is essentially "memory that is committed, but not currently used by the GC", while resident set = all memory committed (and not paged out), thuscurrently in use = all committed - committed but not used` ?

I'd have to add the swapped out RAM (VmSwap from /proc/*/status) to RSS as well, I suppose.

[cshung]

Thanks for such a prompt and detailed response @cshung!

currently in use = resident set - total committed bytes + get total memory

I will try out your formula as soon as I can, but just to make sure I understand it correctly: this is equivalent to resident set - (total committed bytes - get total memory)where(total committed bytes - get total memory)is essentially "memory that is committed, but not currently used by the GC", while resident set = all memory committed (and not paged out), thuscurrently in use = all committed - committed but not used` ?

I'd have to add the swapped out RAM (VmSwap from /proc/*/status) to RSS as well, I suppose.

Yes, that's the idea behind that formula. Although I would say

total committed bytes - get total memory is essentially the "memory that is committed, but not currently used as managed objects by the application".

[loop-evgeny]

Also, just to confirm, it seems that Process.WorkingSet64 returns resident set size (VmRSS in /proc/N/status) and Process.PagedMemorySize64 returns VmSwap. So

currently in use (including swap) = Process.WorkingSet64 + Process.PagedMemorySize64 - (GC.GetGCMemoryInfo().TotalCommittedBytes - GC.GetTotalMemory(false))

right?

[loop-evgeny]

Hmm, it seems that on Windows Process.PagedMemorySize64 returns a very high value (> WorkingSet64), even when the process just started and there is no memory pressure, while PagedSystemMemorySize64 returns the same value as shown by Process Hacker as "Paged pool". On Linux they both return VmRSS. So it seems like the formula is then

currently in use (including swap) = Process.WorkingSet64 + Process.PagedSystemMemorySize64 - (GC.GetGCMemoryInfo().TotalCommittedBytes - GC.GetTotalMemory(false))

[GSPP]

as @PeterSolMS wrote in #52592 (comment), memory is not released to the OS until the server is low on RAM. (This is #37894, right?)

Also, note that the memory currently in use is not the same thing as "how much memory is needed?". These days, many libraries adjust their memory usage based on available memory (and thus trade off other performance characteristics, such as latency or throughput), so you might be trying to hit a moving target.

@Maoni0 There could be some merit in having GC return this kind of "held back" OS memory based on time. If I imagine a busy server running a bunch of .NET processes (incl. 3rd party applications, tools, tray icon applications, services, ...), the amount of "optional" commit could sum up.

Maybe the GC could make a push to release stuff based on time, for example, after a minute of relative idleness.

This is relevant for desktop machines as well. Process Explorer shows quite a few .NET processes (colored in yellow) on my desktop (which is currently running 300 processes).

Applications can do this cleanup themselves but most will not. If .NET included something sensible out of the box, that would help most machines converge to lower long-term memory usage. Defaults matter, and I have always appreciated the relatively low need to tune the .NET GC and to know its internals.

[loop-evgeny]

This new strategy of .NET hogging RAM until it needs to be released is still causing problems for us. Here is the RAM usage of an application that was moved from server A to server B to server C:

On the left "managed" is from GC.GetTotalMemory() and "used" is calculated per the earlier comment. On the right ("committed") is the Resident Set Size.

Server A (Ubuntu 18.04) has 1024 GB RAM, 316 GB free.
Server B (Ubuntu 22.04) has 1536 GB RAM, 391 GB free.
Server C (Ubuntu 22.04) has 128 GB RAM, 119 GB free.

We're seeing this effect with many instances of the application (with different traffic) when moved to server B, not just one. This makes it very difficult to utilize it efficiently, as we don't know how much more RAM it really has available before applications start slowing down. Historically, on .NET 3.1 and earlier, this happened at RSS around 70-80% of physical RAM. It's unclear where the threshold is for .NET 6.

Edit: Apparently this is not a new issue in .NET 6, but something about that server. The same happened with an old version of our application running on .NET 3.1.32 when moved between those same 3 servers: