Fix available memory extraction on Linux #26764
Conversation
The GlobalMemoryStatusEx in PAL is returning number of free physical pages in the ullAvailPhys member. But there are additional pages that are allocated as buffers and caches that get released when there is a memory pressure and thus they are effectively available too. This change extracts the available memory on Linux from the /proc/meminfo MemAvailable row, which is reported by the kernel as the most precise amount of available memory.
|
cc: @kevingosse, @Maoni0, @sergiy-k |
|
Do we need a matching change for standalone GC PAL? |
Yes, we will need it. In fact, I've found that the standalone GC uses a completely different way to get the physical memory used, which is in fact even more incorrect (it is based on memory consumed by the current process instead of the system one). I'll extend this PR with the standalone GC fix. |
| uint64_t available; | ||
| int fieldsParsed = sscanf(line, "MemAvailable: %" SCNu64 " %cB", &available, &units); | ||
|
|
||
| if (fieldsParsed >= 1) |
There was a problem hiding this comment.
Should this be >=2 since we need to parse values for 2 fields?
There was a problem hiding this comment.
No, since the units fields can be missing. So we will either get just the "available" or "available" and "units".
janvorli
added
the
* NO MERGE *
|
I still need to test the standalone GC version, I just needed to push it so that I can fetch the change from my various distros / OS-es. So I am marking it as no merge until I verify that it works as expected on both Linux and OSX. |
|
have you done the perf check with the GC perf infra? @andy-ms added the linux container support |
Is there any doc on how to do that? |
|
@janvorli I'll message you on Teams. |
The check for whether the memory was restricted or not was incorrect.
| static bool ReadMemAvailable(uint64_t* memAvailable) | ||
| { | ||
| bool foundMemAvailable = false; | ||
| FILE* memInfoFile = fopen("/proc/meminfo", "r"); |
There was a problem hiding this comment.
While this is fine -- you could also use sysinfo(2) here, which obtains the information from the same place that procfs uses to populate /proc/meminfo, but should work where procfs isn't available for whatever reason.
There was a problem hiding this comment.
Oh, nevermind, saw it's being used down the line. :)
There was a problem hiding this comment.
Actually, we use sysinfo only to extract the swap size. But looking at the sysinfo, it doesn't seem to include the equivalent of the MemAvailable.
| return 1; | ||
| } | ||
|
|
||
| #ifndef __APPLE__ |
There was a problem hiding this comment.
Isn't it better to use platform-specific #ifdefs here? This should be built on FreeBSD as well, where it's going to fail during runtime (and when it's time to add support to other OSes, it's easier to have the preprocessor #error on an #else).
There was a problem hiding this comment.
It seems that procfs exists on other Unix systems too. And here we just try once and then fall back to using sysconf, so we don't fail.
janvorli
removed
the
* NO MERGE *
* Fix available memory extraction on Linux The GlobalMemoryStatusEx in PAL is returning number of free physical pages in the ullAvailPhys member. But there are additional pages that are allocated as buffers and caches that get released when there is a memory pressure and thus they are effectively available too. This change extracts the available memory on Linux from the /proc/meminfo MemAvailable row, which is reported by the kernel as the most precise amount of available memory.
* Fix available memory extraction on Linux The GlobalMemoryStatusEx in PAL is returning number of free physical pages in the ullAvailPhys member. But there are additional pages that are allocated as buffers and caches that get released when there is a memory pressure and thus they are effectively available too. This change extracts the available memory on Linux from the /proc/meminfo MemAvailable row, which is reported by the kernel as the most precise amount of available memory. Commit migrated from dotnet/coreclr@859f464
* Fix available memory extraction on Linux The GlobalMemoryStatusEx in PAL is returning number of free physical pages in the ullAvailPhys member. But there are additional pages that are allocated as buffers and caches that get released when there is a memory pressure and thus they are effectively available too. This change extracts the available memory on Linux from the /proc/meminfo MemAvailable row, which is reported by the kernel as the most precise amount of available memory. Commit migrated from dotnet/coreclr@859f464
* Fix available memory extraction on Linux The GlobalMemoryStatusEx in PAL is returning number of free physical pages in the ullAvailPhys member. But there are additional pages that are allocated as buffers and caches that get released when there is a memory pressure and thus they are effectively available too. This change extracts the available memory on Linux from the /proc/meminfo MemAvailable row, which is reported by the kernel as the most precise amount of available memory. Commit migrated from dotnet/coreclr@859f464
The GlobalMemoryStatusEx in PAL is returning number of free physical pages in
the ullAvailPhys member. But there are additional pages that are allocated
as buffers and caches that get released when there is a memory pressure and
thus they are effectively available too.
This change extracts the available memory on Linux from the /proc/meminfo
MemAvailable row, which is reported by the kernel as the most precise
amount of available memory.
Close #26568