Bug: Inaccurate reporting of available memory on linux

[amarsjac]

Expected Behaviour

Logging of system resource usage should accurately reflect state of the system. If logging shows "memory-used-percent": "85.0" but that 15% remaining is reserved and non-allocate-able by the OS this should be reflected in logs.

Purposed solution is to add new stat lines to the loging:

• memory-user-available-bytes = memory.total - total_memory_reserve_size
• memory-reserve-used-bytes = total_memory_reserve_size - available_memory_reserve_size
• memory-total-used-bytes = memory.total - memory.available
  Amend memory-used-bytes to be calculated by
• memory-used-bytes = memory.total - (memory.available - available_memory_reserve_size )

examples of the total_memory_reserve_size available_memory_reserve_size values can be derived from df bash command, example below on a stock al2023 ec2

sh-5.2$ df
Filesystem       1K-blocks    Used Available Use% Mounted on
devtmpfs              4096       0      4096   0% /dev
tmpfs              3999624       0   3999624   0% /dev/shm
tmpfs              1599852     424   1599428   1% /run
/dev/nvme0n1p1     8310764 1759672   6551092  22% /
tmpfs              3999628       0   3999628   0% /tmp
/dev/nvme0n1p128     10202    1310      8892  13% /boot/efi

total_memory_reserve_size = 1024 * 1599852
available_memory_reserve_size = 1024 * 1599428
I am unsure what the most pythonic way to derive these values or if there is a way to derive memory reserves values in an OS agnostic way

Current Behaviour

In the reporting of system resource usage psutils.virtual_memory.available reports total physical memory available and does not respect any memory that is unused but not allocate-able to the adaptor runtime or DCC. such as memory reserved be the OS for some purpose, for example Fedora derived linux distributions reserve around 20% of system memory for tmpsfs

Reproduction Steps

See psutil documentation for definitions of "total", "available", "used" and "free"

Code Snippet

short script that on linux gives expected memory usage information

python -c "import subprocess; import psutil;
stdout_string = (subprocess.run([\"df\", \"/run\"], capture_output=True, text=True)).stdout; 
total_memory_reserve_size = int(stdout_string.split('\n')[1].split()[1])*1024; 
available_memory_reserve_size  = int(stdout_string.split('\n')[1].split()[3])*1024; 
memory = psutil.virtual_memory();
print(\"memory-total-bytes:\", str(memory.total));
print(\"memory-user-available-bytes:\", str(memory.total - total_memory_reserve_size));
print(\"memory-total-used-bytes:\", str(memory.total - memory.available));
print(\"memory-used-bytes:\", str(memory.total - memory.available - (total_memory_reserve_size - available_memory_reserve_size  )));
print(\"memory-reserve-used-bytes:\", str(total_memory_reserve_size - available_memory_reserve_size));
print(\"memory-used-percent:\", str(memory.percent));
"