-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Resource Utilization - Memory not showing / 0 bytes #13023
Comments
Hi @EtienneBruines thanks for reporting, it does look like there may be a UI bug here. I do want to point out, starting in cgroups v2 the memory metrics reported by the kernel are more limited than they were with cgroups v1. This response object shows the difference:
That |
Any news about this issue? |
Experiencing this issue as well on Ubuntu 22.04 |
Also running into this issue with nomad |
There is some add'l info in the now closed #12088 (and folks are still pinging that bug) -- I'll just add that I am seeing this in NixOS 21.11 and 22.05, which are cgroups v2. |
I have just been testing to see if the latest version of nomad (1.3.2) fixes this issue as we first discovered it when flatcar upgraded to cgroupsv2 late last year. Currently we are using flatcar 3227.2.0 with cgroupsv2 enabled and we get no memory usage reported on tasks that run as a container. Switching back to cgroupsv1 and memory is reported correctly again. I have seen some release notes (for 1.3.0) that added cgroups v2 support for nomad client, but if the UI doesn't show the memory usage I would rather stick with cgroups v1 for the time being. When we initially rolled out the flatcar release using cgroupsv2, we found that some of our applications would crash with OOM issues, I am not certain if that is still happening, but the only other way we could tell if there was likely to be a memory issue was from checking memory usage reported in the UI.
I can see that with |
@philrenaud just FYI I took second look at this and updated #13670 with a slightly better fix, but I dunno why the test is failing or how to dig into it |
Hi @shoenig I don't think this is resolved, I just deployed a machine with 1.3.3 installed as a client and the server is also updated to 1.3.3, I updated my OS config to use cgroupsv2, and while I can see memory usage in nomad cli:
This is still not reflected in the UI: |
@Rumbles could you check the Devtools in the browser, to get the response to the call to Upgrading the nomad server 1.3.3 did fix the issue for me, so I'd be curious as to the values returned in that API call. (That should help determine whether it's a display issue or whether it's elsewhere.) |
I don't see where to check this in Web Developer Tools sorry @EtienneBruines:
|
Never mind, I tried in another browser, it was working there, I went back to my starting browser and it was working... I guess I wasn't patient enough :/ thanks for fixing this! |
Sorry for hijacking the thread but I'm having the same issue. {
"ResourceUsage": {
"CpuStats": {
"Measured": [
"Throttled Periods",
"Throttled Time",
"Percent"
],
"Percent": 0.10362745098039215,
"SystemMode": 4.541154210028382,
"ThrottledPeriods": 0,
"ThrottledTime": 0,
"TotalTicks": 5.025931372549019,
"UserMode": 1596.9725638599812
},
"DeviceStats": [],
"MemoryStats": {
"Cache": 0,
"KernelMaxUsage": 0,
"KernelUsage": 0,
"MappedFile": 0,
"MaxUsage": 0,
"Measured": [
"Cache",
"Swap",
"Usage"
],
"RSS": 0,
"Swap": 0,
"Usage": 292315136
}
},
"Tasks": {
"foo": {
"Pids": null,
"ResourceUsage": {
"CpuStats": {
"Measured": [
"Throttled Periods",
"Throttled Time",
"Percent"
],
"Percent": 0.10362745098039215,
"SystemMode": 4.541154210028382,
"ThrottledPeriods": 0,
"ThrottledTime": 0,
"TotalTicks": 5.025931372549019,
"UserMode": 1596.9725638599812
},
"DeviceStats": null,
"MemoryStats": {
"Cache": 0,
"KernelMaxUsage": 0,
"KernelUsage": 0,
"MappedFile": 0,
"MaxUsage": 0,
"Measured": [
"Cache",
"Swap",
"Usage"
],
"RSS": 0,
"Swap": 0,
"Usage": 292315136
}
},
"Timestamp": 1660560913426673700
}
},
"Timestamp": 1660560913426673700
} Nomad server version: |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.3.0 (52e95d6)
Operating system and Environment details
Ubuntu 22.04 LTS running cgroups2 and using the docker driver for the jobs.
Issue
The
Memory
usage is not showing for any allocation or task. This seems to be a UI bug.Reproduction steps
Run any task in docker and look at the graphs.
Expected Result
To see memory usage.
Actual Result
Job file (if appropriate)
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
Requests / Responses
There was a HTTP request to
/v1/client/allocation/968a938f-e05e-dbe3-216f-e56988f26ab4/stats
which returned the following:This is the same for all such requests - they all return pretty much
0
for everything except forUsage
(which does have a value).Seems like the API response is correct (it has a
Usage
value). Unsure ifMaxUsage
is supposed to be 0?The
docker stats
command also returns correct values, i.e.:Gathering the information through
nomad alloc status
returns correct values as well:When including
-stats
it even shows:The text was updated successfully, but these errors were encountered: