raw_exec graphs for cpu and memory are misleading

[shantanugadgil]

Nomad version

Nomad v1.2.6 (a6c6b47)

Operating system and Environment details

Amazon Linux 2, Ubuntu 16.04/18.04/20.04

Issue

Misleading CPU and Memory usage graphs when using raw_exec tasks.

Reproduction steps

Configure a raw_exec task with a very low value of cpu and memory like so:

      resources {
        cpu    = 32
        memory = 32
      }

When the process starts using much more memory than 32 MB and much more CPU than 32 MHz, the graph in the Nomad Gui looks misleading.

Expected Result

Maybe the raw_exec graphs could be plotted using the fingerprinted values of the host?

Actual Result

Screenshot:

Job file (if appropriate)

N/A

Nomad Server logs (if appropriate)

N/A

Nomad Client logs (if appropriate)

N/A

[shoenig]

Thanks for the report, @shantanugadgil ! Indeed this seems fairly trivial to reproduce

example jobfile

job "py" {
  datacenters = ["dc1"]
  type        = "service"

  group "py" {
    task "py" {
      driver = "raw_exec"
      user   = "nobody"
      config {
        command = "/usr/bin/python3"
        args    = ["-m", "http.server", "--directory", "/tmp"]
      }
      resources {
        memory = 12
        cpu    = 12
      }
    }
  }
}

Just run that and browse to the task in the UI.

[Lord-Y]

I'm having kind of the same problem with nomad 1.2.6 with docker image.

The memory usage is always having 0 bytes while docker stats tell totally the opposite:

CONTAINER ID   NAME                                          CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
9474873dc105   server-6eb853a9-8ad0-be54-edca-1b4a091b1a16   13.58%    252.7MiB / 256MiB   98.69%    72.5MB / 203MB   35.4MB / 50.3MB   48

In our job spec we reserve 256MB with no limit.
OS:

Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

We have no particular error in agent log in debug.

[Lord-Y]

Okay so I can see that my memory graph issue is related to cgroup v2 and will be fixed in nomad 1.3.0. More informations are here #10251. I'll wait for nomad 1.3.0