Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to launch command with executor: rpc error: #5576

Closed
sirkjohannsen opened this issue Apr 18, 2019 · 5 comments
Closed

failed to launch command with executor: rpc error: #5576

sirkjohannsen opened this issue Apr 18, 2019 · 5 comments
Labels
theme/driver/exec type/bug

Comments

@sirkjohannsen
Copy link

We are testing nomad 0.9 at the moment and discovered that many of the services that ran fine in 0.8 now do not start properly.

Nomad version

Nomad v0.9.0 (18dd59056ee1d7b2df51256fe900a98460d3d6b9)

Operating system and Environment details

  • Ubuntu 16.04
  • VM instance on GCP
  • CPU: 2x Intel(R) Xeon(R) CPU @ 2.60GHz
  • RAM: 8GB

linux version 4.15.0-1029-gcp (buildd@lgw01-amd64-006) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)) #31~16.04.1-Ubuntu SMP Fri Mar 22 13:06:42 UTC 2019

We are running nomad as client and server on single instance here (sandbox)

Issue

with the exec driver we experience these errors at random.
Restarts sometimes help to recover the issue but obviously breaks our deployment flow.

failed to launch command with executor: rpc error: code = Unknown desc = container process is already dead

or

failed to launch command with executor: rpc error: code = Unknown desc = cannot start an already running container

Reproduction steps

Its happening randomly but its happening all the time.

Job file (if appropriate)

one of many jobs where this randomly happens:

{
  "Stop": false,
  "Region": "eu",
  "Namespace": "default",
  "ID": "test-service",
  "ParentID": "",
  "Name": "test-service",
  "Type": "service",
  "Priority": 50,
  "AllAtOnce": false,
  "Datacenters": [
    "sandbox"
  ],
  "Constraints": [
    {
      "LTarget": "${meta.xyz}",
      "RTarget": "True",
      "Operand": "="
    },
    {
      "LTarget": "",
      "RTarget": "true",
      "Operand": "distinct_hosts"
    }
  ],
  "TaskGroups": [
    {
      "Name": "service",
      "Count": 1,
      "Constraints": null,
      "Tasks": [
        {
          "Name": "test",
          "Driver": "exec",
          "User": "www-data",
          "Config": {
            "args": [
              "-debug",
              "prometheus",
              "-listen_addr",
              ":${NOMAD_PORT_test_metrics}",
              "trace",
              "-collector_address",
              "localhost:9411",
              "log",
              "-environment",
              "sandbox",
              "-hostname",
              "${attr.unique.hostname}",
              "consul",
              "-uri",
              "http://localhost:8500",
              "rpc",
              "-address",
              ":${NOMAD_PORT_test_rpc}",
              "database",
              "-host",
              "localhost",
              "-user",
              "postgres",
              "-password",
              "",
              "-database",
              "sandbox"
            ],
            "command": "/usr/local/bin/test-service"
          },
          "Constraints": null,
          "Env": null,
          "Services": [
            {
              "Id": "",
              "Name": "test-service-metrics",
              "Tags": [
                "metrics",
                "test",
                "internal"
              ],
              "CanaryTags": null,
              "PortLabel": "test_metrics",
              "AddressMode": "auto",
              "Checks": [
                {
                  "Id": "",
                  "Name": "alive",
                  "Type": "tcp",
                  "Command": "",
                  "Args": null,
                  "Path": "",
                  "Protocol": "",
                  "PortLabel": "",
                  "AddressMode": "",
                  "Interval": 25000000000,
                  "Timeout": 2000000000,
                  "InitialStatus": "",
                  "TLSSkipVerify": false,
                  "Header": null,
                  "Method": "",
                  "CheckRestart": null,
                  "GRPCService": "",
                  "GRPCUseTLS": false
                }
              ],
              "CheckRestart": null
            },
            {
              "Id": "",
              "Name": "test-service-rpc",
              "Tags": [
                "test",
                "service",
                "internal"
              ],
              "CanaryTags": null,
              "PortLabel": "test_rpc",
              "AddressMode": "auto",
              "Checks": [
                {
                  "Id": "",
                  "Name": "service-health",
                  "Type": "grpc",
                  "Command": "",
                  "Args": null,
                  "Path": "",
                  "Protocol": "",
                  "PortLabel": "test_rpc",
                  "AddressMode": "",
                  "Interval": 25000000000,
                  "Timeout": 15000000000,
                  "InitialStatus": "",
                  "TLSSkipVerify": false,
                  "Header": null,
                  "Method": "",
                  "CheckRestart": null,
                  "GRPCService": "",
                  "GRPCUseTLS": false
                }
              ],
              "CheckRestart": null
            }
          ],
          "Resources": {
            "CPU": 20,
            "MemoryMB": 40,
            "DiskMB": 0,
            "IOPS": 0,
            "Networks": [
              {
                "Device": "",
                "CIDR": "",
                "IP": "",
                "MBits": 5,
                "ReservedPorts": null,
                "DynamicPorts": [
                  {
                    "Label": "test_rpc",
                    "Value": 0
                  },
                  {
                    "Label": "test_metrics",
                    "Value": 0
                  }
                ]
              }
            ]
          },
          "Meta": null,
          "KillTimeout": 5000000000,
          "LogConfig": {
            "MaxFiles": 20,
            "MaxFileSizeMB": 10
          },
          "Artifacts": [
            {
              "GetterSource": "s3::https://s3-eu-west-1.amazonaws.com/xxxxxxxx/test-service-01ba3eacaaeb9337da1a2e5cc18eff5c8f56c3e5.tar.xz",
              "GetterOptions": {
                "aws_access_key_id": "XXXXXXXXXX",
                "aws_access_key_secret": "XXXXXXXXXXXXXXXXXXXXXXXXX"
              },
              "GetterMode": "any",
              "RelativeDest": "local/"
            }
          ],
          "Vault": null,
          "Templates": null,
          "DispatchPayload": null,
          "Leader": false,
          "ShutdownDelay": 0,
          "KillSignal": "",
          "ID": "test-service.service.test"
        }
      ],
      "RestartPolicy": {
        "Interval": 300000000000,
        "Attempts": 5,
        "Delay": 5000000000,
        "Mode": "delay"
      },
      "ReschedulePolicy": {
        "Attempts": 0,
        "Interval": 0,
        "Delay": 30000000000,
        "DelayFunction": "exponential",
        "MaxDelay": 3600000000000,
        "Unlimited": true
      },
      "EphemeralDisk": {
        "Sticky": false,
        "Migrate": false,
        "SizeMB": 250
      },
      "Update": {
        "Stagger": 10000000000,
        "MaxParallel": 1,
        "HealthCheck": "task_states",
        "MinHealthyTime": 10000000000,
        "HealthyDeadline": 1500000000000,
        "ProgressDeadline": 1920000000000,
        "AutoRevert": false,
        "Canary": 0
      },
      "Migrate": {
        "MaxParallel": 1,
        "HealthCheck": "checks",
        "MinHealthyTime": 10000000000,
        "HealthyDeadline": 300000000000
      },
      "Meta": null,
      "ID": "test-service.service"
    }
  ],
  "Update": {
    "Stagger": 10000000000,
    "MaxParallel": 1,
    "HealthCheck": "",
    "MinHealthyTime": 0,
    "HealthyDeadline": 0,
    "ProgressDeadline": 0,
    "AutoRevert": false,
    "Canary": 0
  },
  "Periodic": null,
  "ParameterizedJob": null,
  "Dispatched": false,
  "Payload": null,
  "Reschedule": null,
  "Migrate": null,
  "Meta": null,
  "VaultToken": "",
  "Status": "running",
  "StatusDescription": "",
  "Stable": true,
  "Version": 2,
  "SubmitTime": 1555581591398130200,
  "CreateIndex": 53,
  "ModifyIndex": 147,
  "JobModifyIndex": 135
}

Nomad logs (if appropriate)

alloc-status:

ID                   = 42a141b6
Eval ID              = 21107ced
Name                 = test-service.service[0]
Node ID              = 1fb13a5d
Job ID               = test-service
Job Version          = 0
Client Status        = failed
Client Description   = Failed tasks
Desired Status       = run
Desired Description  = <none>
Created              = 29m7s ago
Modified             = 27m55s ago
Deployment ID        = 6e64f7f8
Deployment Health    = unhealthy
Replacement Alloc ID = 6e172afa

Task "test" is "dead"
Task Resources
CPU     Memory  Disk     Addresses
20 MHz  40 MiB  250 MiB  test_rpc: 10.132.0.14:21227
                                                             test_metrics: 10.132.0.14:23471

Task Events:
Started At     = N/A
Finished At    = 2019-04-18T09:57:39Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type                   Description
2019-04-18T09:57:39Z  Alloc Unhealthy        Unhealthy because of failed task
2019-04-18T09:57:39Z  Not Restarting         Error was unrecoverable
2019-04-18T09:57:39Z  Driver Failure         failed to launch command with executor: rpc error: code = Unknown desc = cannot start an already running container
2019-04-18T09:56:29Z  Downloading Artifacts  Client is downloading artifacts
2019-04-18T09:56:29Z  Task Setup             Building Task Directory
2019-04-18T09:56:29Z  Received               Task received by client

alloc status after retry:

ID                   = 6e172afa
Eval ID              = 296d51f0
Name                 = test-service.service[0]
Node ID              = 1fb13a5d
Job ID               = test-service
Job Version          = 0
Client Status        = failed
Client Description   = Failed tasks
Desired Status       = run
Desired Description  = <none>
Created              = 28m16s ago
Modified             = 27m42s ago
Deployment ID        = 6e64f7f8
Deployment Health    = unhealthy
Replacement Alloc ID = 72725256

Task "test" is "dead"
Task Resources
CPU       Memory         Disk     Addresses
0/20 MHz  48 KiB/40 MiB  250 MiB  test_rpc: 10.132.0.14:31086
                                                                      test_metrics: 10.132.0.14:27131

Task Events:
Started At     = 2019-04-18T09:58:37Z
Finished At    = 2019-04-18T09:58:42Z
Total Restarts = 1
Last Restart   = 2019-04-18T09:58:37Z

Recent Events:
Time                  Type                   Description
2019-04-18T09:58:42Z  Alloc Unhealthy        Unhealthy because of failed task
2019-04-18T09:58:42Z  Not Restarting         Error was unrecoverable
2019-04-18T09:58:42Z  Driver Failure         failed to launch command with executor: rpc error: code = Unknown desc = container process is already dead
2019-04-18T09:58:37Z  Restarting             Task restarting in 5.057962516s
2019-04-18T09:58:37Z  Terminated             Exit Code: 137, Signal: 9
2019-04-18T09:58:37Z  Started                Task started by client
2019-04-18T09:58:10Z  Downloading Artifacts  Client is downloading artifacts
2019-04-18T09:58:09Z  Task Setup             Building Task Directory
2019-04-18T09:58:09Z  Received               Task received by client

logs:

Apr 18 09:56:29 sb-nomad09 nomad[23445]:     2019-04-18T09:56:29.977Z [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test @module=logmon path=/var/nomad/alloc/42a141b6-7704-f0f1-139b-d4332c4c4c12/alloc/logs/.test.stdout.fifo timestamp=2019-04-18T09:56:29.977Z
Apr 18 09:56:29 sb-nomad09 nomad[23445]:     2019-04-18T09:56:29.978Z [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test @module=logmon path=/var/nomad/alloc/42a141b6-7704-f0f1-139b-d4332c4c4c12/alloc/logs/.test.stderr.fifo timestamp=2019-04-18T09:56:29.977Z
Apr 18 09:56:29 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test @module=logmon path=/var/nomad/alloc/42a141b6-7704-f0f1-139b-d4332c4c4c12/alloc/logs/.test.stdout.fifo timestamp=2019-04-18T09:56:29.977Z
Apr 18 09:56:29 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test @module=logmon path=/var/nomad/alloc/42a141b6-7704-f0f1-139b-d4332c4c4c12/alloc/logs/.test.stderr.fifo timestamp=2019-04-18T09:56:29.977Z
Apr 18 09:57:39 sb-nomad09 nomad[23445]:     2019-04-18T09:57:39.814Z [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test error="failed to launch command with executor: rpc error: code = Unknown desc = cannot start an already running container"
Apr 18 09:57:39 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner: running driver failed: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test error="failed to launch command with executor: rpc error: code = Unknown desc = cannot start an already running container"
Apr 18 09:57:39 sb-nomad09 nomad[23445]:     2019-04-18T09:57:39.816Z [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test reason="Error was unrecoverable"
Apr 18 09:57:39 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner: not restarting task: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test reason="Error was unrecoverable"
Apr 18 09:57:39 sb-nomad09 nomad[23445]:     2019-04-18T09:57:39.820Z [INFO ] client.gc: marking allocation for GC: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12
Apr 18 09:57:39 sb-nomad09 nomad[23445]: client.gc: marking allocation for GC: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12
Apr 18 09:57:40 sb-nomad09 nomad[23445]:     2019-04-18T09:57:40.059Z [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test error="read |0: file already closed"
Apr 18 09:57:40 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=42a141b6-7704-f0f1-139b-d4332c4c4c12 task=test error="read |0: file already closed"
Apr 18 10:13:07 sb-nomad09 nomad[23445]:     2019-04-18T10:13:07.697Z [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/42a141b6-7704-f0f1-139b-d4332c4c4c12?task=test&type=stdout&plain=false error="task "test" not started yet. No logs available" code=404
Apr 18 10:13:07 sb-nomad09 nomad[23445]: http: request failed: method=GET path=/v1/client/fs/logs/42a141b6-7704-f0f1-139b-d4332c4c4c12?task=test&type=stdout&plain=false error="task "test" not started yet. No logs available" code=404
Apr 18 10:13:07 sb-nomad09 nomad[23445]:     2019-04-18T10:13:07.699Z [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/42a141b6-7704-f0f1-139b-d4332c4c4c12?task=test&type=stderr&plain=false error="task "test" not started yet. No logs available" code=404
Apr 18 10:13:07 sb-nomad09 nomad[23445]: http: request failed: method=GET path=/v1/client/fs/logs/42a141b6-7704-f0f1-139b-d4332c4c4c12?task=test&type=stderr&plain=false error="task "test" not started yet. No logs available" code=404

2nd allocation (after retry):

Apr 18 09:58:09 sb-nomad09 nomad[23445]:     2019-04-18T09:58:09.995Z [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test @module=logmon path=/var/nomad/alloc/6e172afa-8cdd-97c1-052d-9e918ff736ec/alloc/logs/.test.stdout.fifo timestamp=2019-04-18T09:58:09.995Z
Apr 18 09:58:09 sb-nomad09 nomad[23445]:     2019-04-18T09:58:09.995Z [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test @module=logmon path=/var/nomad/alloc/6e172afa-8cdd-97c1-052d-9e918ff736ec/alloc/logs/.test.stderr.fifo timestamp=2019-04-18T09:58:09.995Z
Apr 18 09:58:09 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test @module=logmon path=/var/nomad/alloc/6e172afa-8cdd-97c1-052d-9e918ff736ec/alloc/logs/.test.stdout.fifo timestamp=2019-04-18T09:58:09.995Z
Apr 18 09:58:09 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test @module=logmon path=/var/nomad/alloc/6e172afa-8cdd-97c1-052d-9e918ff736ec/alloc/logs/.test.stderr.fifo timestamp=2019-04-18T09:58:09.995Z
Apr 18 09:58:37 sb-nomad09 nomad[23445]:     2019-04-18T09:58:37.040Z [ERROR] client.driver_mgr.exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec driver=exec task_name=test error="rpc error: code = Unavailable desc = transport is closing"
Apr 18 09:58:37 sb-nomad09 nomad[23445]: client.driver_mgr.exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec driver=exec task_name=test error="rpc error: code = Unavailable desc = transport is closing"
Apr 18 09:58:37 sb-nomad09 nomad[23445]:     2019-04-18T09:58:37.043Z [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test reason="Restart within policy" delay=5.057962516s
Apr 18 09:58:37 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner: restarting task: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test reason="Restart within policy" delay=5.057962516s
Apr 18 09:58:42 sb-nomad09 nomad[23445]:     2019-04-18T09:58:42.134Z [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test @module=logmon path=/var/nomad/alloc/6e172afa-8cdd-97c1-052d-9e918ff736ec/alloc/logs/.test.stdout.fifo timestamp=2019-04-18T09:58:42.134Z
Apr 18 09:58:42 sb-nomad09 nomad[23445]:     2019-04-18T09:58:42.134Z [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test @module=logmon path=/var/nomad/alloc/6e172afa-8cdd-97c1-052d-9e918ff736ec/alloc/logs/.test.stderr.fifo timestamp=2019-04-18T09:58:42.134Z
Apr 18 09:58:42 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test @module=logmon path=/var/nomad/alloc/6e172afa-8cdd-97c1-052d-9e918ff736ec/alloc/logs/.test.stdout.fifo timestamp=2019-04-18T09:58:42.134Z
Apr 18 09:58:42 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test @module=logmon path=/var/nomad/alloc/6e172afa-8cdd-97c1-052d-9e918ff736ec/alloc/logs/.test.stderr.fifo timestamp=2019-04-18T09:58:42.134Z
Apr 18 09:58:42 sb-nomad09 nomad[23445]:     2019-04-18T09:58:42.750Z [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test error="failed to launch command with executor: rpc error: code = Unknown desc = container process is already dead"
Apr 18 09:58:42 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner: running driver failed: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test error="failed to launch command with executor: rpc error: code = Unknown desc = container process is already dead"
Apr 18 09:58:42 sb-nomad09 nomad[23445]:     2019-04-18T09:58:42.753Z [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test reason="Error was unrecoverable"
Apr 18 09:58:42 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner: not restarting task: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test reason="Error was unrecoverable"
Apr 18 09:58:42 sb-nomad09 nomad[23445]:     2019-04-18T09:58:42.757Z [INFO ] client.gc: marking allocation for GC: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec
Apr 18 09:58:42 sb-nomad09 nomad[23445]: client.gc: marking allocation for GC: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec
Apr 18 09:58:42 sb-nomad09 nomad[23445]:     2019-04-18T09:58:42.765Z [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test error="read |0: file already closed"
Apr 18 09:58:42 sb-nomad09 nomad[23445]: client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=6e172afa-8cdd-97c1-052d-9e918ff736ec task=test error="read |0: file already closed"
@sirkjohannsen
Copy link
Author

This usually is accompanied by OOM messages in the Kernel:

Apr 18 09:31:06 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:31:06 sb-nomad09 kernel: runc:[2:INIT] cpuset÷1739d6-f219-f0b0-68ad-e8df1127f633 mems_allowed=0
Apr 18 09:31:06 sb-nomad09 kernel: CPU: 0 PID: 12513 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:31:06 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:31:06 sb-nomad09 kernel: Call Trace:
Apr 18 09:31:06 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:31:06 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:31:06 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:31:06 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:31:06 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:31:06 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:31:06 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:31:06 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:31:06 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:31:06 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:31:06 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:31:06 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:31:06 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:31:06 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:31:06 sb-nomad09 kernel: RIP: 0033:0x4622a3
Apr 18 09:31:06 sb-nomad09 kernel: RSP: 002b:00007ffef2fb02b8 EFLAGS: 00010206
Apr 18 09:31:06 sb-nomad09 kernel: RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000002000
Apr 18 09:31:06 sb-nomad09 kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000c0000e5000
Apr 18 09:31:06 sb-nomad09 kernel: RBP: 00007ffef2fb0300 R08: 00007ff2d2d0f270 R09: 0000000000000001
Apr 18 09:31:06 sb-nomad09 kernel: R10: 0000000000000072 R11: 0000000000000000 R12: 0000000000000000
Apr 18 09:31:06 sb-nomad09 kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 0000000000000002
Apr 18 09:31:06 sb-nomad09 kernel: Task in /nomad/f71739d6-f219-f0b0-68ad-e8df1127f633 killed as a result of limit of /nomad/f71739d6-f219-f0b0-68ad-e8df1127f633
Apr 18 09:31:06 sb-nomad09 kernel: memory: usage 30720kB, limit 30720kB, failcnt 11
Apr 18 09:31:06 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:31:06 sb-nomad09 kernel: kmem: usage 856kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:31:06 sb-nomad09 kernel: Memory cgroup stats for /nomad/f71739d6-f219-f0b0-68ad-e8df1127f633: cache:25216KB rss:4648KB rss_huge:0KB shmem:25204KB mapped_file:14692KB dirty:0KB writeback:0KB inactive_anon:15048KB active_anon:14804KB inactive_file:8KB active_file:4KB unevictable:0KB
Apr 18 09:31:06 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:31:06 sb-nomad09 kernel: [12513]     0 12513    73292     7848   217088        0             0 runc:[2:INIT]
Apr 18 09:31:06 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 12513 (runc:[2:INIT]) score 997 or sacrifice child
Apr 18 09:31:06 sb-nomad09 kernel: Killed process 12513 (runc:[2:INIT]) total-vm:293168kB, anon-rss:4496kB, file-rss:1076kB, shmem-rss:25820kB
Apr 18 09:31:06 sb-nomad09 kernel: oom_reaper: reaped process 12513 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 18 09:34:21 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:34:21 sb-nomad09 kernel: runc:[2:INIT] cpuset b142be-b338-92d8-fbe1-021811106d22 mems_allowed=0
Apr 18 09:34:21 sb-nomad09 kernel: CPU: 1 PID: 17586 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:34:21 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:34:21 sb-nomad09 kernel: Call Trace:
Apr 18 09:34:21 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:34:21 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:34:21 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:34:21 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:34:21 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:34:21 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:34:21 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:34:21 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:34:21 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:34:21 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:34:21 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:34:21 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:34:21 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:34:21 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:34:21 sb-nomad09 kernel: RIP: 0033:0x454d57
Apr 18 09:34:21 sb-nomad09 kernel: RSP: 002b:00007ffcc972ff48 EFLAGS: 00010293
Apr 18 09:34:21 sb-nomad09 kernel: RAX: 000000c000000300 RBX: 000000c0000ae7d0 RCX: 0000000000000003
Apr 18 09:34:21 sb-nomad09 kernel: RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
Apr 18 09:34:21 sb-nomad09 kernel: RBP: 00007ffcc9730240 R08: 0000000000000000 R09: 0000000000000000
Apr 18 09:34:21 sb-nomad09 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000002c6e0b8
Apr 18 09:34:21 sb-nomad09 kernel: R13: 0000000003669d20 R14: 000000c000000300 R15: 0000000000000003
Apr 18 09:34:21 sb-nomad09 kernel: Task in /nomad/15b142be-b338-92d8-fbe1-021811106d22 killed as a result of limit of /nomad/15b142be-b338-92d8-fbe1-021811106d22
Apr 18 09:34:21 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 32
Apr 18 09:34:21 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:34:21 sb-nomad09 kernel: kmem: usage 824kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:34:21 sb-nomad09 kernel: Memory cgroup stats for /nomad/15b142be-b338-92d8-fbe1-021811106d22: cache:35540KB rss:4596KB rss_huge:0KB shmem:35540KB mapped_file:15980KB dirty:0KB writeback:0KB inactive_anon:20712KB active_anon:19424KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:34:21 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:34:21 sb-nomad09 kernel: [17586]    33 17586    56908     7763   221184        0             0 runc:[2:INIT]
Apr 18 09:34:21 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 17586 (runc:[2:INIT]) score 740 or sacrifice child
Apr 18 09:34:21 sb-nomad09 kernel: Killed process 17586 (runc:[2:INIT]) total-vm:227632kB, anon-rss:4412kB, file-rss:912kB, shmem-rss:25728kB
Apr 18 09:34:21 sb-nomad09 kernel: oom_reaper: reaped process 17586 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 18 09:34:28 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:34:28 sb-nomad09 kernel:   cache: shmem_inode_cache(48614:a20bb32c-7f1c-4c66-ff3f-fbf7262f9929), object size: 704, buffer size: 712, default order: 2, min order: 0
Apr 18 09:34:28 sb-nomad09 kernel:   node 0: slabs: 1, objs: 23, free: 0
Apr 18 09:34:28 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:34:28 sb-nomad09 kernel: runc:[2:INIT] cpuset¢0bb32c-7f1c-4c66-ff3f-fbf7262f9929 mems_allowed=0
Apr 18 09:34:28 sb-nomad09 kernel: CPU: 0 PID: 17696 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:34:28 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:34:28 sb-nomad09 kernel: Call Trace:
Apr 18 09:34:28 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:34:28 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:34:28 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:34:28 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:34:28 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:34:28 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:34:28 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:34:28 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:34:28 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:34:28 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:34:28 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:34:28 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:34:28 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:34:28 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:34:28 sb-nomad09 kernel: RIP: 0033:0x454d57
Apr 18 09:34:28 sb-nomad09 kernel: RSP: 002b:00007ffd1a190f78 EFLAGS: 00010283
Apr 18 09:34:28 sb-nomad09 kernel: RAX: 000000c000000300 RBX: 000000c0001c5620 RCX: 0000000000000020
Apr 18 09:34:28 sb-nomad09 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Apr 18 09:34:28 sb-nomad09 kernel: RBP: 00007ffd1a191270 R08: 0000000000000000 R09: 0000000000000000
Apr 18 09:34:28 sb-nomad09 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000002c884e0
Apr 18 09:34:28 sb-nomad09 kernel: R13: 0000000003669d20 R14: 000000c000000300 R15: 0000000000000020
Apr 18 09:34:28 sb-nomad09 kernel: Task in /nomad/a20bb32c-7f1c-4c66-ff3f-fbf7262f9929 killed as a result of limit of /nomad/a20bb32c-7f1c-4c66-ff3f-fbf7262f9929
Apr 18 09:34:28 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 18
Apr 18 09:34:28 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:34:28 sb-nomad09 kernel: kmem: usage 836kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:34:28 sb-nomad09 kernel: Memory cgroup stats for /nomad/a20bb32c-7f1c-4c66-ff3f-fbf7262f9929: cache:35484KB rss:4640KB rss_huge:0KB shmem:35484KB mapped_file:18108KB dirty:0KB writeback:0KB inactive_anon:20132KB active_anon:19992KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:34:28 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:34:28 sb-nomad09 kernel: [17696]     0 17696    58957     8143   225280        0             0 runc:[2:INIT]
Apr 18 09:34:28 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 17696 (runc:[2:INIT]) score 776 or sacrifice child
Apr 18 09:34:28 sb-nomad09 kernel: Killed process 17696 (runc:[2:INIT]) total-vm:235828kB, anon-rss:4568kB, file-rss:1012kB, shmem-rss:26992kB
Apr 18 09:34:28 sb-nomad09 kernel: oom_reaper: reaped process 17696 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 18 09:35:52 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:35:52 sb-nomad09 kernel:   cache: proc_inode_cache(48768:98ec397d-e687-c4f2-813e-3c6de0987992), object size: 672, buffer size: 680, default order: 1, min order: 0
Apr 18 09:35:52 sb-nomad09 kernel:   node 0: slabs: 1, objs: 12, free: 0
Apr 18 09:35:52 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:35:52 sb-nomad09 kernel:   cache: proc_inode_cache(48768:98ec397d-e687-c4f2-813e-3c6de0987992), object size: 672, buffer size: 680, default order: 1, min order: 0
Apr 18 09:35:52 sb-nomad09 kernel:   node 0: slabs: 1, objs: 12, free: 0
Apr 18 09:35:52 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:35:52 sb-nomad09 kernel:   cache: proc_inode_cache(48768:98ec397d-e687-c4f2-813e-3c6de0987992), object size: 672, buffer size: 680, default order: 1, min order: 0
Apr 18 09:35:52 sb-nomad09 kernel:   node 0: slabs: 1, objs: 12, free: 0
Apr 18 09:35:52 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:35:52 sb-nomad09 kernel:   cache: proc_inode_cache(48768:98ec397d-e687-c4f2-813e-3c6de0987992), object size: 672, buffer size: 680, default order: 1, min order: 0
Apr 18 09:35:52 sb-nomad09 kernel:   node 0: slabs: 1, objs: 12, free: 0
Apr 18 09:35:52 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:35:52 sb-nomad09 kernel:   cache: proc_inode_cache(48768:98ec397d-e687-c4f2-813e-3c6de0987992), object size: 672, buffer size: 680, default order: 1, min order: 0
Apr 18 09:35:52 sb-nomad09 kernel:   node 0: slabs: 1, objs: 12, free: 0
Apr 18 09:35:52 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:35:52 sb-nomad09 kernel:   cache: proc_inode_cache(48768:98ec397d-e687-c4f2-813e-3c6de0987992), object size: 672, buffer size: 680, default order: 1, min order: 0
Apr 18 09:35:52 sb-nomad09 kernel:   node 0: slabs: 1, objs: 12, free: 0
Apr 18 09:35:52 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:35:52 sb-nomad09 kernel:   cache: proc_inode_cache(48768:98ec397d-e687-c4f2-813e-3c6de0987992), object size: 672, buffer size: 680, default order: 1, min order: 0
Apr 18 09:35:52 sb-nomad09 kernel:   node 0: slabs: 1, objs: 12, free: 0
Apr 18 09:35:52 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:35:52 sb-nomad09 kernel:   cache: proc_inode_cache(48768:98ec397d-e687-c4f2-813e-3c6de0987992), object size: 672, buffer size: 680, default order: 1, min order: 0
Apr 18 09:35:52 sb-nomad09 kernel:   node 0: slabs: 1, objs: 12, free: 0
Apr 18 09:35:52 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:35:52 sb-nomad09 kernel: runc:[2:INIT] cpuset˜ec397d-e687-c4f2-813e-3c6de0987992 mems_allowed=0
Apr 18 09:35:52 sb-nomad09 kernel: CPU: 0 PID: 19097 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:35:52 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:35:52 sb-nomad09 kernel: Call Trace:
Apr 18 09:35:52 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:35:52 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:35:52 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:35:52 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:35:52 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:35:52 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:35:52 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:35:52 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:35:52 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:35:52 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:35:52 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:35:52 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:35:52 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:35:52 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:35:52 sb-nomad09 kernel: RIP: 0033:0x462662
Apr 18 09:35:52 sb-nomad09 kernel: RSP: 002b:000000c0004cf820 EFLAGS: 00010246
Apr 18 09:35:52 sb-nomad09 kernel: RAX: 000000c000095000 RBX: 0000000000000080 RCX: 0000000000000080
Apr 18 09:35:52 sb-nomad09 kernel: RDX: 000000000056ab4c RSI: 000000c0000a8b80 RDI: 000000c000095000
Apr 18 09:35:52 sb-nomad09 kernel: RBP: 000000c0004cf880 R08: 0000000000000001 R09: 0000000000000000
Apr 18 09:35:52 sb-nomad09 kernel: R10: 000000c000095000 R11: 0000000000000000 R12: ffffffffffffffff
Apr 18 09:35:52 sb-nomad09 kernel: R13: 0000000000000011 R14: 0000000000000010 R15: 0000000000000020
Apr 18 09:35:52 sb-nomad09 kernel: Task in /nomad/98ec397d-e687-c4f2-813e-3c6de0987992 killed as a result of limit of /nomad/98ec397d-e687-c4f2-813e-3c6de0987992
Apr 18 09:35:52 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 91
Apr 18 09:35:52 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:35:52 sb-nomad09 kernel: kmem: usage 756kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:35:52 sb-nomad09 kernel: Memory cgroup stats for /nomad/98ec397d-e687-c4f2-813e-3c6de0987992: cache:35652KB rss:4552KB rss_huge:0KB shmem:35652KB mapped_file:14972KB dirty:0KB writeback:0KB inactive_anon:21484KB active_anon:18720KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:35:52 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:35:52 sb-nomad09 kernel: [19097]     0 19097    54859     7943   204800        0             0 runc:[2:INIT]
Apr 18 09:35:52 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 19097 (runc:[2:INIT]) score 757 or sacrifice child
Apr 18 09:35:52 sb-nomad09 kernel: Killed process 19097 (runc:[2:INIT]) total-vm:219436kB, anon-rss:4432kB, file-rss:912kB, shmem-rss:26428kB
Apr 18 09:35:52 sb-nomad09 kernel: oom_reaper: reaped process 19097 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 18 09:37:12 sb-nomad09 kernel: docker0: port 2(veth9a0e9b1) entered disabled state
Apr 18 09:37:12 sb-nomad09 kernel: veth0b58298: renamed from eth0
Apr 18 09:37:12 sb-nomad09 kernel: docker0: port 2(veth9a0e9b1) entered disabled state
Apr 18 09:37:12 sb-nomad09 kernel: device veth9a0e9b1 left promiscuous mode
Apr 18 09:37:12 sb-nomad09 kernel: docker0: port 2(veth9a0e9b1) entered disabled state
Apr 18 09:37:14 sb-nomad09 kernel: docker0: port 3(vethaf87d3e) entered disabled state
Apr 18 09:37:14 sb-nomad09 kernel: veth76bcc23: renamed from eth0
Apr 18 09:37:14 sb-nomad09 kernel: docker0: port 3(vethaf87d3e) entered disabled state
Apr 18 09:37:14 sb-nomad09 kernel: device vethaf87d3e left promiscuous mode
Apr 18 09:37:14 sb-nomad09 kernel: docker0: port 3(vethaf87d3e) entered disabled state
Apr 18 09:37:15 sb-nomad09 kernel: docker0: port 8(vethd3b05bd) entered disabled state
Apr 18 09:37:15 sb-nomad09 kernel: veth3a064a0: renamed from eth0
Apr 18 09:37:15 sb-nomad09 kernel: docker0: port 8(vethd3b05bd) entered disabled state
Apr 18 09:37:15 sb-nomad09 kernel: device vethd3b05bd left promiscuous mode
Apr 18 09:37:15 sb-nomad09 kernel: docker0: port 8(vethd3b05bd) entered disabled state
Apr 18 09:37:22 sb-nomad09 kernel: veth82f4a5d: renamed from eth0
Apr 18 09:37:22 sb-nomad09 kernel: docker0: port 1(veth2ad8aa8) entered disabled state
Apr 18 09:37:22 sb-nomad09 kernel: docker0: port 1(veth2ad8aa8) entered disabled state
Apr 18 09:37:22 sb-nomad09 kernel: device veth2ad8aa8 left promiscuous mode
Apr 18 09:37:22 sb-nomad09 kernel: docker0: port 1(veth2ad8aa8) entered disabled state
Apr 18 09:37:26 sb-nomad09 kernel: docker0: port 4(veth73dfcde) entered disabled state
Apr 18 09:37:26 sb-nomad09 kernel: veth35205cb: renamed from eth0
Apr 18 09:37:26 sb-nomad09 kernel: docker0: port 4(veth73dfcde) entered disabled state
Apr 18 09:37:26 sb-nomad09 kernel: device veth73dfcde left promiscuous mode
Apr 18 09:37:26 sb-nomad09 kernel: docker0: port 4(veth73dfcde) entered disabled state
Apr 18 09:37:37 sb-nomad09 kernel: docker0: port 6(veth266b95e) entered disabled state
Apr 18 09:37:37 sb-nomad09 kernel: vetha283064: renamed from eth0
Apr 18 09:37:37 sb-nomad09 kernel: docker0: port 6(veth266b95e) entered disabled state
Apr 18 09:37:37 sb-nomad09 kernel: device veth266b95e left promiscuous mode
Apr 18 09:37:37 sb-nomad09 kernel: docker0: port 6(veth266b95e) entered disabled state
Apr 18 09:37:38 sb-nomad09 kernel: docker0: port 5(veth037982f) entered disabled state
Apr 18 09:37:38 sb-nomad09 kernel: veth458fae1: renamed from eth0
Apr 18 09:37:38 sb-nomad09 kernel: docker0: port 5(veth037982f) entered disabled state
Apr 18 09:37:38 sb-nomad09 kernel: device veth037982f left promiscuous mode
Apr 18 09:37:38 sb-nomad09 kernel: docker0: port 5(veth037982f) entered disabled state
Apr 18 09:37:43 sb-nomad09 kernel: docker0: port 7(vethbb14476) entered disabled state
Apr 18 09:37:43 sb-nomad09 kernel: veth2caa676: renamed from eth0
Apr 18 09:37:43 sb-nomad09 kernel: docker0: port 7(vethbb14476) entered disabled state
Apr 18 09:37:43 sb-nomad09 kernel: device vethbb14476 left promiscuous mode
Apr 18 09:37:43 sb-nomad09 kernel: docker0: port 7(vethbb14476) entered disabled state
Apr 18 09:47:10 sb-nomad09 kernel: docker0: port 1(vethbbdf82a) entered blocking state
Apr 18 09:47:10 sb-nomad09 kernel: docker0: port 1(vethbbdf82a) entered disabled state
Apr 18 09:47:10 sb-nomad09 kernel: device vethbbdf82a entered promiscuous mode
Apr 18 09:47:10 sb-nomad09 kernel: IPv6: ADDRCONF(NETDEV_UP): vethbbdf82a: link is not ready
Apr 18 09:47:11 sb-nomad09 kernel: eth0: renamed from vethb697621
Apr 18 09:47:11 sb-nomad09 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethbbdf82a: link becomes ready
Apr 18 09:47:11 sb-nomad09 kernel: docker0: port 1(vethbbdf82a) entered blocking state
Apr 18 09:47:11 sb-nomad09 kernel: docker0: port 1(vethbbdf82a) entered forwarding state
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: SLUB: Unable to allocate memory on node -1, gfp=0x14000c0(GFP_KERNEL)
Apr 18 09:57:39 sb-nomad09 kernel:   cache: cred_jar(49123:ea4d6e12-b266-42af-efa3-5b45e53e46cd), object size: 168, buffer size: 192, default order: 0, min order: 0
Apr 18 09:57:39 sb-nomad09 kernel:   node 0: slabs: 2, objs: 42, free: 0
Apr 18 09:57:39 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:57:39 sb-nomad09 kernel: runc:[2:INIT] cpusetê4d6e12-b266-42af-efa3-5b45e53e46cd mems_allowed=0
Apr 18 09:57:39 sb-nomad09 kernel: CPU: 0 PID: 27630 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:57:39 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:57:39 sb-nomad09 kernel: Call Trace:
Apr 18 09:57:39 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:57:39 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:57:39 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:57:39 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:57:39 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:57:39 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:57:39 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:57:39 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:57:39 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:57:39 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:57:39 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:57:39 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:57:39 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:57:39 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:57:39 sb-nomad09 kernel: RIP: 0033:0x40604c
Apr 18 09:57:39 sb-nomad09 kernel: RSP: 002b:000000c0004c7698 EFLAGS: 00010283
Apr 18 09:57:39 sb-nomad09 kernel: RAX: 00000000033feff0 RBX: 0000000000000d8d RCX: 00000000033fe273
Apr 18 09:57:39 sb-nomad09 kernel: RDX: 0000000000000000 RSI: 00000000033fe273 RDI: 00000000033fe273
Apr 18 09:57:39 sb-nomad09 kernel: RBP: 000000c0004c76e8 R08: 000000c0004c76b8 R09: 00000000022b15c0
Apr 18 09:57:39 sb-nomad09 kernel: R10: 00000000009ccf8a R11: 00000000007a1216 R12: 0000000000000000
Apr 18 09:57:39 sb-nomad09 kernel: R13: 0000000000000003 R14: 000000c000000300 R15: 0000000000000003
Apr 18 09:57:39 sb-nomad09 kernel: Task in /nomad/ea4d6e12-b266-42af-efa3-5b45e53e46cd killed as a result of limit of /nomad/ea4d6e12-b266-42af-efa3-5b45e53e46cd
Apr 18 09:57:39 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 198
Apr 18 09:57:39 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:57:39 sb-nomad09 kernel: kmem: usage 728kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:57:39 sb-nomad09 kernel: Memory cgroup stats for /nomad/ea4d6e12-b266-42af-efa3-5b45e53e46cd: cache:35744KB rss:4488KB rss_huge:0KB shmem:35744KB mapped_file:14868KB dirty:0KB writeback:0KB inactive_anon:20856KB active_anon:19376KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:57:39 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:57:39 sb-nomad09 kernel: [27630]    33 27630    52810     8135   212992        0             0 runc:[2:INIT]
Apr 18 09:57:39 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 27630 (runc:[2:INIT]) score 775 or sacrifice child
Apr 18 09:57:39 sb-nomad09 kernel: Killed process 27630 (runc:[2:INIT]) total-vm:211240kB, anon-rss:4360kB, file-rss:996kB, shmem-rss:27184kB
Apr 18 09:57:39 sb-nomad09 kernel: oom_reaper: reaped process 27630 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 18 09:57:44 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:57:44 sb-nomad09 kernel: runc:[2:INIT] cpusetëb8fbe9-aef0-95e3-1429-7747f66b7a82 mems_allowed=0
Apr 18 09:57:44 sb-nomad09 kernel: CPU: 0 PID: 27688 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:57:44 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:57:44 sb-nomad09 kernel: Call Trace:
Apr 18 09:57:44 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:57:44 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:57:44 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:57:44 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:57:44 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:57:44 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:57:44 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:57:44 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:57:44 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:57:44 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:57:44 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:57:44 sb-nomad09 kernel:  ? task_work_run+0x7f/0xc0
Apr 18 09:57:44 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:57:44 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:57:44 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:57:44 sb-nomad09 kernel: RIP: 0033:0x454d57
Apr 18 09:57:44 sb-nomad09 kernel: RSP: 002b:00007ffdc3d83108 EFLAGS: 00010293
Apr 18 09:57:44 sb-nomad09 kernel: RAX: 000000c000000300 RBX: 000000c0002a4960 RCX: 0000000000000003
Apr 18 09:57:44 sb-nomad09 kernel: RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
Apr 18 09:57:44 sb-nomad09 kernel: RBP: 00007ffdc3d83400 R08: 0000000000000000 R09: 0000000000000000
Apr 18 09:57:44 sb-nomad09 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000002c6e0b8
Apr 18 09:57:44 sb-nomad09 kernel: R13: 0000000003669d20 R14: 000000c000000300 R15: 0000000000000003
Apr 18 09:57:44 sb-nomad09 kernel: Task in /nomad/ebb8fbe9-aef0-95e3-1429-7747f66b7a82 killed as a result of limit of /nomad/ebb8fbe9-aef0-95e3-1429-7747f66b7a82
Apr 18 09:57:44 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 31
Apr 18 09:57:44 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:57:44 sb-nomad09 kernel: kmem: usage 904kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:57:44 sb-nomad09 kernel: Memory cgroup stats for /nomad/ebb8fbe9-aef0-95e3-1429-7747f66b7a82: cache:35640KB rss:4416KB rss_huge:0KB shmem:35640KB mapped_file:23004KB dirty:0KB writeback:0KB inactive_anon:20124KB active_anon:19932KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:57:44 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:57:44 sb-nomad09 kernel: [27688]    33 27688    54507     7661   204800        0             0 runc:[2:INIT]
Apr 18 09:57:44 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 27688 (runc:[2:INIT]) score 730 or sacrifice child
Apr 18 09:57:44 sb-nomad09 kernel: Killed process 27688 (runc:[2:INIT]) total-vm:218028kB, anon-rss:4264kB, file-rss:860kB, shmem-rss:25520kB
Apr 18 09:57:45 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:57:45 sb-nomad09 kernel: runc:[2:INIT] cpusetÎ35e4b6-b21a-3c65-1035-ae4ddab40dda mems_allowed=0
Apr 18 09:57:45 sb-nomad09 kernel: CPU: 0 PID: 27789 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:57:45 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:57:45 sb-nomad09 kernel: Call Trace:
Apr 18 09:57:45 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:57:45 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:57:45 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:57:45 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:57:45 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:57:45 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:57:45 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:57:45 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:57:45 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:57:45 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:57:45 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:57:45 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:57:45 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:57:45 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:57:45 sb-nomad09 kernel: RIP: 0033:0x4622a3
Apr 18 09:57:45 sb-nomad09 kernel: RSP: 002b:00007fffee8fb218 EFLAGS: 00010297
Apr 18 09:57:45 sb-nomad09 kernel: RAX: 0000000000000000 RBX: 0000000000002000 RCX: 0000000000002000
Apr 18 09:57:45 sb-nomad09 kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000c0000dc000
Apr 18 09:57:45 sb-nomad09 kernel: RBP: 00007fffee8fb260 R08: 00007f6acbc91dd0 R09: 0000000000000001
Apr 18 09:57:45 sb-nomad09 kernel: R10: 000000000000006e R11: 0000000000000000 R12: 0000000000000000
Apr 18 09:57:45 sb-nomad09 kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 0000000000000002
Apr 18 09:57:45 sb-nomad09 kernel: Task in /nomad/ce35e4b6-b21a-3c65-1035-ae4ddab40dda killed as a result of limit of /nomad/ce35e4b6-b21a-3c65-1035-ae4ddab40dda
Apr 18 09:57:45 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 12
Apr 18 09:57:45 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:57:45 sb-nomad09 kernel: kmem: usage 832kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:57:45 sb-nomad09 kernel: Memory cgroup stats for /nomad/ce35e4b6-b21a-3c65-1035-ae4ddab40dda: cache:35508KB rss:4620KB rss_huge:0KB shmem:35508KB mapped_file:19580KB dirty:0KB writeback:0KB inactive_anon:20128KB active_anon:19988KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:57:45 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:57:45 sb-nomad09 kernel: [27789]     0 27789    54859     7771   221184        0             0 runc:[2:INIT]
Apr 18 09:57:45 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 27789 (runc:[2:INIT]) score 741 or sacrifice child
Apr 18 09:57:45 sb-nomad09 kernel: Killed process 27789 (runc:[2:INIT]) total-vm:219436kB, anon-rss:4428kB, file-rss:880kB, shmem-rss:25776kB
Apr 18 09:57:45 sb-nomad09 kernel: oom_reaper: reaped process 27789 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 18 09:57:50 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:57:50 sb-nomad09 kernel: runc:[2:INIT] cpuset cba8e3-3031-bbfa-c9b3-a5630489fb93 mems_allowed=0
Apr 18 09:57:50 sb-nomad09 kernel: CPU: 0 PID: 27855 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:57:50 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:57:50 sb-nomad09 kernel: Call Trace:
Apr 18 09:57:50 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:57:50 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:57:50 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:57:50 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:57:50 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:57:50 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:57:50 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:57:50 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:57:50 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:57:50 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:57:50 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:57:50 sb-nomad09 kernel:  ? task_work_run+0x7f/0xc0
Apr 18 09:57:50 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:57:50 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:57:50 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:57:50 sb-nomad09 kernel: RIP: 0033:0x454d57
Apr 18 09:57:50 sb-nomad09 kernel: RSP: 002b:00007ffe6fbf4a28 EFLAGS: 00010293
Apr 18 09:57:50 sb-nomad09 kernel: RAX: 000000c000000300 RBX: 000000c000322910 RCX: 0000000000000003
Apr 18 09:57:50 sb-nomad09 kernel: RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
Apr 18 09:57:50 sb-nomad09 kernel: RBP: 00007ffe6fbf4d20 R08: 0000000000000000 R09: 0000000000000000
Apr 18 09:57:50 sb-nomad09 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000002c6e0b8
Apr 18 09:57:50 sb-nomad09 kernel: R13: 0000000003669d20 R14: 000000c000000300 R15: 0000000000000003
Apr 18 09:57:50 sb-nomad09 kernel: Task in /nomad/90cba8e3-3031-bbfa-c9b3-a5630489fb93 killed as a result of limit of /nomad/90cba8e3-3031-bbfa-c9b3-a5630489fb93
Apr 18 09:57:50 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 25
Apr 18 09:57:50 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:57:50 sb-nomad09 kernel: kmem: usage 972kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:57:50 sb-nomad09 kernel: Memory cgroup stats for /nomad/90cba8e3-3031-bbfa-c9b3-a5630489fb93: cache:35360KB rss:4628KB rss_huge:0KB shmem:35360KB mapped_file:22644KB dirty:0KB writeback:0KB inactive_anon:20068KB active_anon:19904KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:57:50 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:57:50 sb-nomad09 kernel: [27855]    33 27855    58957     8201   233472        0             0 runc:[2:INIT]
Apr 18 09:57:50 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 27855 (runc:[2:INIT]) score 782 or sacrifice child
Apr 18 09:57:50 sb-nomad09 kernel: Killed process 27855 (runc:[2:INIT]) total-vm:235828kB, anon-rss:4600kB, file-rss:976kB, shmem-rss:27228kB
Apr 18 09:58:36 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:58:36 sb-nomad09 kernel: runc:[2:INIT] cpusetÞ25a563-2409-843f-08ad-5bc6053df174 mems_allowed=0
Apr 18 09:58:36 sb-nomad09 kernel: CPU: 1 PID: 28042 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:58:36 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:58:36 sb-nomad09 kernel: Call Trace:
Apr 18 09:58:36 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:58:36 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:58:36 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:58:36 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:58:36 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:58:36 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:58:36 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:58:36 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:58:36 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:58:36 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:58:36 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:58:36 sb-nomad09 kernel:  ? task_work_run+0x7f/0xc0
Apr 18 09:58:36 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:58:36 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:58:36 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:58:36 sb-nomad09 kernel: RIP: 0033:0x40604c
Apr 18 09:58:36 sb-nomad09 kernel: RSP: 002b:000000c0004d7698 EFLAGS: 00010283
Apr 18 09:58:36 sb-nomad09 kernel: RAX: 000000000341fff0 RBX: 0000000000000978 RCX: 000000000341f688
Apr 18 09:58:36 sb-nomad09 kernel: RDX: 0000000000000000 RSI: 000000000341f688 RDI: 000000000341f688
Apr 18 09:58:36 sb-nomad09 kernel: RBP: 000000c0004d76e8 R08: 000000c0004d76b8 R09: 0000000000000001
Apr 18 09:58:36 sb-nomad09 kernel: R10: 000000000113156b R11: 000000000003cc37 R12: 0000000000000000
Apr 18 09:58:36 sb-nomad09 kernel: R13: 0000000000000003 R14: 000000c000000300 R15: 0000000000000003
Apr 18 09:58:36 sb-nomad09 kernel: Task in /nomad/de25a563-2409-843f-08ad-5bc6053df174 killed as a result of limit of /nomad/de25a563-2409-843f-08ad-5bc6053df174
Apr 18 09:58:36 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 28
Apr 18 09:58:36 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:58:36 sb-nomad09 kernel: kmem: usage 892kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:58:36 sb-nomad09 kernel: Memory cgroup stats for /nomad/de25a563-2409-843f-08ad-5bc6053df174: cache:35124KB rss:4944KB rss_huge:0KB shmem:35124KB mapped_file:25060KB dirty:0KB writeback:0KB inactive_anon:20160KB active_anon:19908KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:58:36 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:58:36 sb-nomad09 kernel: [28042]    33 28042    56908     8268   212992        0             0 runc:[2:INIT]
Apr 18 09:58:36 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 28042 (runc:[2:INIT]) score 788 or sacrifice child
Apr 18 09:58:36 sb-nomad09 kernel: Killed process 28042 (runc:[2:INIT]) total-vm:227632kB, anon-rss:4872kB, file-rss:1068kB, shmem-rss:27132kB
Apr 18 09:58:36 sb-nomad09 kernel: oom_reaper: reaped process 28042 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 18 09:58:42 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:58:42 sb-nomad09 kernel: runc:[2:INIT] cpusetæ72655b-a757-796d-9ef8-3f3f50e1f9d5 mems_allowed=0
Apr 18 09:58:42 sb-nomad09 kernel: CPU: 0 PID: 28090 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:58:42 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:58:42 sb-nomad09 kernel: Call Trace:
Apr 18 09:58:42 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:58:42 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:58:42 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:58:42 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:58:42 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:58:42 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:58:42 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:58:42 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:58:42 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:58:42 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:58:42 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:58:42 sb-nomad09 kernel:  ? task_work_run+0x7f/0xc0
Apr 18 09:58:42 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:58:42 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:58:42 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:58:42 sb-nomad09 kernel: RIP: 0033:0x454d57
Apr 18 09:58:42 sb-nomad09 kernel: RSP: 002b:00007ffc05760978 EFLAGS: 00010297
Apr 18 09:58:42 sb-nomad09 kernel: RAX: 000000c000000300 RBX: 000000c0000a5640 RCX: 0000000000000003
Apr 18 09:58:42 sb-nomad09 kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
Apr 18 09:58:42 sb-nomad09 kernel: RBP: 00007ffc05760c70 R08: 0000000000000000 R09: 0000000000000000
Apr 18 09:58:42 sb-nomad09 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000002c7daf0
Apr 18 09:58:42 sb-nomad09 kernel: R13: 0000000003669d20 R14: 000000c000000300 R15: 0000000000000003
Apr 18 09:58:42 sb-nomad09 kernel: Task in /nomad/e672655b-a757-796d-9ef8-3f3f50e1f9d5 killed as a result of limit of /nomad/e672655b-a757-796d-9ef8-3f3f50e1f9d5
Apr 18 09:58:42 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 33
Apr 18 09:58:42 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:58:42 sb-nomad09 kernel: kmem: usage 940kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:58:42 sb-nomad09 kernel: Memory cgroup stats for /nomad/e672655b-a757-796d-9ef8-3f3f50e1f9d5: cache:35460KB rss:4560KB rss_huge:0KB shmem:35460KB mapped_file:23596KB dirty:0KB writeback:0KB inactive_anon:20004KB active_anon:20004KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:58:42 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:58:42 sb-nomad09 kernel: [28090]    33 28090    77390     8006   229376        0             0 runc:[2:INIT]
Apr 18 09:58:42 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 28090 (runc:[2:INIT]) score 763 or sacrifice child
Apr 18 09:58:42 sb-nomad09 kernel: Killed process 28090 (runc:[2:INIT]) total-vm:309560kB, anon-rss:4460kB, file-rss:944kB, shmem-rss:26620kB
Apr 18 09:58:42 sb-nomad09 kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Apr 18 09:58:42 sb-nomad09 kernel: runc:[2:INIT] cpuset fb6886-6e8a-e534-687d-0bf0d9e1cdfe mems_allowed=0
Apr 18 09:58:42 sb-nomad09 kernel: CPU: 1 PID: 28134 Comm: runc:[2:INIT] Not tainted 4.15.0-1029-gcp #31~16.04.1-Ubuntu
Apr 18 09:58:42 sb-nomad09 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Apr 18 09:58:42 sb-nomad09 kernel: Call Trace:
Apr 18 09:58:42 sb-nomad09 kernel:  dump_stack+0x85/0xcb
Apr 18 09:58:42 sb-nomad09 kernel:  dump_header+0x77/0x285
Apr 18 09:58:42 sb-nomad09 kernel:  oom_kill_process+0x22e/0x450
Apr 18 09:58:42 sb-nomad09 kernel:  out_of_memory+0x11d/0x4c0
Apr 18 09:58:42 sb-nomad09 kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Apr 18 09:58:42 sb-nomad09 kernel:  mem_cgroup_oom_synchronize+0x32a/0x350
Apr 18 09:58:42 sb-nomad09 kernel:  ? mem_cgroup_css_reset+0xe0/0xe0
Apr 18 09:58:42 sb-nomad09 kernel:  pagefault_out_of_memory+0x36/0x7b
Apr 18 09:58:42 sb-nomad09 kernel:  mm_fault_error+0x8f/0x190
Apr 18 09:58:42 sb-nomad09 kernel:  ? handle_mm_fault+0xcc/0x1c0
Apr 18 09:58:42 sb-nomad09 kernel:  __do_page_fault+0x4d7/0x500
Apr 18 09:58:42 sb-nomad09 kernel:  ? vfs_read+0x11d/0x130
Apr 18 09:58:42 sb-nomad09 kernel:  do_page_fault+0x2e/0xf0
Apr 18 09:58:42 sb-nomad09 kernel:  ? page_fault+0x2f/0x50
Apr 18 09:58:42 sb-nomad09 kernel:  page_fault+0x45/0x50
Apr 18 09:58:42 sb-nomad09 kernel: RIP: 0033:0xfec3f6
Apr 18 09:58:42 sb-nomad09 kernel: RSP: 002b:000000c0004c6dc8 EFLAGS: 00010246
Apr 18 09:58:42 sb-nomad09 kernel: RAX: 000000c000085000 RBX: 000000c0004c6fb0 RCX: 0000000000000007
Apr 18 09:58:42 sb-nomad09 kernel: RDX: 0000000000000007 RSI: 000000c000123780 RDI: 0000000000000002
Apr 18 09:58:42 sb-nomad09 kernel: RBP: 000000c0004c6e48 R08: 000000000000000a R09: 000000c0001b68b3
Apr 18 09:58:42 sb-nomad09 kernel: R10: 0000000000000005 R11: 000000c000085000 R12: 0000000001a24c40
Apr 18 09:58:42 sb-nomad09 kernel: R13: 000000005a175aed R14: 0000000001a24c40 R15: 0000000000000038
Apr 18 09:58:42 sb-nomad09 kernel: Task in /nomad/14fb6886-6e8a-e534-687d-0bf0d9e1cdfe killed as a result of limit of /nomad/14fb6886-6e8a-e534-687d-0bf0d9e1cdfe
Apr 18 09:58:42 sb-nomad09 kernel: memory: usage 40960kB, limit 40960kB, failcnt 12
Apr 18 09:58:42 sb-nomad09 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:58:42 sb-nomad09 kernel: kmem: usage 828kB, limit 9007199254740988kB, failcnt 0
Apr 18 09:58:42 sb-nomad09 kernel: Memory cgroup stats for /nomad/14fb6886-6e8a-e534-687d-0bf0d9e1cdfe: cache:35512KB rss:4620KB rss_huge:0KB shmem:35512KB mapped_file:19952KB dirty:0KB writeback:0KB inactive_anon:20132KB active_anon:20000KB inactive_file:0KB active_file:0KB unevictable:0KB
Apr 18 09:58:42 sb-nomad09 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 18 09:58:42 sb-nomad09 kernel: [28134]     0 28134    68842     7724   217088        0             0 runc:[2:INIT]
Apr 18 09:58:42 sb-nomad09 kernel: Memory cgroup out of memory: Kill process 28134 (runc:[2:INIT]) score 736 or sacrifice child
Apr 18 09:58:42 sb-nomad09 kernel: Killed process 28134 (runc:[2:INIT]) total-vm:275368kB, anon-rss:4452kB, file-rss:912kB, shmem-rss:25532kB
Apr 18 09:58:42 sb-nomad09 kernel: oom_reaper: reaped process 28134 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
-- cursor: sÂd284497420495e855994e92198b5e7;i94191;b 01cc79a56c4ce8acd614c62b018a2b;m&922da378;tX6cb0c13322d;xØ7ca4c56c8e56bf

@preetapan
Copy link
Contributor

@sirkjohannsen Sorry that you are experiencing this. We did refactor Nomad's client and executor in 0.9 for supporting runtime plugins, and currently the RSS utilization for just the Nomad executor (used in exec driver jobs) ranges from 35MB to 45MB. I see that your Resource stanza limits to 40MB and while that might have been just enough in Nomad 0.8.7, 0.9 is seeing more OOM killing. I suggest increasing it to 60MB as a workaround.

We are aware of this and are investigating ways to reduce memory utilization. Related tickets- #4491 and #4495

I am going to close this one out, feel free to reopen if increasing memory in your resource stanza doesn't work.

@notnoop
Copy link
Contributor

notnoop commented May 31, 2019

@sirkjohannsen Sorry for you experiencing this again. We believe Nomad 0.9.2 is going to address this issue. The underlying cause was a CVE fix in runc [1] that was fixed in runc/libcontainer [2] and we picked it up in [3]. With 0.9.2, you can run tasks with as little memory as 10MB (the minimum for scheduling). Please try the RC1[4] and let us know your experience.

[1] opencontainers/runc#1980
[2] opencontainers/runc#1984
[3] #5437
[4] https://releases.hashicorp.com/nomad/0.9.2-rc1/

@sirkjohannsen
Copy link
Author

@notnoop thank you very much. First tests with nomad-0.9.2-rc1 are very promising. We were able to fully deploy our test environment without modifying the resource stanza.

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
theme/driver/exec type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@notnoop @preetapan @endocrimes @sirkjohannsen