Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic in os/user.Lookup CGO call #12342

Closed
jwz-ecust opened this issue Mar 22, 2022 · 5 comments
Closed

panic in os/user.Lookup CGO call #12342

jwz-ecust opened this issue Mar 22, 2022 · 5 comments
Assignees
@tgross
Labels
stage/waiting-reply theme/client theme/crash type/bug
Projects
Nomad - Community Issues Triage

Comments

@jwz-ecust
Copy link

Nomad version

Nomad v1.2.6

Operating system and Environment details

x86 ubuntu

Issue

Nomad collapsed when concurrently requesting nomad API on the same node.

Expected Result

The first request returned 200 and the second returned 200 too.

Actual Result

The first request returned 200 but the second returned 500.

Job file (if appropriate)

{
  "Stop": false,
  "Region": "global",
  "Namespace": "kube-system",
  "ID": "sly-test-simple2-kube-system-e9b9ado",
  "ParentID": "",
  "Name": "sly-test-simple2-kube-system-e9b9ado",
  "Type": "batch",
  "Priority": 50,
  "AllAtOnce": false,
  "Datacenters": [
    "dc1"
  ],
  "Constraints": [
    {
      "LTarget": "${node.unique.name}",
      "RTarget": "mgt04",
      "Operand": "="
    }
  ],
  "Affinities": null,
  "Spreads": null,
  "TaskGroups": [
    {
      "Name": "step1",
      "Count": 0,
      "Update": null,
      "Migrate": null,
      "Constraints": null,
      "Scaling": null,
      "RestartPolicy": {
        "Attempts": 3,
        "Interval": 86400000000000,
        "Delay": 15000000000,
        "Mode": "fail"
      },
      "Tasks": [
        {
          "Name": "task",
          "Driver": "raw_exec",
          "User": "",
          "Config": {
            "args": [
              "-c",
              "echo hello"
            ],
            "command": "bash"
          },
          "Env": null,
          "Services": null,
          "Vault": null,
          "Templates": null,
          "Constraints": null,
          "Affinities": null,
          "Resources": {
            "CPU": 100,
            "Cores": 0,
            "MemoryMB": 300,
            "MemoryMaxMB": 0,
            "DiskMB": 0,
            "IOPS": 0,
            "Networks": null,
            "Devices": null
          },
          "RestartPolicy": {
            "Attempts": 3,
            "Interval": 86400000000000,
            "Delay": 15000000000,
            "Mode": "fail"
          },
          "DispatchPayload": null,
          "Lifecycle": null,
          "Meta": null,
          "KillTimeout": 5000000000,
          "LogConfig": {
            "MaxFiles": 10,
            "MaxFileSizeMB": 10
          },
          "Artifacts": null,
          "Leader": false,
          "ShutdownDelay": 0,
          "VolumeMounts": null,
          "ScalingPolicies": null,
          "KillSignal": "",
          "Kind": "",
          "CSIPluginConfig": null
        },
        {
          "Name": "control",
          "Driver": "docker",
          "User": "",
          "Config": {
            "image": "docker.inspur.com:5000/lma/taskmanager-api-dag:1.0.0-20220228",
            "args": []
          },
          "Env": {
            "NOMAD_ADDR": "10.110.29.1:4646"
          },
          "Services": null,
          "Vault": null,
          "Templates": null,
          "Constraints": null,
          "Affinities": null,
          "Resources": {
            "CPU": 100,
            "Cores": 0,
            "MemoryMB": 300,
            "MemoryMaxMB": 0,
            "DiskMB": 0,
            "IOPS": 0,
            "Networks": null,
            "Devices": null
          },
          "RestartPolicy": {
            "Attempts": 3,
            "Interval": 86400000000000,
            "Delay": 15000000000,
            "Mode": "fail"
          },
          "DispatchPayload": null,
          "Lifecycle": {
            "Hook": "poststop",
            "Sidecar": false
          },
          "Meta": null,
          "KillTimeout": 5000000000,
          "LogConfig": {
            "MaxFiles": 10,
            "MaxFileSizeMB": 10
          },
          "Artifacts": null,
          "Leader": false,
          "ShutdownDelay": 0,
          "VolumeMounts": null,
          "ScalingPolicies": null,
          "KillSignal": "",
          "Kind": "",
          "CSIPluginConfig": null
        }
      ],
      "EphemeralDisk": {
        "Sticky": false,
        "SizeMB": 300,
        "Migrate": false
      },
      "Meta": null,
      "ReschedulePolicy": {
        "Attempts": 1,
        "Interval": 86400000000000,
        "Delay": 5000000000,
        "DelayFunction": "constant",
        "MaxDelay": 0,
        "Unlimited": false
      },
      "Affinities": null,
      "Spreads": null,
      "Networks": null,
      "Consul": {
        "Namespace": ""
      },
      "Services": null,
      "Volumes": null,
      "ShutdownDelay": null,
      "StopAfterClientDisconnect": null
    },
    {
      "Name": "control",
      "Count": 0,
      "Update": null,
      "Migrate": null,
      "Constraints": null,
      "Scaling": null,
      "RestartPolicy": {
        "Attempts": 3,
        "Interval": 86400000000000,
        "Delay": 15000000000,
        "Mode": "fail"
      },
      "Tasks": [
        {
          "Name": "control",
          "Driver": "docker",
          "User": "",
          "Config": {
            "args": [
              "step1"
            ],
            "image": "docker.inspur.com:5000/lma/taskmanager-api-dag:1.0.0-20220228"
          },
          "Env": {
            "NOMAD_ADDR": "10.110.29.1:4646"
          },
          "Services": null,
          "Vault": null,
          "Templates": null,
          "Constraints": null,
          "Affinities": null,
          "Resources": {
            "CPU": 100,
            "Cores": 0,
            "MemoryMB": 300,
            "MemoryMaxMB": 0,
            "DiskMB": 0,
            "IOPS": 0,
            "Networks": null,
            "Devices": null
          },
          "RestartPolicy": {
            "Attempts": 3,
            "Interval": 86400000000000,
            "Delay": 15000000000,
            "Mode": "fail"
          },
          "DispatchPayload": null,
          "Lifecycle": null,
          "Meta": null,
          "KillTimeout": 5000000000,
          "LogConfig": {
            "MaxFiles": 10,
            "MaxFileSizeMB": 10
          },
          "Artifacts": null,
          "Leader": false,
          "ShutdownDelay": 0,
          "VolumeMounts": null,
          "ScalingPolicies": null,
          "KillSignal": "",
          "Kind": "",
          "CSIPluginConfig": null
        }
      ],
      "EphemeralDisk": {
        "Sticky": false,
        "SizeMB": 300,
        "Migrate": false
      },
      "Meta": null,
      "ReschedulePolicy": {
        "Attempts": 1,
        "Interval": 86400000000000,
        "Delay": 5000000000,
        "DelayFunction": "constant",
        "MaxDelay": 0,
        "Unlimited": false
      },
      "Affinities": null,
      "Spreads": null,
      "Networks": null,
      "Consul": {
        "Namespace": ""
      },
      "Services": null,
      "Volumes": null,
      "ShutdownDelay": null,
      "StopAfterClientDisconnect": null
    }
  ],
  "Update": {
    "Stagger": 0,
    "MaxParallel": 0,
    "HealthCheck": "",
    "MinHealthyTime": 0,
    "HealthyDeadline": 0,
    "ProgressDeadline": 0,
    "AutoRevert": false,
    "AutoPromote": false,
    "Canary": 0
  },
  "Multiregion": null,
  "Periodic": null,
  "ParameterizedJob": null,
  "Dispatched": false,
  "DispatchIdempotencyToken": "",
  "Payload": null,
  "Meta": null,
  "ConsulToken": "",
  "ConsulNamespace": "",
  "VaultToken": "",
  "VaultNamespace": "",
  "NomadTokenID": "",
  "Status": "dead",
  "StatusDescription": "",
  "Stable": false,
  "Version": 2,
  "SubmitTime": 1647913889928897300,
  "CreateIndex": 2078647,
  "ModifyIndex": 2078678,
  "JobModifyIndex": 2078675
}

Nomad Server logs (if appropriate)

Mar 22 09:50:53 mgt04 nomad[1352590]: fatal error: unexpected signal during runtime execution
Mar 22 09:50:53 mgt04 nomad[1352590]: [signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7f83705f3248]
Mar 22 09:50:53 mgt04 nomad[1352590]: runtime stack:
Mar 22 09:50:53 mgt04 nomad[1352590]: runtime.throw({0x28b6d2a, 0x7f834c000dd0})
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/panic.go:1198 +0x71
Mar 22 09:50:53 mgt04 nomad[1352590]: runtime.sigpanic()
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/signal_unix.go:719 +0x396
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 193295 [syscall]:
Mar 22 09:50:53 mgt04 nomad[1352590]: runtime.cgocall(0x1f316d0, 0xc0004e99a8)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/cgocall.go:156 +0x5c fp=0xc0004e9980 sp=0xc0004e9948 pc=0x4053dc
Mar 22 09:50:53 mgt04 nomad[1352590]: os/user._Cfunc_mygetpwnam_r(0xc000aa8190, 0xc00083d740, 0x7f834c000dd0, 0x400, 0xc000e8e008)
Mar 22 09:50:53 mgt04 nomad[1352590]:         _cgo_gotypes.go:155 +0x4c fp=0xc0004e99a8 sp=0xc0004e9980 pc=0xa4f90c
Mar 22 09:50:53 mgt04 nomad[1352590]: os/user.lookupUser.func1.1({0xc000aa8190, 0xc0002911e0, 0x0}, 0xc000374800, 0xc0004e9a50, 0x40f227)
Mar 22 09:50:53 mgt04 nomad[1352590]:         os/user/cgo_lookup_unix.go:72 +0xbb fp=0xc0004e9a08 sp=0xc0004e99a8 pc=0xa5021b
Mar 22 09:50:53 mgt04 nomad[1352590]: os/user.lookupUser.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         os/user/cgo_lookup_unix.go:72 +0x31 fp=0xc0004e9a48 sp=0xc0004e9a08 pc=0xa50131
Mar 22 09:50:53 mgt04 nomad[1352590]: os/user.retryWithBuffer(0xc0002911e0, 0xc0004e9b38)
Mar 22 09:50:53 mgt04 nomad[1352590]:         os/user/cgo_lookup_unix.go:249 +0x39 fp=0xc0004e9a90 sp=0xc0004e9a48 pc=0xa51339
Mar 22 09:50:53 mgt04 nomad[1352590]: os/user.lookupUser({0x2820d34, 0x6})
Mar 22 09:50:53 mgt04 nomad[1352590]:         os/user/cgo_lookup_unix.go:63 +0x170 fp=0xc0004e9b88 sp=0xc0004e9a90 pc=0xa4ff10
Mar 22 09:50:53 mgt04 nomad[1352590]: os/user.Lookup({0x2820d34, 0x0})
Mar 22 09:50:53 mgt04 nomad[1352590]:         os/user/lookup.go:36 +0x91 fp=0xc0004e9bd0 sp=0xc0004e9b88 pc=0xa4f391
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/allocdir.dropDirPermissions({0xc000d71a40, 0x44}, 0x4e9d38)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocdir/fs_unix.go:43 +0x12f fp=0xc0004e9ca8 sp=0xc0004e9bd0 pc=0x106ae8f
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/allocdir.(*AllocDir).Build(0xc000a51f80)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocdir/alloc_dir.go:329 +0x12e fp=0xc0004e9d60 sp=0xc0004e9ca8 pc=0x10680ee
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/allocrunner.(*allocDirHook).Prerun(0xc000d6e840)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocrunner/allocdir_hook.go:28 +0x1c fp=0xc0004e9d78 sp=0xc0004e9d60 pc=0x16d6e1c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).prerun(0xc000702900)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner_hooks.go:196 +0x424 fp=0xc0004e9ed0 sp=0xc0004e9d78 pc=0x16d4084
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).Run(0xc000702900)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:321 +0x14a fp=0xc0004e9fc8 sp=0xc0004e9ed0 pc=0x16cb70a
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).addAlloc·dwrap·39()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:2491 +0x26 fp=0xc0004e9fe0 sp=0xc0004e9fc8 pc=0x178a8e6
Mar 22 09:50:53 mgt04 nomad[1352590]: runtime.goexit()
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/asm_amd64.s:1581 +0x1 fp=0xc0004e9fe8 sp=0xc0004e9fe0 pc=0x46b601
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*Client).addAlloc
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:2491 +0x778
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 1 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/command/agent.(*Command).handleSignals(0xc000304f00)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/command/agent/command.go:867 +0x1af
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc000304f00, {0xc0001100a0, 0x2, 0x2})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/command/agent/command.go:782 +0xdd5
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/mitchellh/cli.(*CLI).Run(0xc00053a640)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/mitchellh/cli@v1.1.0/cli.go:260 +0x5f8
Mar 22 09:50:53 mgt04 nomad[1352590]: main.RunCustom({0xc000110090, 0x22076c0, 0xc0000001a0})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/main.go:116 +0x33d
Mar 22 09:50:53 mgt04 nomad[1352590]: main.Run(...)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/main.go:86
Mar 22 09:50:53 mgt04 nomad[1352590]: main.main()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/main.go:82 +0x50
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 6 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: go.opencensus.io/stats/view.(*worker).start(0xc000368000)
Mar 22 09:50:53 mgt04 nomad[1352590]:         go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xb9
Mar 22 09:50:53 mgt04 nomad[1352590]: created by go.opencensus.io/stats/view.init.0
Mar 22 09:50:53 mgt04 nomad[1352590]:         go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x92
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 397 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/csimanager.(*csiManager).runLoop(0xc000b2e060)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/csimanager/manager.go:101 +0x170
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/csimanager.(*csiManager).Run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/csimanager/manager.go:93 +0x5b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 485 [runnable]:
Mar 22 09:50:53 mgt04 nomad[1352590]: strings.Trim({0xc000c460e0, 0x6a}, {0x292f7a8, 0x1})
Mar 22 09:50:53 mgt04 nomad[1352590]:         strings/strings.go:836 +0x7e
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/shirou/gopsutil/v3/internal/common.ReadLinesOffsetN({0xc0009574e8, 0x7f83885218b8}, 0x0, 0xffffffffffffffff)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/shirou/gopsutil/v3@v3.21.6-0.20210619153009-7ea8062810b6/internal/common/common.go:137 +0x30a
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/shirou/gopsutil/v3/internal/common.ReadLines(...)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/shirou/gopsutil/v3@v3.21.6-0.20210619153009-7ea8062810b6/internal/common/common.go:111
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/shirou/gopsutil/v3/disk.PartitionsWithContext({0x1011ecf, 0x2318f00}, 0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/shirou/gopsutil/v3@v3.21.6-0.20210619153009-7ea8062810b6/disk/disk_linux.go:225 +0xa5
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/shirou/gopsutil/v3/disk.Partitions(...)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/shirou/gopsutil/v3@v3.21.6-0.20210619153009-7ea8062810b6/disk/disk.go:77
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/stats.(*HostStatsCollector).collectDiskStats(0xc000b2e0c0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/stats/host.go:184 +0x46
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/stats.(*HostStatsCollector).collectLocked(0xc000b2e0c0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/stats/host.go:143 +0x3af
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/stats.(*HostStatsCollector).Collect(0xc000b2e0c0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/stats/host.go:108 +0x7c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*AllocGarbageCollector).keepUsageBelowThreshold(0xc000b2e120)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/gc.go:126 +0x88
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*AllocGarbageCollector).Run(0xc000b2e120)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/gc.go:100 +0xd7
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.NewClient
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:500 +0x1aef
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 497 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).allocSync(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:1970 +0x15b
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*group).Go.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3104 +0x5a
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*group).Go
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3102 +0x88
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 118 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/command/agent/consul.checkConsulTLSSkipVerify({0x2dadb58, 0xc000b76580}, {0x2e0eee0, 0xc000d6ec60}, {0x2dd9cf0, 0xc000de6bd0}, 0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/command/agent/consul/version_checker.go:42 +0x17c
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/command/agent/consul.(*ServiceClient).Run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/command/agent/consul/service_client.go:613 +0x1c8
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 137 [sleep]:
Mar 22 09:50:53 mgt04 nomad[1352590]: time.Sleep(0x3b9aca00)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/time.go:193 +0x12e
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/armon/go-metrics.(*Metrics).collectStats(0xc0007374a0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/armon/go-metrics@v0.3.10/metrics.go:230 +0x25
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/armon/go-metrics.New
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/armon/go-metrics@v0.3.10/start.go:84 +0x17b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 138 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/command/agent/consul.(*ServiceClient).Run(0xc000737680)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/command/agent/consul/service_client.go:619 +0x26d
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/command/agent.(*Agent).setupConsul
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/command/agent/agent.go:1211 +0x345
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 136 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/armon/go-metrics.(*InmemSignal).run(0xc000253a80)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/armon/go-metrics@v0.3.10/inmem_signal.go:64 +0x6c
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/armon/go-metrics.NewInmemSignal
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/armon/go-metrics@v0.3.10/inmem_signal.go:38 +0x174
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 135 [syscall, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: os/signal.signal_recv()
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/sigqueue.go:169 +0x98
Mar 22 09:50:53 mgt04 nomad[1352590]: os/signal.loop()
Mar 22 09:50:53 mgt04 nomad[1352590]:         os/signal/signal_unix.go:24 +0x19
Mar 22 09:50:53 mgt04 nomad[1352590]: created by os/signal.Notify.func1.1
Mar 22 09:50:53 mgt04 nomad[1352590]:         os/signal/signal.go:151 +0x2c
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 398 [semacquire, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.runtime_Semacquire(0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/sema.go:56 +0x25
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.(*WaitGroup).Wait(0xc0008ca480)
Mar 22 09:50:53 mgt04 nomad[1352590]:         sync/waitgroup.go:130 +0x71
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run(0xc0008ca480)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1b2
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x2af
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 165 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/helper/pool.(*ConnPool).reap(0xc0008c76d0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/helper/pool/pool.go:519 +0x9b
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/helper/pool.NewPool
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/helper/pool/pool.go:222 +0x17f
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 166 [select, 1 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/servers.(*Manager).Start(0xc0008f9c20)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/servers/manager.go:177 +0x7d
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.NewClient
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:382 +0xa8f
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 167 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/lib/cgutil.(*cpusetManager).reconcileLoop(0xc0008f9b60)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/lib/cgutil/cpuset_manager_linux.go:179 +0x10b
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/lib/cgutil.(*cpusetManager).Init
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/lib/cgutil/cpuset_manager_linux.go:167 +0x49d
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 168 [select, 1 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).rpcConnListener(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/rpc.go:288 +0xae
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*Client).setupClientRpc
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/rpc.go:266 +0x513
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 169 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*FingerprintManager).runFingerprint(0xc0008c7810, {0x2d7e918, 0xc00083d560}, 0x0, {0x28203da, 0x6})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/fingerprint_manager.go:167 +0x1fa
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupFingerprinters
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/fingerprint_manager.go:147 +0x349
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 172 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*FingerprintManager).runFingerprint(0xc0008c7810, {0x2d7e990, 0xc00083daa0}, 0x8000080008, {0x2820434, 0x6})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/fingerprint_manager.go:167 +0x1fa
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupFingerprinters
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/fingerprint_manager.go:147 +0x349
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 515 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).fingerprint(0xc0008ca600)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:301 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x13e
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 548 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).handleEvents(0xc0008ca780)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:437 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func2()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1a8
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 546 [select]:
Mar 22 09:50:53 mgt04 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc0005141e0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0xd5
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0x11b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 366 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc00040b0e0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0xd5
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0x11b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 480 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/exec.(*Driver).handleFingerprint(0xc0008cc5a0, {0x2dadb58, 0xc001064200}, 0xc000116900)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/exec/driver.go:317 +0x158
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/exec.(*Driver).Fingerprint
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/exec/driver.go:309 +0xbd
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 547 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).fingerprint(0xc0008ca780)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:301 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x13e
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 494 [select, 1111 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).consulDiscovery(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:2708 +0x6c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*group).Go.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3104 +0x5a
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*group).Go
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3102 +0x88
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 264 [select]:
Mar 22 09:50:53 mgt04 systemd[1]: nomad.service: Failed with result 'exit-code'.
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/docker.(*Driver).handleFingerprint(0xc000734000, {0x2dadb58, 0xc000d20000}, 0xc000d1e180)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/docker/fingerprint.go:68 +0x185
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/docker.(*Driver).Fingerprint
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/docker/fingerprint.go:21 +0xcf
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 479 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).handleEvents(0xc0008ca480)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:437 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func2()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1a8
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 253 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*FingerprintManager).runFingerprint(0xc0008c7810, {0x2d7eb20, 0xc000d6e180}, 0xc000b16fd0, {0x281eef8, 0x5})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/fingerprint_manager.go:167 +0x1fa
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupFingerprinters
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/fingerprint_manager.go:147 +0x349
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 192841 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/yamux.(*Session).send(0xc0006e1340)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/session.go:399 +0xc5
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/yamux.newSession
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/session.go:114 +0x4dd
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 477 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000113200)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0xd5
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0x11b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 367 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).fingerprint(0xc0008ca840)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:301 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x13e
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 399 [semacquire, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.runtime_Semacquire(0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/sema.go:56 +0x25
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.(*WaitGroup).Wait(0xc0008ca540)
Mar 22 09:50:53 mgt04 nomad[1352590]:         sync/waitgroup.go:130 +0x71
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run(0xc0008ca540)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1b2
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x2af
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 478 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).fingerprint(0xc0008ca480)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:301 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x13e
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 368 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).handleEvents(0xc0008ca840)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:437 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func2()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1a8
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 161 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/dynamicplugins.(*dynamicRegistry).PluginsUpdatedCh.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/dynamicplugins/registry.go:322 +0x85
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/dynamicplugins.(*dynamicRegistry).PluginsUpdatedCh
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/dynamicplugins/registry.go:321 +0xd1
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 160 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/dynamicplugins.(*pluginEventBroadcaster).run(0xc000b76140)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/dynamicplugins/registry.go:369 +0x87
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/dynamicplugins.newPluginEventBroadcaster
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/dynamicplugins/registry.go:363 +0x11b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 369 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/java.(*Driver).handleFingerprint(0xc0008b6000, {0x2dadb58, 0xc000282640}, 0xc000dff3c0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/java/driver.go:305 +0x115
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/java.(*Driver).Fingerprint
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/java/driver.go:298 +0xbd
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 159 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/dynamicplugins.(*dynamicRegistry).PluginsUpdatedCh.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/dynamicplugins/registry.go:322 +0x85
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/dynamicplugins.(*dynamicRegistry).PluginsUpdatedCh
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/dynamicplugins/registry.go:321 +0xd1
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 158 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/dynamicplugins.(*pluginEventBroadcaster).run(0xc000b76100)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/dynamicplugins/registry.go:369 +0x87
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/dynamicplugins.newPluginEventBroadcaster
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/dynamicplugins/registry.go:363 +0x11b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 263 [select, 4 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/docker.(*containerReconciler).removeDanglingContainersGoroutine(0xc000113260)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/docker/reconciler.go:74 +0xda
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/docker.(*containerReconciler).Start.func1
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/docker/reconciler.go:53 +0x5f
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 484 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*heartbeatStop).watch(0xc0000f4000)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/heartbeatstop.go:91 +0x188
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.NewClient
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:484 +0x18c5
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 400 [semacquire, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.runtime_Semacquire(0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/sema.go:56 +0x25
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.(*WaitGroup).Wait(0xc0008ca600)
Mar 22 09:50:53 mgt04 nomad[1352590]:         sync/waitgroup.go:130 +0x71
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run(0xc0008ca600)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1b2
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x2af
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 584 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).watchNodeEvents(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:1765 +0x145
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:1592 +0xc5
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 583 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).watchNodeUpdates(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:2245 +0xee
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:1589 +0x7d
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 496 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).periodicSnapshot(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:1696 +0x85
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*group).Go.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3104 +0x5a
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*group).Go
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3102 +0x88
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 495 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:1605 +0x1a5
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*group).Go.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3104 +0x5a
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*group).Go
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3102 +0x88
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 482 [semacquire, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.runtime_Semacquire(0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/sema.go:56 +0x25
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.(*WaitGroup).Wait(0xc0008ca840)
Mar 22 09:50:53 mgt04 nomad[1352590]:         sync/waitgroup.go:130 +0x71
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run(0xc0008ca840)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1b2
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x2af
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 578 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).run(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:1716 +0xf9
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.NewClient
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:562 +0x224f
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 401 [semacquire, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.runtime_Semacquire(0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/sema.go:56 +0x25
Mar 22 09:50:53 mgt04 nomad[1352590]: sync.(*WaitGroup).Wait(0xc0008ca780)
Mar 22 09:50:53 mgt04 nomad[1352590]:         sync/waitgroup.go:130 +0x71
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run(0xc0008ca780)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:174 +0x1b2
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.newInstanceManager
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:132 +0x2af
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 579 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).emitStats(0xc0002d6580)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:2840 +0x258
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*group).Go.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3104 +0x5a
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*group).Go
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:3102 +0x88
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 104 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).fingerprint(0xc0008ca540)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:301 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:162 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:161 +0x13e
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 103 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000636360)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0xd5
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0x11b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 514 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/shared/eventer.(*Eventer).eventLoop(0xc000e8a000)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:66 +0xd5
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/shared/eventer.NewEventer
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/shared/eventer/eventer.go:58 +0x11b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 105 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).handleEvents(0xc0008ca540)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:437 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func2()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1a8
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 530 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/rawexec.(*Driver).handleFingerprint(0xc000d72080, {0x2dadb58, 0xc000c22000}, 0xc000c26000)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/rawexec/driver.go:212 +0x15e
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/rawexec.(*Driver).Fingerprint
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/rawexec/driver.go:204 +0xbd
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 516 [select, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).handleEvents(0xc0008ca600)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:437 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run.func2()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:169 +0x26
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/pluginmanager/drivermanager.(*instanceManager).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/pluginmanager/drivermanager/instance.go:168 +0x1a8
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 295 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/drivers/qemu.(*Driver).handleFingerprint(0xc0002a6000, {0x2dadb58, 0xc000436000}, 0xc0005141e0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/qemu/driver.go:220 +0x115
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/drivers/qemu.(*Driver).Fingerprint
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/drivers/qemu/driver.go:213 +0xbd
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 581 [select, 1415 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/go-checkpoint.CheckInterval.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-checkpoint@v0.0.0-20171009173528-1545e56e46de/checkpoint.go:314 +0xa7
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/go-checkpoint.CheckInterval
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-checkpoint@v0.0.0-20171009173528-1545e56e46de/checkpoint.go:312 +0xd3
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 580 [IO wait, 5364 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: internal/poll.runtime_pollWait(0x7f8389638be0, 0x72)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/netpoll.go:234 +0x89
Mar 22 09:50:53 mgt04 nomad[1352590]: internal/poll.(*pollDesc).wait(0xc0000ced80, 0x2, 0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         internal/poll/fd_poll_runtime.go:84 +0x32
Mar 22 09:50:53 mgt04 nomad[1352590]: internal/poll.(*pollDesc).waitRead(...)
Mar 22 09:50:53 mgt04 nomad[1352590]:         internal/poll/fd_poll_runtime.go:89
Mar 22 09:50:53 mgt04 nomad[1352590]: internal/poll.(*FD).Accept(0xc0000ced80)
Mar 22 09:50:53 mgt04 nomad[1352590]:         internal/poll/fd_unix.go:402 +0x22c
Mar 22 09:50:53 mgt04 nomad[1352590]: net.(*netFD).accept(0xc0000ced80)
Mar 22 09:50:53 mgt04 nomad[1352590]:         net/fd_unix.go:173 +0x35
Mar 22 09:50:53 mgt04 nomad[1352590]: net.(*TCPListener).accept(0xc0008fd560)
Mar 22 09:50:53 mgt04 nomad[1352590]:         net/tcpsock_posix.go:140 +0x28
Mar 22 09:50:53 mgt04 nomad[1352590]: net.(*TCPListener).Accept(0xc0008fd560)
Mar 22 09:50:53 mgt04 nomad[1352590]:         net/tcpsock.go:262 +0x3d
Mar 22 09:50:53 mgt04 nomad[1352590]: net/http.(*Server).Serve(0xc000e1d0a0, {0x2d884e8, 0xc0008fd560})
Mar 22 09:50:53 mgt04 nomad[1352590]:         net/http/server.go:3002 +0x394
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/command/agent.NewHTTPServers.func1()
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/command/agent/http.go:171 +0x68
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/command/agent.NewHTTPServers
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/command/agent/http.go:169 +0xacf
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 271 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/yamux.(*Stream).Read(0xc000058300, {0xc000c9d000, 0x1000, 0xc000c99620})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/stream.go:133 +0x41a
Mar 22 09:50:53 mgt04 nomad[1352590]: bufio.(*Reader).fill(0xc000e8a5a0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         bufio/bufio.go:101 +0x103
Mar 22 09:50:53 mgt04 nomad[1352590]: bufio.(*Reader).ReadByte(0xc000e8a5a0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         bufio/bufio.go:253 +0x2c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/go-msgpack/codec.(*ioDecReader).readn1(0xc000e46000)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-msgpack@v1.1.5/codec/decode.go:447 +0x2e
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/go-msgpack/codec.(*decReaderSwitch).readn1IO(0xc000058380)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-msgpack@v1.1.5/codec/decode.go:2230 +0x38
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/go-msgpack/codec.(*decReaderSwitch).readn1(0xc00083d620)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-msgpack@v1.1.5/codec/decode.go:2224 +0x48
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).readNextBd(...)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-msgpack@v1.1.5/codec/msgpack.go:742
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).TryDecodeAsNil(0xc000744000)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-msgpack@v1.1.5/codec/msgpack.go:791 +0x2c
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/go-msgpack/codec.(*Decoder).mustDecode(0xc00053a3c0, {0x21bf060, 0xc00083d620})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-msgpack@v1.1.5/codec/decode.go:2536 +0x3a
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc00083d620, {0x21bf060, 0xc00083d620})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/go-msgpack@v1.1.5/codec/decode.go:2519 +0xa6
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/net-rpc-msgpackrpc.(*MsgpackCodec).read(0x30, {0x21bf060, 0xc00083d620})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/net-rpc-msgpackrpc@v0.0.0-20151116020338-a14192a58a69/codec.go:121 +0x35
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/net-rpc-msgpackrpc.(*MsgpackCodec).ReadResponseHeader(0x2d80bc8, 0xc00083c8d0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/net-rpc-msgpackrpc@v0.0.0-20151116020338-a14192a58a69/codec.go:74 +0x25
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/net-rpc-msgpackrpc.CallWithCodec({0x2dae568, 0xc001064000}, {0x284bba1, 0x14}, {0x26e2880, 0xc00089c460}, {0x2485700, 0xc000d20280})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/net-rpc-msgpackrpc@v0.0.0-20151116020338-a14192a58a69/client.go:29 +0x10d
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/helper/pool.(*ConnPool).RPC(0x2, {0x28208d8, 0x0}, {0x2d80bc8, 0xc00083c8d0}, 0x0, {0x284bba1, 0x14}, {0x26e2880, 0xc00089c460}, ...)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/helper/pool/pool.go:489 +0x158
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).RPC(0xc0002d6580, {0x284bba1, 0x14}, {0x26e2880, 0xc00089c460}, {0x2485700, 0xc000d20280})
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/rpc.go:77 +0x205
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).watchAllocations(0xc0002d6580, 0xc00034d0e0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:2071 +0x2a7
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*Client).run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/client.go:1713 +0x98
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 193314 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdates(0xc000702900)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:845 +0x8b
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).Run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:307 +0x105
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 192843 [select, 14 minutes]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/yamux.(*Session).AcceptStream(0xc0006e1340)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/session.go:212 +0x75
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/helper/pool.(*Conn).AcceptStream(0xc000e8a420)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/helper/pool/pool.go:139 +0x29
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client.(*Client).listenConn(0xc0002d6580, 0xc000e8a420)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/rpc.go:305 +0x31
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client.(*Client).rpcConnListener
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/rpc.go:296 +0x125
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 192842 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/yamux.(*Session).keepalive(0xc0006e1340)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/session.go:313 +0x86
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/yamux.newSession
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/session.go:116 +0x52f
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 192840 [IO wait]:
Mar 22 09:50:53 mgt04 nomad[1352590]: internal/poll.runtime_pollWait(0x7f8389638db0, 0x72)
Mar 22 09:50:53 mgt04 nomad[1352590]:         runtime/netpoll.go:234 +0x89
Mar 22 09:50:53 mgt04 nomad[1352590]: internal/poll.(*pollDesc).wait(0xc000e46580, 0xc00098b000, 0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         internal/poll/fd_poll_runtime.go:84 +0x32
Mar 22 09:50:53 mgt04 nomad[1352590]: internal/poll.(*pollDesc).waitRead(...)
Mar 22 09:50:53 mgt04 nomad[1352590]:         internal/poll/fd_poll_runtime.go:89
Mar 22 09:50:53 mgt04 nomad[1352590]: internal/poll.(*FD).Read(0xc000e46580, {0xc00098b000, 0x1000, 0x1000})
Mar 22 09:50:53 mgt04 nomad[1352590]:         internal/poll/fd_unix.go:167 +0x25a
Mar 22 09:50:53 mgt04 nomad[1352590]: net.(*netFD).Read(0xc000e46580, {0xc00098b000, 0x7f83b1c64a68, 0xc000bb0d06})
Mar 22 09:50:53 mgt04 nomad[1352590]:         net/fd_posix.go:56 +0x29
Mar 22 09:50:53 mgt04 nomad[1352590]: net.(*conn).Read(0xc000538000, {0xc00098b000, 0xc00016c780, 0xc0004a2780})
Mar 22 09:50:53 mgt04 nomad[1352590]:         net/net.go:183 +0x45
Mar 22 09:50:53 mgt04 nomad[1352590]: bufio.(*Reader).Read(0xc000e8a120, {0xc0004a2780, 0xc, 0x1b})
Mar 22 09:50:53 mgt04 nomad[1352590]:         bufio/bufio.go:227 +0x1b4
Mar 22 09:50:53 mgt04 nomad[1352590]: io.ReadAtLeast({0x2d65fa0, 0xc000e8a120}, {0xc0004a2780, 0xc, 0xc}, 0xc)
Mar 22 09:50:53 mgt04 nomad[1352590]:         io/io.go:328 +0x9a
Mar 22 09:50:53 mgt04 nomad[1352590]: io.ReadFull(...)
Mar 22 09:50:53 mgt04 nomad[1352590]:         io/io.go:347
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/yamux.(*Session).recvLoop(0xc0006e1340)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/session.go:458 +0xbe
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/yamux.(*Session).recv(0x0)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/session.go:437 +0x1e
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/yamux.newSession
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/yamux@v0.0.0-20181012175058-2f1d1f20f75d/session.go:113 +0x49b
Mar 22 09:50:53 mgt04 nomad[1352590]: goroutine 193313 [select]:
Mar 22 09:50:53 mgt04 nomad[1352590]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleTaskStateUpdates(0xc000702900)
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:492 +0x188
Mar 22 09:50:53 mgt04 nomad[1352590]: created by github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).Run
Mar 22 09:50:53 mgt04 nomad[1352590]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:304 +0xba
Mar 22 09:50:55 mgt04 systemd[1]: nomad.service: Service hold-off time over, scheduling restart.
Mar 22 09:50:55 mgt04 systemd[1]: nomad.service: Scheduled restart job, restart counter is at 583.
Mar 22 09:50:55 mgt04 systemd[1]: Stopped Nomad.
Mar 22 09:50:55 mgt04 systemd[1]: nomad.service: Found left-over process 1731885 (nomad) in control group while starting unit. Ignoring.
Mar 22 09:50:55 mgt04 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 22 09:50:55 mgt04 systemd[1]: nomad.service: Found left-over process 1732442 (nomad) in control group while starting unit. Ignoring.
Mar 22 09:50:55 mgt04 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 22 09:50:55 mgt04 systemd[1]: nomad.service: Found left-over process 1733152 (nomad) in control group while starting unit. Ignoring.
Mar 22 09:50:55 mgt04 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 22 09:50:55 mgt04 systemd[1]: nomad.service: Found left-over process 1733704 (nomad) in control group while starting unit. Ignoring.
Mar 22 09:50:55 mgt04 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 22 09:50:55 mgt04 systemd[1]: nomad.service: Found left-over process 1734246 (nomad) in control group while starting unit. Ignoring.
@jrasell
Copy link
Member

jrasell commented Mar 22, 2022

Hi @jwz-ecust and thanks for raising this issue. Could you provide any additional details regarding the exact steps you took (commands, APIs etc) to hit this panic? This will help us reproduce the error and fix it up quickly.

@jrasell jrasell added this to Needs Triage in Nomad - Community Issues Triage via automation Mar 22, 2022
@jwz-ecust
Copy link
Author

@jrasell v1/jobs and v1/jobs/parse. We request these APIs on the same node (mgt01) in a concurrent way. mgt01 is also the nomad server node. During requests, the nomad collapsed and would recover after.

@tgross
Copy link
Member

tgross commented Mar 22, 2022

Hi @jwz-ecust! @shoenig and I have looked at the stack trace, and it looks like the panic happened while the Nomad node was creating an allocation. The API calls you sent just happened to be sent when that happened and don't look to be related to the problem... just bad luck.

The panic happened when the go runtime was making a Linux syscall to lookup the nobody user (ref cgo_lookup_unix.go#L68-L72). This usually is just a lookup in /etc/passwd but could also be in LDAP or NIS depending on your setup. (See man 3 getpwnam_r).

This might be a bug in go, or it might be an unexpected issue in your environment. A few questions:

  • Did you get your Nomad binary from an official HashiCorp package? For example, the apt package or from downloading via releases.hashicorp.com?
  • Are you running on a x86_64 machine and not a x86?
  • Does the nobody user exist on this host? Do you use /etc/passwd for users and not LDAP/NIS?

@tgross tgross moved this from Needs Triage to In Progress in Nomad - Community Issues Triage Mar 22, 2022
@tgross tgross self-assigned this Mar 22, 2022
@tgross tgross changed the title Nomad Collapse panic in os/user.Lookup CGO call Mar 23, 2022
@tgross
Copy link
Member

tgross commented May 2, 2022

I think we've done what we can with this issue without more information. I'm going to close this out but please feel free to re-open if you have answers to the questions I've asked above. Thanks!

@tgross tgross closed this as completed May 2, 2022
Nomad - Community Issues Triage automation moved this from In Progress to Done May 2, 2022
@tgross tgross mentioned this issue Aug 23, 2022
Closed
@shoenig shoenig mentioned this issue Aug 23, 2022
Closed
@tgross tgross mentioned this issue Aug 24, 2022
Merged
@github-actions
Copy link

github-actions bot commented Oct 8, 2022

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 8, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tgross tgross

Labels
stage/waiting-reply theme/client theme/crash type/bug
Projects
Nomad - Community Issues Triage
Archived in project
Nomad - Community Issues Triage
Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tgross @jrasell @jwz-ecust