Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nomad Client: fatal error: concurrent map read and map write #4154

Closed
ivanprostran opened this issue Apr 15, 2018 · 5 comments
Closed

Nomad Client: fatal error: concurrent map read and map write #4154

ivanprostran opened this issue Apr 15, 2018 · 5 comments

Comments

@ivanprostran
Copy link

ivanprostran commented Apr 15, 2018

Nomad version

Output from nomad version:

Nomad v0.8.0 (751b42a)

Client: true
         Log Level: INFO
            Region: ipdr (DC: dc1)
            Server: false
           Version: 0.8.0

client.hcl

data_dir = "/opt/nomad/data/client/"
client {
  enabled       = true
  cpu_total_compute = 20000
  reserved {
                cpu = 5000
                memory = 28500
                disk = 1
        }
  network_speed = 10000
  servers = ["127.0.0.1:4647"]
  options {
    "driver.raw_exec.enable" = "1"
  }
}
ports {
    http = 5656
}

Output from 'consul version':

Consul v1.0.6
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when > speaking to compatible agents)

Operating system and Environment details

Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux node03 3.10.0-327.10.1.el7.x86_64 #1 SMP Sat Jan 23 04:54:55 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Issue

The Nomad Client exits after some time with the following error:
fatal error: concurrent map read and map write

Nomad cluster setup is as follows:

  • 10 nodes (Every node has: nomad client, nomad server, consul)
  • Client driver being used is "raw_exec " ( i.e. no isolation, no docker, no cgroup)

Reproduction steps

Everything works smoothly and, unfortunately, after some time server stops working.

Notice: This was not the case with the previous version:

Nomad v0.7.1 (0b295d3)

Nomad Server logs

Nothing in server log.

Nomad Client logs

fatal error: concurrent map read and map write

goroutine 99 [running]:
runtime.throw(0x16f1387, 0x21)
	/usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc4204c7b70 sp=0xc4204c7b50 pc=0x42dca5
runtime.mapaccess2_faststr(0x14a27a0, 0xc4204ed3b0, 0x16e3937, 0x18, 0x4131b8, 0x10)
	/usr/local/go/src/runtime/hashmap_fast.go:324 +0x47a fp=0xc4204c7bc8 sp=0xc4204c7b70 pc=0x40e43a
github.com/hashicorp/nomad/client/driver.cgroupsMounted(...)
	/opt/gopath/src/github.com/hashicorp/nomad/client/driver/utils.go:29
github.com/hashicorp/nomad/client/driver.(*ExecDriver).Fingerprint(0xc4201d4960, 0xc4201d8aa0, 0xc420d27220, 0x4, 0xc4204628b0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/driver/exec_linux.go:20 +0x61 fp=0xc4204c7c20 sp=0xc4204c7bc8 pc=0xe0f3d1
github.com/hashicorp/nomad/client.(*FingerprintManager).fingerprintDriver(0xc4201d2940, 0x16c5a59, 0x4, 0x7f6af40c21a8, 0xc4201d4960, 0x0, 0xc4200c2101, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:341 +0x17d fp=0xc4204c7d38 sp=0xc4204c7c20 pc=0xfac8bd
github.com/hashicorp/nomad/client.(*FingerprintManager).watchDriver(0xc4201d2940, 0x219a6c0, 0xc4201d4960, 0x16c5a59, 0x4)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:295 +0x3d1 fp=0xc4204c7fb8 sp=0xc4204c7d38 pc=0xfac081
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc4204c7fc0 sp=0xc4204c7fb8 pc=0x45f071
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupDrivers
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:192 +0x251

goroutine 1 [select, 240 minutes]:
github.com/hashicorp/nomad/command/agent.(*Command).handleSignals(0xc4201f6770, 0x1738010)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:562 +0x22d
github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc4201f6770, 0xc420098020, 0x5, 0x5, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:551 +0xd45
github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc420461680, 0xc420461680, 0xc420447c00, 0x37)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:255 +0x1eb
main.RunCustom(0xc420098010, 0x6, 0x6, 0xc4204d9f60)
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:127 +0x4c9
main.Run(0xc420098010, 0x6, 0x6, 0x12f3d54)
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:72 +0x3f
main.main()
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:68 +0x63

goroutine 19 [syscall, 240 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 72 [select, 240 minutes]:
github.com/hashicorp/nomad/command/agent/consul.(*checkWatcher).Run(0xc4201d2240, 0x2191400, 0xc4204fc240)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/check_watcher.go:186 +0x297
created by github.com/hashicorp/nomad/command/agent/consul.(*ServiceClient).Run
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/client.go:300 +0x33a

goroutine 114 [semacquire]:
sync.runtime_SemacquireMutex(0xc4201d2954, 0x0)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc4201d2950)
	/usr/local/go/src/sync/mutex.go:134 +0xee
github.com/hashicorp/nomad/client.(*FingerprintManager).fingerprintDriver(0xc4201d2940, 0x16c4eef, 0x3, 0x7f6af40c2268, 0xc4201d4b40, 0x0, 0xc42004a000, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:327 +0x6e
github.com/hashicorp/nomad/client.(*FingerprintManager).watchDriver(0xc4201d2940, 0x219a8c0, 0xc4201d4b40, 0x16c4eef, 0x3)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:295 +0x3d1
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupDrivers
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:192 +0x251

goroutine 133 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc4200ae9a0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:392 +0x2ca
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x406

goroutine 252 [select, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc4202e4540, 0xc420392000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:133 +0x342
io/ioutil.devNull.ReadFrom(0x0, 0x21833c0, 0xc4202e4540, 0x14db760, 0x1, 0x7f6af07bd0d0)
	/usr/local/go/src/io/ioutil/ioutil.go:144 +0x92
io/ioutil.(*devNull).ReadFrom(0x22544e0, 0x21833c0, 0xc4202e4540, 0x7f6af07bd0d0, 0x22544e0, 0x1)
	<autogenerated>:1 +0x50
io.copyBuffer(0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4540, 0x0, 0x0, 0x0, 0xc4202c77b0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:386 +0x2bb
io.Copy(0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4540, 0x0, 0x0, 0xc4202c7790)
	/usr/local/go/src/io/io.go:362 +0x68
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x16c8c34, 0x6, 0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4540)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x79
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCClient).SyncStreams
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:109 +0x119

....

goroutine 52 [select, 240 minutes]:
github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*InmemSignal).run(0xc4201d2180)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:64 +0xf8
created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.NewInmemSignal
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:38 +0x162

goroutine 53 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/local/go/src/runtime/time.go:65 +0x130
github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*Metrics).collectStats(0xc420208210)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/metrics.go:168 +0x2f
created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.New
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/start.go:75 +0x12b

goroutine 54 [select, 237 minutes]:
github.com/hashicorp/nomad/command/agent/consul.(*ServiceClient).Run(0xc420208420)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/client.go:310 +0x4fe
created by github.com/hashicorp/nomad/command/agent.(*Agent).setupConsul
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:950 +0x145

goroutine 55 [select]:
github.com/hashicorp/nomad/helper/pool.(*ConnPool).reap(0xc420356f00)
	/opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:439 +0x37b
created by github.com/hashicorp/nomad/helper/pool.NewPool
	/opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:170 +0x179

goroutine 56 [select]:
github.com/hashicorp/nomad/client.(*Client).rpcConnListener(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:241 +0x127
created by github.com/hashicorp/nomad/client.(*Client).setupClientRpc
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:221 +0x134

goroutine 74 [select, 4 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc420500540, 0xc4208d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:133 +0x342
bufio.(*Reader).fill(0xc42059fbc0)
	/usr/local/go/src/bufio/bufio.go:97 +0x11a
bufio.(*Reader).ReadByte(0xc42059fbc0, 0xc42001d900, 0xc42001d970, 0xc42001d900)
	/usr/local/go/src/bufio/bufio.go:239 +0x39
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*ioDecReader).readn1(0xc420ca74a0, 0xc42055b610)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x7a
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).initReadNext(0xc420afa520)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x45
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc42059fc20, 0x13e3ce0, 0xc420ca7650)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x49
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc42059fc20, 0x13e3ce0, 0xc420ca7650, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7b
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.(*MsgpackCodec).read(0xc420d95300, 0x13e3ce0, 0xc420ca7650, 0x1, 0xc420ca7650)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/codec.go:121 +0x57
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.(*MsgpackCodec).ReadResponseHeader(0xc420d95300, 0xc420ca7650, 0x15c8940, 0xc4201e4080)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/codec.go:74 +0x41
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.CallWithCodec(0x2191c80, 0xc420d95300, 0x16dc6d1, 0x14, 0x15c8940, 0xc4201e4080, 0x150a6a0, 0xc42007e570, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/client.go:29 +0x10d
github.com/hashicorp/nomad/helper/pool.(*ConnPool).RPC(0xc420356f00, 0x7ffd3f881ef5, 0x4, 0x218a240, 0xc420747080, 0x1, 0x16dc6d1, 0x14, 0x15c8940, 0xc4201e4080, ...)
	/opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:422 +0x1a8
github.com/hashicorp/nomad/client.(*Client).RPC(0xc42045c000, 0x16dc6d1, 0x14, 0x15c8940, 0xc4201e4080, 0x150a6a0, 0xc42007e570, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:68 +0x136
github.com/hashicorp/nomad/client.(*Client).watchAllocations(0xc42045c000, 0xc4200a8360)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1619 +0x27c
created by github.com/hashicorp/nomad/client.(*Client).run
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1280 +0x7c

goroutine 57 [select]:
github.com/hashicorp/nomad/client.(*AllocGarbageCollector).Run(0xc4201d4640)
	/opt/gopath/src/github.com/hashicorp/nomad/client/gc.go:91 +0x150
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:255 +0x7cc

goroutine 60 [select]:
github.com/hashicorp/nomad/client.(*FingerprintManager).runFingerprint(0xc4201d2940, 0x2188740, 0xc4204ed560, 0x37e11d600, 0x16c810c, 0x6)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:212 +0x243
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupFingerprinters
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:158 +0x216

goroutine 26408 [select, 15 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).AcceptStream(0xc42094c420, 0xd2eab1, 0x1, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:207 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).Accept(0xc42094c420, 0xc4202c37c0, 0xd3317d, 0xc420758b00, 0xc4203178d0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:197 +0x2b
github.com/hashicorp/nomad/client.(*Client).listenConn(0xc42045c000, 0xc42094c420)
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:258 +0x2f
created by github.com/hashicorp/nomad/client.(*Client).rpcConnListener
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:249 +0x173

goroutine 64 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af40b7eb0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42041cb98, 0x72, 0xffffffffffffff00, 0x2187440, 0x217f978)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42041cb98, 0xc4203bf000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42041cb80, 0xc4203bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc42041cb80, 0xc4203bf000, 0x1000, 0x1000, 0x42f99b, 0xc4203cca40, 0x45b0e0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc420092770, 0xc4203bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
net/http.(*persistConn).Read(0xc42035d560, 0xc4203bf000, 0x1000, 0x1000, 0xc4203282b0, 0xc4202bf738, 0x459140)
	/usr/local/go/src/net/http/transport.go:1391 +0x140
bufio.(*Reader).fill(0xc420357d40)
	/usr/local/go/src/bufio/bufio.go:97 +0x11a
bufio.(*Reader).Peek(0xc420357d40, 0x1, 0x0, 0x0, 0x0, 0xc420723b60, 0x0)
	/usr/local/go/src/bufio/bufio.go:129 +0x3a
net/http.(*persistConn).readLoop(0xc42035d560)
	/usr/local/go/src/net/http/transport.go:1539 +0x185
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1186 +0xa2e

goroutine 65 [select]:
net/http.(*persistConn).writeLoop(0xc42035d560)
	/usr/local/go/src/net/http/transport.go:1759 +0x165
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1187 +0xa53

goroutine 251 [select, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc4202e4240, 0xc42038c000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:133 +0x342
io/ioutil.devNull.ReadFrom(0x0, 0x21833c0, 0xc4202e4240, 0x14db760, 0x1, 0x7f6af07bd0d0)
	/usr/local/go/src/io/ioutil/ioutil.go:144 +0x92
io/ioutil.(*devNull).ReadFrom(0x22544e0, 0x21833c0, 0xc4202e4240, 0x7f6af07bd0d0, 0x22544e0, 0x1)
	<autogenerated>:1 +0x50
io.copyBuffer(0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4240, 0x0, 0x0, 0x0, 0xc4204af7b0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:386 +0x2bb
io.Copy(0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4240, 0x0, 0x0, 0xc4204af790)
	/usr/local/go/src/io/io.go:362 +0x68
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x16c8c3a, 0x6, 0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4240)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x79
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCClient).SyncStreams
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:108 +0x9d

goroutine 250 [select, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc42056c180, 0xc42037e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:133 +0x342
bufio.(*Reader).Read(0xc420510660, 0xc4204ee6f0, 0x1, 0x9, 0xc42057bca8, 0x42cc38, 0x3)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2181600, 0xc420510660, 0xc4204ee6f0, 0x1, 0x9, 0x1, 0xc42001d900, 0x1737740, 0xc42005ad80)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2181600, 0xc420510660, 0xc4204ee6f0, 0x1, 0x9, 0xc42057bd10, 0x40629b, 0x3)
	/usr/local/go/src/io/io.go:327 +0x58
encoding/gob.decodeUintReader(0x2181600, 0xc420510660, 0xc4204ee6f0, 0x9, 0x9, 0xc42057bd88, 0x42c579, 0xc400000008, 0xc420096100)
	/usr/local/go/src/encoding/gob/decode.go:120 +0x63
encoding/gob.(*Decoder).recvMessage(0xc4201e4e00, 0xc42057bda0)
	/usr/local/go/src/encoding/gob/decoder.go:80 +0x57
encoding/gob.(*Decoder).decodeTypeSequence(0xc4201e4e00, 0x173c100, 0xc4201e4e00)
	/usr/local/go/src/encoding/gob/decoder.go:144 +0x14c
encoding/gob.(*Decoder).DecodeValue(0xc4201e4e00, 0x13e3ce0, 0xc4204147b0, 0x16, 0x0, 0x0)
	/usr/local/go/src/encoding/gob/decoder.go:212 +0xe6
encoding/gob.(*Decoder).Decode(0xc4201e4e00, 0x13e3ce0, 0xc4204147b0, 0x0, 0x0)
	/usr/local/go/src/encoding/gob/decoder.go:189 +0x16d
net/rpc.(*gobClientCodec).ReadResponseHeader(0xc4204edb00, 0xc4204147b0, 0xc4204ee6fc, 0x0)
	/usr/local/go/src/net/rpc/client.go:223 +0x45
net/rpc.(*Client).input(0xc420510720)
	/usr/local/go/src/net/rpc/client.go:109 +0xb5
created by net/rpc.NewClientWithCodec
	/usr/local/go/src/net/rpc/client.go:201 +0xc5

goroutine 87 [select]:
github.com/hashicorp/nomad/client.(*FingerprintManager).runFingerprint(0xc4201d2940, 0x2188a00, 0xc420361be0, 0x37e11d600, 0x16c756f, 0x5)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:212 +0x243
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupFingerprinters
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:158 +0x216

goroutine 249 [select, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).AcceptStream(0xc4200ae9a0, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:207 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*MuxBroker).Run(0xc420588340)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/mux_broker.go:139 +0x33
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.NewRPCClient
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:88 +0x22f

goroutine 98 [semacquire]:
sync.runtime_SemacquireMutex(0xc4201d2954, 0x1416900)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc4201d2950)
	/usr/local/go/src/sync/mutex.go:134 +0xee
github.com/hashicorp/nomad/client.(*FingerprintManager).setNode(0xc4201d2940, 0xc42035cb40)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:54 +0x3a
github.com/hashicorp/nomad/client.(*FingerprintManager).fingerprintDriver(0xc4201d2940, 0x16c822c, 0x6, 0x7f6af40c1f98, 0xc4204f6780, 0x1, 0xc4202c8f00, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:380 +0x382
github.com/hashicorp/nomad/client.(*FingerprintManager).watchDriver(0xc4201d2940, 0x219a640, 0xc4204f6780, 0x16c822c, 0x6)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:295 +0x3d1
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupDrivers
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:192 +0x251

goroutine 248 [select, 238 minutes]:
github.com/hashicorp/nomad/client.(*AllocRunner).Run(0xc42051d980)
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:938 +0xcdc
created by github.com/hashicorp/nomad/client.(*Client).addAlloc
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1912 +0x3e7

goroutine 121 [select]:
github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1175 +0x1d1
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:318 +0xed3

goroutine 122 [select, 1 minutes]:
github.com/hashicorp/nomad/client.(*Client).periodicSnapshot(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1263 +0x103
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:321 +0xef8

goroutine 123 [select]:
github.com/hashicorp/nomad/client.(*Client).allocSync(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1531 +0x209
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:324 +0xf1d

goroutine 124 [select, 4 minutes]:
github.com/hashicorp/nomad/client.(*Client).run(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1283 +0x153
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:327 +0xf42

goroutine 125 [select]:
github.com/hashicorp/nomad/client.(*Client).emitStats(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:2183 +0x238
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:330 +0xf67

goroutine 126 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af40b7c70, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420205098, 0x72, 0xc4204c6c00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420205098, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420205080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
net.(*netFD).accept(0xc420205080, 0xc4204c6e40, 0x4020c7, 0xc4202e2300)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4201da280, 0x715e58, 0x45b7b0, 0xc4204c6e88)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4201da280, 0x173b8f0, 0xc4202e2280, 0x21914c0, 0xc42007f770)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
net/http.(*Server).Serve(0xc420514000, 0x218f800, 0xc4201da280, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2695 +0x1b2
net/http.Serve(0x218f800, 0xc4201da280, 0x2186940, 0xc42007f500, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2323 +0x73
github.com/hashicorp/nomad/command/agent.NewHTTPServer.func1(0xc4201d2d40, 0x218f800, 0xc4201da280, 0xc4202eda80, 0xc4204f8f90)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/http.go:111 +0x82
created by github.com/hashicorp/nomad/command/agent.NewHTTPServer
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/http.go:109 +0x40f

goroutine 25645 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af40b78b0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420c38218, 0x72, 0xffffffffffffff00, 0x2187440, 0x217f978)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420c38218, 0xc420636000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420c38200, 0xc420636000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc420c38200, 0xc420636000, 0x1000, 0x1000, 0x18, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42051aac8, 0xc420636000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc4204113e0, 0xc4203b74c0, 0xc, 0xc, 0x40f2c0, 0xc420a5c180, 0xc420c992f8)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2181600, 0xc4204113e0, 0xc4203b74c0, 0xc, 0xc, 0xc, 0x2e0000000c, 0xc42050dce0, 0xc420759384)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2181600, 0xc4204113e0, 0xc4203b74c0, 0xc, 0xc, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc420759340, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:451 +0xfc
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc420759340)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:430 +0x2b
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x3e4

goroutine 75 [select, 239 minutes]:
github.com/hashicorp/nomad/client.(*Client).watchNodeUpdates(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1774 +0x195
created by github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1159 +0x68

goroutine 76 [select, 240 minutes]:
github.com/hashicorp/nomad/client.(*Client).watchNodeEvents(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1328 +0x1f4
created by github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1162 +0x8d

goroutine 26407 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).keepalive(0xc42094c420)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:308 +0x102
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:111 +0x455

goroutine 27585 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af3ff01e8, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4209dad18, 0x72, 0xffffffffffffff00, 0x2187440, 0x217f978)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4209dad18, 0xc4209a3000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4209dad00, 0xc4209a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc4209dad00, 0xc4209a3000, 0x1000, 0x1000, 0xc, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc4205ca988, 0xc4209a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc42081dda0, 0xc420a4f2a0, 0xc, 0xc, 0x436f7b, 0xc420780e98, 0xd2fe74)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2181600, 0xc42081dda0, 0xc420a4f2a0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2181600, 0xc42081dda0, 0xc420a4f2a0, 0xc, 0xc, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc420c89550, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:451 +0xfc
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc420c89550)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:430 +0x2b
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x3e4

goroutine 27386 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc420ad2000)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:392 +0x2ca
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x406

goroutine 27650 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc420c89550)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:392 +0x2ca
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x406

goroutine 132 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af40b7430, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4205b0618, 0x72, 0xffffffffffffff00, 0x2187440, 0x217f978)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4205b0618, 0xc4205f0000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4205b0600, 0xc4205f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc4205b0600, 0xc4205f0000, 0x1000, 0x1000, 0xe1, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc4201da070, 0xc4205f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc42059e4e0, 0xc4204ee6a0, 0xc, 0xc, 0xc, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2181600, 0xc42059e4e0, 0xc4204ee6a0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2181600, 0xc42059e4e0, 0xc4204ee6a0, 0xc, 0xc, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc4200ae9a0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:451 +0xfc
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc4200ae9a0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:430 +0x2b
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x3e4

goroutine 245 [select, 238 minutes]:
github.com/hashicorp/nomad/client.(*AllocRunner).Run(0xc42051d380)
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:938 +0xcdc
created by github.com/hashicorp/nomad/client.(*Client).addAlloc
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1912 +0x3e7

goroutine 131 [chan receive, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func5.1(0xc4200c2660)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:570 +0x6d
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func5
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:569 +0x3f

goroutine 229 [select, 238 minutes]:
github.com/hashicorp/nomad/client.(*AllocRunner).dirtySyncState(0xc42051d380)
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:642 +0xfe
created by github.com/hashicorp/nomad/client.(*AllocRunner).Run
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:834 +0x288

goroutine 231 [select, 238 minutes]:
github.com/hashicorp/nomad/client.(*TaskRunner).run(0xc420186000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:1107 +0x2cc
github.com/hashicorp/nomad/client.(*TaskRunner).Run(0xc420186000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:624 +0x6fc
created by github.com/hashicorp/nomad/client.(*AllocRunner).Run
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:927 +0xaed

Job file

N/A

@ivanprostran
Copy link
Author

I would appreciate any help regarding this issue.
Thank you!

@chelseakomlo
Copy link
Contributor

Thanks for reporting this- we will be working on a fix today.

@dadgar
Copy link
Contributor

dadgar commented Apr 16, 2018

@ivanprostran Can you share the full stack dump and a bit of the client logs before the panic?

@ivanprostran
Copy link
Author

Hi, sorry for the late response.
File attached (just search for the following string "fatal error: concurrent map read and map write")
Thanks !!
Best regards,
Ivan Prostran

nomad-client.log

@github-actions
Copy link

github-actions bot commented Dec 1, 2022

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 1, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants