Random hang of test suite #209

Raynos · 2017-09-19T20:22:43Z

Sometimes the ./test/ package itself will hang in a test case

Here is a dump of sigabort to see why the tests / goroutines are hanging








{"level":"info","ts":1505849366.9648693,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4765,"zone":"unknown","realHTTPAddr":"10.32.106.72:34521","realTChannelAddr":"10.32.106.72:40161","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:38385","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:40114","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"error","ts":1505849367.029912,"msg":"Failed to Bootstrap in TestStartGateway()","error":"error listening on port: listen tcp 10.32.106.72:34521: bind: address already in use"}
{"level":"error","ts":1505849367.0298743,"msg":"Error listening on port","hostname":"raynos-ThinkPad-T440p","pid":4776,"zone":"unknown","error":"listen tcp 10.32.106.72:34521: bind: address already in use"}
{"level":"info","ts":1505849367.0834262,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4784,"zone":"unknown","realHTTPAddr":"10.32.106.72:36721","realTChannelAddr":"10.32.106.72:37823","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:33303","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:49837","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"info","ts":1505849367.0845563,"msg":"Finished an incoming server HTTP request","hostname":"raynos-ThinkPad-T440p","pid":4784,"zone":"unknown","endpointID":"health","handlerID":"health","method":"GET","remoteAddr":"10.32.106.72:33878","pathname":"/health","host":"10.32.106.72:36721","timestamp-started":1505849367.0845108,"timestamp-finished":1505849367.0845234,"statusCode":200,"Request Body":"","Response Body":"{\"ok\":true,\"message\":\"Healthy, from example-gateway\"}\n","Request-Header-User-Agent":"Go-http-client/1.1","Request-Header-Accept-Encoding":"gzip","Response-Header-Content-Type":"application/json"}
{"level":"info","ts":1505849367.126711,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4793,"zone":"unknown","realHTTPAddr":"10.32.106.72:35879","realTChannelAddr":"10.32.106.72:35419","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:40493","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:51699","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"info","ts":1505849367.1276844,"msg":"Finished an incoming server HTTP request","hostname":"raynos-ThinkPad-T440p","pid":4793,"zone":"unknown","endpointID":"health","handlerID":"health","method":"GET","remoteAddr":"10.32.106.72:37640","pathname":"/health","host":"10.32.106.72:35879","timestamp-started":1505849367.127651,"timestamp-finished":1505849367.1276588,"statusCode":200,"Request Body":"","Response Body":"{\"ok\":true,\"message\":\"Healthy, from example-gateway\"}\n","Request-Header-User-Agent":"Go-http-client/1.1","Request-Header-Accept-Encoding":"gzip","Response-Header-Content-Type":"application/json"}
{"level":"info","ts":1505849367.1945999,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4802,"zone":"unknown","realHTTPAddr":"10.32.106.72:39537","realTChannelAddr":"10.32.106.72:44285","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:57802","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:49706","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":true,"metrics.runtime.enableGCMetrics":true,"metrics.runtime.enableMemMetrics":true,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
fatal error: concurrent map iteration and map write

goroutine 50 [running]:
runtime.throw(0xba8457, 0x26)
	/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc42004d988 sp=0xc42004d968
runtime.mapiternext(0xc420443f80)
	/usr/local/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc42004da38 sp=0xc42004d988
reflect.mapiternext(0xc420443f80)
	/usr/local/go/src/runtime/hashmap.go:1156 +0x2b fp=0xc42004da50 sp=0xc42004da38
reflect.Value.MapKeys(0xac5220, 0xc420332b10, 0x15, 0xc42004dbe9, 0x98, 0xc42004dc38)
	/usr/local/go/src/reflect/value.go:1112 +0x1bc fp=0xc42004db00 sp=0xc42004da50
github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert.includeElement(0xac5220, 0xc420332b10, 0xa8aac0, 0xc420390b10, 0xc420040000)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert/assertions.go:617 +0x2db fp=0xc42004dbc8 sp=0xc42004db00
github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert.Contains(0xf3cdc0, 0xc420338340, 0xac5220, 0xc420332b10, 0xa8aac0, 0xc420390b10, 0xc42004dd98, 0x2, 0x2, 0x1)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert/assertions.go:645 +0x59 fp=0xc42004dc48 sp=0xc42004dbc8
github.com/uber/zanzibar/test_test.TestRuntimeMetrics(0xc420338340)
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/health_test.go:223 +0x731 fp=0xc42004dfa8 sp=0xc42004dc48
testing.tRunner(0xc420338340, 0xbb66a8)
	/usr/local/go/src/testing/testing.go:657 +0x96 fp=0xc42004dfd0 sp=0xc42004dfa8
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42004dfd8 sp=0xc42004dfd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:697 +0x2ca

goroutine 1 [chan receive]:
testing.(*T).Run(0xc420071520, 0xb98dc4, 0x12, 0xbb66a8, 0xc4202d5d01)
	/usr/local/go/src/testing/testing.go:698 +0x2f4
testing.runTests.func1(0xc420071520)
	/usr/local/go/src/testing/testing.go:882 +0x67
testing.tRunner(0xc420071520, 0xc4202d5de0)
	/usr/local/go/src/testing/testing.go:657 +0x96
testing.runTests(0xc420299d40, 0xf723e0, 0x4, 0x4, 0xaf4c20)
	/usr/local/go/src/testing/testing.go:888 +0x2c1
testing.(*M).Run(0xc420049f20, 0xc4202d5f20)
	/usr/local/go/src/testing/testing.go:822 +0xfc
main.main()
	github.com/uber/zanzibar/test/_test/_testmain.go:50 +0xf7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 37 [IO wait]:
net.runtime_pollWait(0x7f7d951b1948, 0x72, 0xd)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420446298, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420446298, 0xc42048c028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc420446230, 0xc42048c028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc4203ca0a8, 0xc42048c028, 0xfde8, 0xfde8, 0xc42048bfc8, 0x411598, 0xc42048bfc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc420394480, 0xc42048c028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc420388300, 0xc4204332d0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 52 [IO wait]:
net.runtime_pollWait(0x7f7d951b1708, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42042c458, 0x72, 0xf386e0, 0xc420421800)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42042c458, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42042c3f0, 0x0, 0xf3ca80, 0xc420421800)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc4204c8088, 0xc420373a40, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4204c8088, 0xc420373a10, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4204e00b0, 0xf43180, 0xc4204c8088, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4204cca20)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 8 [IO wait]:
net.runtime_pollWait(0x7f7d951b1e88, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202daa78, 0x72, 0xf386e0, 0xc42033c000)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202daa78, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202daa10, 0x0, 0xf3ca80, 0xc42033c000)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e148, 0xc4203320c0, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e148, 0xc420332090, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001ec60, 0xf43180, 0xc42000e148, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc42006fce0)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 9 [IO wait]:
net.runtime_pollWait(0x7f7d951b1f48, 0x72, 0x5)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202daa08, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202daa08, 0xc420312028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202da9a0, 0xc420312028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e138, 0xc420312028, 0xfde8, 0xfde8, 0xc420311fc8, 0x411598, 0xc420311fc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2480, 0xc420312028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc420019d80, 0xc4202aed80)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 36 [IO wait]:
net.runtime_pollWait(0x7f7d951b1888, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420446308, 0x72, 0xf386e0, 0xc42033c0c0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420446308, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4204462a0, 0x0, 0xf3ca80, 0xc42033c0c0)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc4203ca0b8, 0xc420332330, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4203ca0b8, 0xc420332300, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4204480b0, 0xf43180, 0xc4203ca0b8, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc420442180)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 53 [IO wait]:
net.runtime_pollWait(0x7f7d951b17c8, 0x72, 0xf)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42042c3e8, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42042c3e8, 0xc42051e028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc42042c380, 0xc42051e028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc4204c8078, 0xc42051e028, 0xfde8, 0xfde8, 0x411f6d, 0xc4204d0500, 0xc42051dfc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc42033a630, 0xc42051e028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fe380, 0xc420337a40)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 12 [IO wait]:
net.runtime_pollWait(0x7f7d951b1c48, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db1e8, 0x72, 0xf386e0, 0xc420396000)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db1e8, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202db180, 0x0, 0xf3ca80, 0xc420396000)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e270, 0xc42038c0c0, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e270, 0xc42038c090, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001efd0, 0xf43180, 0xc42000e270, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4203441e0)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 13 [IO wait]:
net.runtime_pollWait(0x7f7d951b1d08, 0x72, 0x9)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db178, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db178, 0xc4203b8028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202db110, 0xc4203b8028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e260, 0xc4203b8028, 0xfde8, 0xfde8, 0xc4203b7fc8, 0x411598, 0xc4203b7fc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2870, 0xc4203b8028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fc180, 0xc420360110)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 51 [runnable]:
bytes.(*Buffer).WriteByte(0xc420110f50, 0xc42037592b, 0x1c, 0x1c)
	/usr/local/go/src/bytes/buffer.go:236 +0x82
github.com/uber/zanzibar/vendor/github.com/uber-go/tally.KeyForPrefixedStringMap(0xc420375900, 0x1c, 0xc420038ca0, 0xc420038dd8, 0xc420038ca0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/key_gen.go:68 +0x267
github.com/uber/zanzibar/test/lib/test_m3_server.(*FakeM3Service).EmitMetricBatch(0xc420300550, 0xc4203755e0, 0x0, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_m3_server/test_m3_server.go:159 +0x30b
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift.(*m3ProcessorEmitMetricBatch).Process(0xc420337960, 0xc400000006, 0xf4bd60, 0xc42044a1e0, 0xf4bd60, 0xc42044a1e0, 0xfa00, 0x525, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift/m3.go:163 +0xeb
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift.(*M3Processor).Process(0xc42033d7c0, 0xf4bd60, 0xc42044a1e0, 0xf4bd60, 0xc42044a1e0, 0x0, 0x0, 0xc4203755c0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift/m3.go:137 +0x306
github.com/uber/zanzibar/test/lib/test_m3_server.(*FakeM3Server).Serve(0xc4203005a0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_m3_server/test_m3_server.go:95 +0x269
created by github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).setupMetrics
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway.go:127 +0xc5

goroutine 16 [IO wait]:
net.runtime_pollWait(0x7f7d951b1ac8, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db728, 0x72, 0xf386e0, 0xc42033c020)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db728, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202db6c0, 0x0, 0xf3ca80, 0xc42033c020)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e390, 0xc4203321b0, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e390, 0xc420332180, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001f340, 0xf43180, 0xc42000e390, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4203444e0)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 18 [IO wait]:
net.runtime_pollWait(0x7f7d951b1b88, 0x72, 0x8)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db6b8, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db6b8, 0xc4203fe028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202db650, 0xc4203fe028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e380, 0xc4203fe028, 0xfde8, 0xfde8, 0xc4203fdfc8, 0x411598, 0xc4203fdfc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2d80, 0xc4203fe028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fc680, 0xc4203619c0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 23 [select]:
net/http.(*persistConn).readLoop(0xc42045aea0)
	/usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 21 [select]:
net/http.(*persistConn).readLoop(0xc420424000)
	/usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 22 [select]:
net/http.(*persistConn).writeLoop(0xc420424000)
	/usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 24 [select]:
net/http.(*persistConn).writeLoop(0xc42045aea0)
	/usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 43 [syscall]:
syscall.Syscall(0x0, 0x11, 0xc4204e6000, 0x1000, 0x20, 0x20, 0xaf6700)
	/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x11, 0xc4204e6000, 0x1000, 0x1000, 0x7f7d951f5960, 0x0, 0xc420027cf0)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55
syscall.Read(0x11, 0xc4204e6000, 0x1000, 0x1000, 0xaf6700, 0x736901, 0xc42000c560)
	/usr/local/go/src/syscall/syscall_unix.go:162 +0x49
os.(*File).read(0xc42000e5d0, 0xc4204e6000, 0x1000, 0x1000, 0xc, 0xc, 0x0)
	/usr/local/go/src/os/file_unix.go:165 +0x4f
os.(*File).Read(0xc42000e5d0, 0xc4204e6000, 0x1000, 0x1000, 0x0, 0x0, 0x8)
	/usr/local/go/src/os/file.go:101 +0x76
bufio.(*Reader).fill(0xc420442060)
	/usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).ReadSlice(0xc420442060, 0x73c70a, 0xc4203ca008, 0xf4bd60, 0xc420094000, 0x0, 0xb331c0)
	/usr/local/go/src/bufio/bufio.go:338 +0xbb
bufio.(*Reader).ReadBytes(0xc420442060, 0xf4bd0a, 0xc420094000, 0xf4bd60, 0xc420094000, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:416 +0x66
bufio.(*Reader).ReadString(0xc420442060, 0xc42001010a, 0xc4200140e0, 0xc420027fc0, 0xc420094000, 0xc42052e000)
	/usr/local/go/src/bufio/bufio.go:456 +0x38
github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).copyToStdout(0xc4204ec000, 0xc420442060)
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway_process.go:221 +0x34
created by github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).createAndSpawnChild
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway_process.go:125 +0x61f
SIGABRT: abort
PC=0x45c2d3 m=2 sigcode=0

goroutine 0 [idle]:
runtime.futex(0xf23f58, 0x0, 0x7f111938adc8, 0x0, 0x7f1100000000, 0x45bf66, 0x3c, 0x0, 0x7f111938ae10, 0x40f720, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:426 +0x23
runtime.futexsleep(0xf23f58, 0x0, 0xdf8475800)
	/usr/local/go/src/runtime/os_linux.go:62 +0xd7
runtime.notetsleep_internal(0xf23f58, 0xdf8475800, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:174 +0xd0
runtime.notetsleep(0xf23f58, 0xdf8475800, 0x104363f71d3d01)
	/usr/local/go/src/runtime/lock_futex.go:194 +0x56
runtime.sysmon()
	/usr/local/go/src/runtime/proc.go:3805 +0x135
runtime.mstart1()
	/usr/local/go/src/runtime/proc.go:1179 +0x11e
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1149 +0x64

goroutine 1 [chan receive, 49 minutes]:
testing.(*T).Run(0xc420063380, 0xb5b1dc, 0x10, 0xb78860, 0xc4202bdcf0)
	/usr/local/go/src/testing/testing.go:698 +0x2f4
testing.runTests.func1(0xc420063380)
	/usr/local/go/src/testing/testing.go:882 +0x67
testing.tRunner(0xc420063380, 0xc4202bddb0)
	/usr/local/go/src/testing/testing.go:657 +0x96
testing.runTests(0xc420287b20, 0xf15000, 0x1, 0x1, 0xf24600)
	/usr/local/go/src/testing/testing.go:888 +0x2c1
testing.(*M).Run(0xc42004ff20, 0x1e)
	/usr/local/go/src/testing/testing.go:822 +0xfc
command-line-arguments.TestMain(0xc4202bdf20)
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:44 +0x58
main.main()
	command-line-arguments/_test/_testmain.go:40 +0xf7

goroutine 17 [syscall, 49 minutes, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 6 [syscall, 49 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 7 [select, 49 minutes, locked to thread]:
runtime.gopark(0xb7a250, 0x0, 0xb532ec, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc42002cf50, 0x0, 0x18)
	/usr/local/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc42002cf50)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 8 [chan receive, 49 minutes]:
command-line-arguments.listenOnSignals.func1(0xc420061740)
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:57 +0x53
created by command-line-arguments.listenOnSignals
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:62 +0xbc

goroutine 9 [semacquire, 49 minutes]:
sync.runtime_Semacquire(0xc4202c2a6c)
	/usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc4202c2a60)
	/usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/uber/zanzibar/runtime.(*Gateway).Wait(0xc420089400)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:263 +0x2f
command-line-arguments.logAndWait(0xc420089400)
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main.go:102 +0x2f0
command-line-arguments.TestStartGateway(0xc420063450)
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:92 +0x461
testing.tRunner(0xc420063450, 0xb78860)
	/usr/local/go/src/testing/testing.go:657 +0x96
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:697 +0x2ca

goroutine 10 [chan receive]:
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3.(*reporter).process(0xc4202cac80)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/reporter.go:483 +0xe3
created by github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3.NewReporter
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/reporter.go:241 +0x811

goroutine 11 [select]:
github.com/uber/zanzibar/vendor/github.com/uber-go/tally.(*scope).reportLoop(0xc420089540, 0x989680)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/scope.go:241 +0x158
created by github.com/uber/zanzibar/vendor/github.com/uber-go/tally.newRootScope
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/scope.go:180 +0x6b2

goroutine 12 [runnable]:
github.com/uber/zanzibar/runtime.(*runtimeCollector).Start.func1(0xc42001d680)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/runtime_metrics.go:184 +0x13f
created by github.com/uber/zanzibar/runtime.(*runtimeCollector).Start
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/runtime_metrics.go:192 +0x84

goroutine 13 [select]:
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc4203cb440)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/reporter.go:231 +0x324
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go.NewRemoteReporter
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/reporter.go:201 +0x288

goroutine 20 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4e00, 0x72, 0xee8660)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498a78, 0x72, 0xee2660, 0xc420716d60)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498a78, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420498a10, 0x0, 0xee6720, 0xc420716d60)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e078, 0xc42070d1a0, 0xc42003ad90, 0x5dbb5d)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e078, 0x6f1463, 0xc42003ada8, 0xc42003ada0)
	/usr/local/go/src/net/tcpsock.go:215 +0x49
github.com/uber/zanzibar/runtime.tcpKeepAliveListener.Accept(0xc42000e078, 0xc42070d170, 0xa86f20, 0xf13fa0, 0xaeb820)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:107 +0x2f
net/http.(*Server).Serve(0xc4204b0000, 0xeece20, 0xc42000e078, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
github.com/uber/zanzibar/runtime.(*HTTPServer).JustServe(0xc420474500, 0xc4202c2a60)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:74 +0x78
created by github.com/uber/zanzibar/runtime.(*Gateway).Bootstrap
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:164 +0x5cc

goroutine 21 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4ec0, 0x72, 0xee8660)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498a08, 0x72, 0xee2660, 0xc420716d80)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498a08, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4204989a0, 0x0, 0xee6720, 0xc420716d80)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e070, 0xc42070d290, 0xc42003ed90, 0x5dbb5d)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e070, 0x6f1463, 0xc42003eda8, 0xc42003eda0)
	/usr/local/go/src/net/tcpsock.go:215 +0x49
github.com/uber/zanzibar/runtime.tcpKeepAliveListener.Accept(0xc42000e070, 0xc42070d260, 0xa86f20, 0xf13fa0, 0xaeb820)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:107 +0x2f
net/http.(*Server).Serve(0xc4204b00b0, 0xeece20, 0xc42000e070, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
github.com/uber/zanzibar/runtime.(*HTTPServer).JustServe(0xc420474550, 0xc4202c2a60)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:74 +0x78
created by github.com/uber/zanzibar/runtime.(*Gateway).Bootstrap
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:168 +0xce9

goroutine 22 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4d40, 0x72, 0xee8660)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498ae8, 0x72, 0xee2660, 0xc420716760)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498ae8, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420498a80, 0x0, 0xee6720, 0xc420716760)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e080, 0xc400000008, 0xc420480000, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e080, 0xb796e8, 0xc420716560, 0x1, 0x0)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/tnet.(*listener).Accept(0xc420716560, 0x0, 0x0, 0x0, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/tnet/listener.go:80 +0x8f
github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go.(*Channel).serve(0xc4204b6000)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/channel.go:390 +0x56
created by github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go.(*Channel).Serve
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/channel.go:269 +0x4b3

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x45c2d3
rdx    0x0
rdi    0xf23f58
rsi    0x0
rbp    0x7f111938add8
rsp    0x7f111938ad90
r8     0x0
r9     0x0
r10    0x7f111938adc8
r11    0x246
r12    0x0
r13    0x7ffee299717f
r14    0x7f111938b9c0
r15    0x7ffee2997200
rip    0x45c2d3
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
*** Test killed: ran too long (10m0s).
FAIL	github.com/uber/zanzibar/test	2973.475s
ok  	github.com/uber/zanzibar/test/clients/bar	0.945s
ok  	github.com/uber/zanzibar/test/clients/baz	0.839s
ok  	github.com/uber/zanzibar/test/config	0.005s
ok  	github.com/uber/zanzibar/test/endpoints/bar	1.215s
ok  	github.com/uber/zanzibar/test/endpoints/baz	1.354s
ok  	github.com/uber/zanzibar/test/endpoints/contacts	0.054s
ok  	github.com/uber/zanzibar/test/endpoints/googlenow	0.390s
ok  	github.com/uber/zanzibar/test/endpoints/tchannel/baz	0.044s
?   	github.com/uber/zanzibar/test/lib	[no test files]
?   	github.com/uber/zanzibar/test/lib/bench_gateway	[no test files]
?   	github.com/uber/zanzibar/test/lib/test_backend	[no test files]
?   	github.com/uber/zanzibar/test/lib/test_gateway	[no test files]
?   	github.com/uber/zanzibar/test/lib/test_m3_server	[no test files]
?   	github.com/uber/zanzibar/test/lib/util	[no test files]

The text was updated successfully, but these errors were encountered:

Raynos · 2017-09-19T20:25:19Z

either /health_test.go:223 or the test block before that hangs. so TestRuntimeMetrics or TestHealthMetrics

Raynos · 2017-09-19T20:27:10Z

raynos at raynos-ThinkPad-T440p  
~/gocode/src/github.com/uber/zanzibar on url-params-server-side
$ while ZANZIBAR_CACHE=1 go test ./test/ -run TestRuntimeMetrics ; do sleep 1 ; done
ok  	github.com/uber/zanzibar/test	0.035s
ok  	github.com/uber/zanzibar/test	0.047s
ok  	github.com/uber/zanzibar/test	0.057s
ok  	github.com/uber/zanzibar/test	0.036s

Confirmed it is TestRuntimeMetrics that hangs.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Random hang of test suite #209

Random hang of test suite #209

Raynos commented Sep 19, 2017

Raynos commented Sep 19, 2017

Raynos commented Sep 19, 2017

Random hang of test suite #209

Random hang of test suite #209

Comments

Raynos commented Sep 19, 2017

Raynos commented Sep 19, 2017

Raynos commented Sep 19, 2017