storage: TestGossipSystemConfigOnLeaseChange failed under stress #22407

cockroach-teamcity · 2018-02-06T10:55:14Z

SHA: https://github.com/cockroachdb/cockroach/commits/4b2c64a7107e392229939b24b5bda303c1950a1f

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=508297&tab=buildLog


I180206 10:55:11.423252 1179407 gossip/gossip.go:332  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:32835" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W180206 10:55:11.434778 1179407 gossip/gossip.go:1292  [n2] no incoming or outgoing connections
I180206 10:55:11.435175 1180756 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:32835
I180206 10:55:11.435281 1179407 storage/store.go:1307  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I180206 10:55:11.435417 1179407 gossip/gossip.go:332  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:45485" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W180206 10:55:11.445019 1179407 gossip/gossip.go:1292  [n3] no incoming or outgoing connections
I180206 10:55:11.445410 1179407 storage/store.go:1307  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I180206 10:55:11.445543 1179407 gossip/gossip.go:332  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:35211" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I180206 10:55:11.446566 1181586 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:32835
POTENTIAL DEADLOCK: Duplicate locking, saw callers this locks in one goroutine:
current goroutine 1179407 lock &{{{0 0} 0 0 1 0}} 
all callers to this lock in the goroutine
client_test.go:1066 storage_test.(*multiTestContext).replicateRangeNonFatal { m.mu.RLock() } <<<<<
client_test.go:1059 storage_test.(*multiTestContext).replicateRange { if err := m.replicateRangeNonFatal(rangeID, dests...); err != nil { }
client_lease_test.go:249 storage_test.TestGossipSystemConfigOnLeaseChange { mtc.replicateRange(rangeID, 1, 2) }

client_test.go:472 storage_test.(*multiTestContextKVTransport).SendNext { t.mtc.mu.RLock() } <<<<<
../kv/dist_sender.go:1278 kv.(*DistSender).sendToReplicas { transport.SendNext(ctx, done) }
../kv/dist_sender.go:382 kv.(*DistSender).sendRPC { return ds.sendToReplicas(ctx, SendOptions{metrics: &ds.metrics}, rangeID, replicas, ba, ds.rpcContext) }
../kv/dist_sender.go:446 kv.(*DistSender).sendSingleRange { br, err := ds.sendRPC(ctx, desc.RangeID, replicas, ba) }
../kv/dist_sender.go:1056 kv.(*DistSender).sendPartialBatch { reply, pErr = ds.sendSingleRange(ctx, ba, desc) }
../kv/dist_sender.go:724 kv.(*DistSender).divideAndSendBatchToRanges { resp := ds.sendPartialBatch(ctx, ba, rs, ri.Desc(), ri.Token(), batchIdx, false /* needsTruncate */) }
../kv/dist_sender.go:640 kv.(*DistSender).Send { rpl, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, 0 /* batchIdx */) }
../kv/txn_coord_sender.go:485 kv.(*TxnCoordSender).Send { if br, pErr = tc.wrapped.Send(ctx, ba); pErr != nil { }
../internal/client/db.go:555 client.(*DB).sendUsingSender { br, pErr := sender.Send(ctx, ba) }
../internal/client/db.go:529 client.(*DB).send { return db.sendUsingSender(ctx, ba, db.GetSender()) }
../internal/client/db.go:482 client.send)-fm { return sendAndFill(ctx, db.send, b) }
../internal/client/db.go:459 client.sendAndFill { b.response, b.pErr = send(ctx, ba) }
../internal/client/db.go:482 client.(*DB).Run { return sendAndFill(ctx, db.send, b) }
../internal/client/db.go:221 client.(*DB).Get { return getOneRow(db.Run(ctx, b), b) }
../internal/client/db.go:229 client.(*DB).GetProto { r, err := db.Get(ctx, key) }
client_test.go:1013 storage_test.(*multiTestContext).changeReplicasLocked { if err := m.dbs[0].GetProto(ctx, keys.RangeDescriptorKey(startKey), &desc); err != nil { }
client_test.go:1074 storage_test.(*multiTestContext).replicateRangeNonFatal { expectedReplicaIDs[i], err = m.changeReplicasLocked(rangeID, dest, roachpb.ADD_REPLICA) }
client_test.go:1059 storage_test.(*multiTestContext).replicateRange { if err := m.replicateRangeNonFatal(rangeID, dests...); err != nil { }
client_lease_test.go:249 storage_test.TestGossipSystemConfigOnLeaseChange { mtc.replicateRange(rangeID, 1, 2) }


Other goroutines holding locks:
goroutine 1138042 lock 0xc434de4388
store.go:4072 storage.(*Store).tryGetOrCreateReplica { repl.raftMu.Lock() } <<<<<
store.go:3980 storage.(*Store).getOrCreateReplica { r, created, err := s.tryGetOrCreateReplica( }
store_test.go:2775 storage.TestRemovedReplicaTombstone.func1 { _, created, err := s.getOrCreateReplica(ctx, rangeID, c.createReplicaID, &creatingReplica) }

goroutine 1142018 lock 0xc434de4988
store.go:4072 storage.(*Store).tryGetOrCreateReplica { repl.raftMu.Lock() } <<<<<
store.go:3980 storage.(*Store).getOrCreateReplica { r, created, err := s.tryGetOrCreateReplica( }
store_test.go:2775 storage.TestRemovedReplicaTombstone.func1 { _, created, err := s.getOrCreateReplica(ctx, rangeID, c.createReplicaID, &creatingReplica) }

goroutine 1143292 lock 0xc429768c88
store.go:4072 storage.(*Store).tryGetOrCreateReplica { repl.raftMu.Lock() } <<<<<
store.go:3980 storage.(*Store).getOrCreateReplica { r, created, err := s.tryGetOrCreateReplica( }
store_test.go:2775 storage.TestRemovedReplicaTombstone.func1 { _, created, err := s.getOrCreateReplica(ctx, rangeID, c.createReplicaID, &creatingReplica) }





ERROR: exit status 2

context canceled
make: *** [stress] Error 1

The text was updated successfully, but these errors were encountered:

a-robinson · 2018-02-06T20:46:43Z

This was broken by the update to the deadlock detector in #21477.

a-robinson · 2018-02-06T20:53:14Z

I'm following up on sasha-s/go-deadlock#7 before modifying this test infrastructure that has been in place for years.

cockroach-teamcity · 2018-02-07T10:51:49Z

SHA: https://github.com/cockroachdb/cockroach/commits/27384c5c82a6f20679aa7f833875eda6f068e4de

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=510122&tab=buildLog

cockroach-teamcity · 2018-02-08T10:43:13Z

SHA: https://github.com/cockroachdb/cockroach/commits/f45bcea79f3d26f9484075ee174e270e9ccdc714

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=512001&tab=buildLog

cockroach-teamcity · 2018-02-09T11:03:49Z

SHA: https://github.com/cockroachdb/cockroach/commits/c93d304f8b31b5b0678a46d36a7b751cf50fedb1

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=513875&tab=buildLog

cockroach-teamcity · 2018-02-10T10:59:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/fb37bf0c50592df50959fdb25903616a682e3c97

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=515071&tab=buildLog

cockroach-teamcity · 2018-02-11T11:15:01Z

SHA: https://github.com/cockroachdb/cockroach/commits/2ced98d8cd4d13abae6331e73effc5b13f369c9a

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=515999&tab=buildLog

cockroach-teamcity · 2018-02-12T11:12:47Z

SHA: https://github.com/cockroachdb/cockroach/commits/135c4999731e5e67f95fc8557f95cee2a0aeadc0

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=516895&tab=buildLog

cockroach-teamcity · 2018-02-13T10:50:58Z

SHA: https://github.com/cockroachdb/cockroach/commits/b089c2110c41f7d610ea87fc0432bf1634ce9638

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=519129&tab=buildLog

cockroach-teamcity · 2018-02-14T11:01:42Z

SHA: https://github.com/cockroachdb/cockroach/commits/0c433fe884a153edafbb2fa97672327b1d10b762

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=520567&tab=buildLog

cockroach-teamcity · 2018-02-15T10:53:34Z

SHA: https://github.com/cockroachdb/cockroach/commits/b36ad3075ce26fb3c436f6b67db3796aa6273157

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=522369&tab=buildLog

cockroach-teamcity · 2018-02-16T10:38:43Z

SHA: https://github.com/cockroachdb/cockroach/commits/dcee806fbe40a2b95ece93bcfb1b2cc4431fff7d

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=524639&tab=buildLog

cockroach-teamcity · 2018-02-17T10:43:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/bb33afcaed789bdf52d313722b010b41889d8b0e

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=525544&tab=buildLog

cockroach-teamcity · 2018-02-18T10:40:46Z

SHA: https://github.com/cockroachdb/cockroach/commits/f9f3d43ca646b6b8a84c6d09b091936ac30bc1ae

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=526378&tab=buildLog

cockroach-teamcity · 2018-02-19T10:40:05Z

SHA: https://github.com/cockroachdb/cockroach/commits/f9f3d43ca646b6b8a84c6d09b091936ac30bc1ae

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=527029&tab=buildLog

cockroach-teamcity · 2018-02-20T10:54:32Z

SHA: https://github.com/cockroachdb/cockroach/commits/cecbeb79c96f564a1cd20d4928eafc5cfec88e3c

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=528023&tab=buildLog

cockroach-teamcity · 2018-02-21T10:51:00Z

SHA: https://github.com/cockroachdb/cockroach/commits/71720f67e5a9bcd768bb3f681fc5ed95fa205a14

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=529590&tab=buildLog

cockroach-teamcity · 2018-02-22T10:50:20Z

SHA: https://github.com/cockroachdb/cockroach/commits/5af643083fa542b2166a35380fb6dfd19c070f02

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=531413&tab=buildLog

cockroach-teamcity · 2018-02-23T10:43:24Z

SHA: https://github.com/cockroachdb/cockroach/commits/f69c298417360e9179c55970476aadf4c69397d6

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=532961&tab=buildLog

cockroach-teamcity · 2018-02-24T10:56:04Z

SHA: https://github.com/cockroachdb/cockroach/commits/1bdafbe4c9d649407c3a417236f55a9970f95787

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=534402&tab=buildLog

cockroach-teamcity · 2018-02-25T11:11:34Z

SHA: https://github.com/cockroachdb/cockroach/commits/ffdf42487ebecc65d47008fbdd8d5d89810e8e13

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=535398&tab=buildLog

cockroach-teamcity · 2018-02-26T11:03:19Z

SHA: https://github.com/cockroachdb/cockroach/commits/e88d11d979dd32237197b0f86b456476b4d4f12c

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=536121&tab=buildLog

Fixes cockroachdb#22407 Release note: None

cockroach-teamcity assigned a-robinson Feb 6, 2018

cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Feb 6, 2018

petermattis added this to the 2.0 milestone Feb 21, 2018

a-robinson mentioned this issue Feb 26, 2018

storage: avoid duplicate locking in multiTestContext replication code #23112

Merged

a-robinson added a commit to a-robinson/cockroach that referenced this issue Feb 26, 2018

storage: avoid duplicate locking in multiTestContext replication code

a5a3888

Fixes cockroachdb#22407 Release note: None

a-robinson closed this as completed in #23112 Feb 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

storage: TestGossipSystemConfigOnLeaseChange failed under stress #22407

storage: TestGossipSystemConfigOnLeaseChange failed under stress #22407

cockroach-teamcity commented Feb 6, 2018

a-robinson commented Feb 6, 2018

a-robinson commented Feb 6, 2018

cockroach-teamcity commented Feb 7, 2018

cockroach-teamcity commented Feb 8, 2018

cockroach-teamcity commented Feb 9, 2018

cockroach-teamcity commented Feb 10, 2018

cockroach-teamcity commented Feb 11, 2018

cockroach-teamcity commented Feb 12, 2018

cockroach-teamcity commented Feb 13, 2018

cockroach-teamcity commented Feb 14, 2018

cockroach-teamcity commented Feb 15, 2018

cockroach-teamcity commented Feb 16, 2018

cockroach-teamcity commented Feb 17, 2018

cockroach-teamcity commented Feb 18, 2018

cockroach-teamcity commented Feb 19, 2018

cockroach-teamcity commented Feb 20, 2018

cockroach-teamcity commented Feb 21, 2018

cockroach-teamcity commented Feb 22, 2018

cockroach-teamcity commented Feb 23, 2018

cockroach-teamcity commented Feb 24, 2018

cockroach-teamcity commented Feb 25, 2018

cockroach-teamcity commented Feb 26, 2018

storage: TestGossipSystemConfigOnLeaseChange failed under stress #22407

storage: TestGossipSystemConfigOnLeaseChange failed under stress #22407

Comments

cockroach-teamcity commented Feb 6, 2018

a-robinson commented Feb 6, 2018

a-robinson commented Feb 6, 2018

cockroach-teamcity commented Feb 7, 2018

cockroach-teamcity commented Feb 8, 2018

cockroach-teamcity commented Feb 9, 2018

cockroach-teamcity commented Feb 10, 2018

cockroach-teamcity commented Feb 11, 2018

cockroach-teamcity commented Feb 12, 2018

cockroach-teamcity commented Feb 13, 2018

cockroach-teamcity commented Feb 14, 2018

cockroach-teamcity commented Feb 15, 2018

cockroach-teamcity commented Feb 16, 2018

cockroach-teamcity commented Feb 17, 2018

cockroach-teamcity commented Feb 18, 2018

cockroach-teamcity commented Feb 19, 2018

cockroach-teamcity commented Feb 20, 2018

cockroach-teamcity commented Feb 21, 2018

cockroach-teamcity commented Feb 22, 2018

cockroach-teamcity commented Feb 23, 2018

cockroach-teamcity commented Feb 24, 2018

cockroach-teamcity commented Feb 25, 2018

cockroach-teamcity commented Feb 26, 2018