Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed test: TestParallel invalid retryable err #40328

Closed
cockroach-teamcity opened this issue Aug 29, 2019 · 3 comments · Fixed by #40356
Closed

teamcity: failed test: TestParallel invalid retryable err #40328

cockroach-teamcity opened this issue Aug 29, 2019 · 3 comments · Fixed by #40356
Assignees
@nvanbenschoten
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed on master (testrace): TestParallel, TestParallel/create_stats

You may want to check for open issues.

#1459621:

TestParallel/create_stats
...torage/stores.go:181 +0x1e8
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x5cad860, 0xc00631abd0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x254
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0014e65a0, 0x5cad860, 0xc00631abd0, 0x4e21843, 0x10, 0xc008d21438, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xf9
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000530500, 0x5cad860, 0xc00631abd0, 0xc003582f00, 0xc00631abd0, 0x66d28c, 0xc00004e000)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x2b0
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000530500, 0x5cad860, 0xc00631aba0, 0xc003582f00, 0xc000530500, 0xc0002acda0, 0x5c4b100)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0xb0
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x5cad860, 0xc00631aba0, 0x4de4e80, 0xc003582f00, 0x4de4e80, 0xc003582f00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9981 +0xa2
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x5cad860, 0xc00631aba0, 0x4de4e80, 0xc003582f00, 0xc006e7ce60, 0xc006e7ce80, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0xbc9
github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x5cad860, 0xc00631aba0, 0x4de4e80, 0xc003582f00, 0xc006e7ce60, 0xc006e7ce80, 0xc006e7ce80, 0x4247fe0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:221 +0x10f
github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x5cad860, 0xc00631aba0, 0x4de4e80, 0xc003582f00, 0xc006e7ce60, 0xc006e7ce80, 0x86f1da0, 0xc0003157c0, 0xc0070759c8, 0x670a58)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:251 +0xea
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x4db1720, 0xc000530500, 0x5cad860, 0xc00631aba0, 0xc005d97aa0, 0xc00037bd40, 0x5cad860, 0xc00631aba0, 0xc003652600, 0x200)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9983 +0x1d9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc001172000, 0x5d0a800, 0xc0034bcd80, 0xc00174a300, 0xc0018ec2d0, 0x844b7c0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x99c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001172000, 0x5d0a800, 0xc0034bcd80, 0xc00174a300, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0x1352
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00169d390, 0xc001172000, 0x5d0a800, 0xc0034bcd80, 0xc00174a300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0xad
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xb9


****************************************************************************

This node experienced a fatal error (printed above), and as a result the
process is terminating.

Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
problem in CockroachDB. With your help, the support team at Cockroach Labs
will try to determine the root cause, recommend next steps, and we can
improve CockroachDB based on your report.

Please submit a crash report by following the instructions here:

    https://github.com/cockroachdb/cockroach/issues/new/choose

If you would rather not post publicly, please contact us directly at:

    support@cockroachlabs.com

The Cockroach Labs team appreciates your feedback.



TestParallel
--- FAIL: testrace/TestParallel (0.000s)
Test ended in panic.

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 29, 2019
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Aug 29, 2019
@tbg
Copy link
Member

tbg commented Aug 29, 2019

cc @nvanbenschoten

F190829 05:35:59.694130 34381 roachpb/data.go:1265  [n2,split,s2,r60/3:/System/tsd/cr.node.sql.txn.…,txn=44f2c9de] invalid retryable err (*roachpb.MixedSuccessError): the batch experienced mixed success and failure: TransactionAbortedError(ABORT_REASON_ABORT_SPAN)
goroutine 34381 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x90b9b01, 0xc0003d4600, 0x0, 0x6c4e82)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1016 +0xbf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x86f0d00, 0xc000000004, 0x81132f8, 0xf, 0x4f1, 0xc006349700, 0xd6)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:874 +0xb0d
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x5cad860, 0xc00693ac90, 0x4, 0x2, 0x4e446b7, 0x1e, 0xc00252d998, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x291
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x5cad860, 0xc00693ac90, 0x1, 0xc000000004, 0x4e446b7, 0x1e, 0xc00252d998, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:180
github.com/cockroachdb/cockroach/pkg/roachpb.PrepareTransactionForRetry(0x5cad860, 0xc00693ac90, 0xc006542ff0, 0x0, 0xc0018b2050, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/data.go:1265 +0x2ae
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).handleRetryableErrLocked(0xc001798480, 0x5cad860, 0xc00693ac90, 0xc006542ff0, 0xc000f95380)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:1008 +0x1f9
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).updateStateLocked(0xc001798480, 0x5cad860, 0xc00693ac90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:1095 +0x33f
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc001798480, 0x5cad860, 0xc00693ac90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:802 +0x889
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc000f4ba80, 0x5cad820, 0xc00257c6c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:701 +0x174
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc0064dc2d0, 0x5cad820, 0xc00257c6c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:807 +0x1d2
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x5cad820, 0xc00257c6c0, 0xc002c68bd8, 0xc004456b00, 0x0, 0xc002c68da8)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:626 +0x165
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc0064dc2d0, 0x5cad820, 0xc00257c6c0, 0xc004456b00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:500 +0xfb
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor.func2(0x5cad820, 0xc00257c6c0, 0xc0064dc2d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:360 +0x9dc
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x5cad820, 0xc00257c6c0, 0xc0064dc2d0, 0x9d051c23b937ffb5, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:664 +0x5f
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).exec(0xc0064dc2d0, 0x5cad820, 0xc00257c6c0, 0xc00139dfa0, 0x1, 0xc0064dc2d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:704 +0xea
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc000f4ba80, 0x5cad820, 0xc00257c6c0, 0xc0018d29c0, 0x4e99ef7, 0x37)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:663 +0x162
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor(0xc004ce8000, 0x5cad820, 0xc00257c6c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:287 +0x173a
github.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).processAttempt(0xc0003372c0, 0x5cad820, 0xc00257c6c0, 0xc004ce8000, 0xc000c48410, 0x5cad820, 0xc00257c6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:187 +0xb3e
github.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).process(0xc0003372c0, 0x5cad820, 0xc00257c6c0, 0xc004ce8000, 0xc000c48410, 0x4e1ba01, 0xd)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:141 +0x7a
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x5cad820, 0xc00257c6c0, 0xdf8475800, 0x5cad820)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:890 +0x24e
github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x5cad820, 0xc00257c6c0, 0xc00319a5e0, 0x1e, 0xdf8475800, 0xc00693b5c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0xb1
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc00047f340, 0x5cad860, 0xc00693b530, 0x5d12340, 0xc004ce8000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:852 +0x36a
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x5cad860, 0xc00693b4d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:779 +0xfa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0014e65a0, 0x5cad860, 0xc00693b4d0, 0xc0036582a0, 0x29, 0x0, 0x0, 0xc00693b500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x14b

@tbg tbg mentioned this issue Aug 29, 2019
Closed
14 tasks
@tbg tbg changed the title teamcity: failed test: TestParallel teamcity: failed test: TestParallel F190829 05:35:59.694130 34381 roachpb/data.go:1265 [n2,split,s2,r60/3:/System/tsd/cr.node.sql.txn.…,txn=44f2c9de]: invalid retryable err Aug 29, 2019
@tbg tbg changed the title teamcity: failed test: TestParallel F190829 05:35:59.694130 34381 roachpb/data.go:1265 [n2,split,s2,r60/3:/System/tsd/cr.node.sql.txn.…,txn=44f2c9de]: invalid retryable err teamcity: failed test: TestParallel invalid retryable err Aug 29, 2019
@tbg
Copy link
Member

tbg commented Aug 29, 2019

This seems pretty straightforward,

cockroach/pkg/roachpb/errors.go

Lines 793 to 799 in e99ffed

func (e *MixedSuccessError) canRestartTransaction() TransactionRestart {
// MixedSuccessError inherits its "restartability" from the error that it wraps.
if r, ok := e.Wrapped.GetInner().(transactionRestartError); ok {
return r.canRestartTransaction()
}
return TransactionRestart_NONE
}

but then we don't handle it properly where the assertion fails in handleRetryableErrLocked. Is that all there is to it @nvanbenschoten?

tbg added a commit to tbg/cockroach that referenced this issue Aug 29, 2019
@tbg
roachpb: unwrap MixedSuccess when preparing txn for retry
3f0bc0d 
Closes cockroachdb#40328.

Release note: None
@tbg tbg mentioned this issue Aug 29, 2019
Closed
@nvanbenschoten
Copy link
Member

How did this error get past the txnSpanRefresher? That's where it should be unwrapped, and we include that interceptor on both root and leaf nodes.

@nvanbenschoten nvanbenschoten mentioned this issue Aug 29, 2019
Merged
craig bot pushed a commit that referenced this issue Aug 30, 2019
@nvanbenschoten
Merge #40356
75a7258 
40356: kv: unwrap MixedSuccessError even when retry attempts exhausted r=nvanbenschoten a=nvanbenschoten

Fixes #40328.
Closes #40332.

This commit fixes a bug where MixedSuccessErrors would not be unwrapped by the txnSpanRefresher when a request had exhausted its refresh attempt limit.

The theory is that the race test plus the background stats computation in `TestParallel/create_stats` was causing a lot of transaction retries, which exhausted `maxTxnRefreshAttempts`.

The PR also improves the documentation on the `txnSpanRefresher`. It adds a comment describing the interceptors role in the transaction model, which is similar to the comments we have on the other interceptors.

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
@craig craig bot closed this as completed in b356962 Aug 30, 2019
@andy-kimball andy-kimball mentioned this issue Sep 3, 2019
Closed
53 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
3 participants
@tbg @nvanbenschoten @cockroach-teamcity