kv/kvserver: TestRangefeedCheckpointsRecoverFromLeaseExpiration failed

[cockroach-teamcity]

kv/kvserver.TestRangefeedCheckpointsRecoverFromLeaseExpiration failed with artifacts on master @ 28944f9adf4df64a7e059a98b5b90adcc4b20d1d:

=== RUN   TestRangefeedCheckpointsRecoverFromLeaseExpiration
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRangefeedCheckpointsRecoverFromLeaseExpiration1528879470
    test_log_scope.go:80: use -show-logs to present logs inline
    replica_rangefeed_test.go:1049: 
        	Error Trace:	replica_rangefeed_test.go:1049
        	Error:      	Received unexpected error:
        	            	request not evaluated locally; evaluated by s1 instead of local s2
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*TestServer).GetRangeLease
        	            	  | 	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:1315
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).FindRangeLeaseEx
        	            	  | 	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1046
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRangefeedCheckpointsRecoverFromLeaseExpiration
        	            	  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed_test.go:1048
        	            	  | testing.tRunner
        	            	  | 	/usr/local/go/src/testing/testing.go:1259
        	            	  | runtime.goexit
        	            	  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
        	            	Wraps: (2) request not evaluated locally; evaluated by s1 instead of local s2
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        	Test:       	TestRangefeedCheckpointsRecoverFromLeaseExpiration
    panic.go:642: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRangefeedCheckpointsRecoverFromLeaseExpiration1528879470
--- FAIL: TestRangefeedCheckpointsRecoverFromLeaseExpiration (0.92s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• GOFLAGS=-json

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

[irfansharif]

Lines up with #73876. Easy to repro:

$ dev test pkg/kv/kvserver -f TestRangefeedCheckpointsRecoverFromLeaseExpiration --stress
[...]
3 runs completed, 1 failures, over 3s

[irfansharif]

I220126 17:42:37.542758 1862 kv/kvserver/pkg/kv/kvserver/replica_range_lease.go:1187  [n2,s2,r44/2:/{Table/Max-Max}] 170  request range lease (attempt #1)
[...]
I220126 17:42:37.546498 1862 kv/kvserver/pkg/kv/kvserver/replica_range_lease.go:1273  [n2,s2,r44/2:/{Table/Max-Max}] 176  lease acquisition succeeded: <empty>
[...]
I220126 17:42:37.741712 26 testutils/testcluster/testcluster.go:1037  [-] 181  DEBUG: using local query policy for r44:/{Table/Max-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=3, sticky=9223372036.854775807,2147483647] for store: 2
    replica_rangefeed_test.go:1049: 
        	Error Trace:	replica_rangefeed_test.go:1049
        	Error:      	Received unexpected error:
        	            	request not evaluated locally; evaluated by s1 instead of local s2

Hm, strange. The test expects r44's lease to be on n2, and it is, but the assertion trips up anyway. Is the LeaseInfo not returning the most up-to-date thing?

[cockroach-teamcity]

kv/kvserver.TestRangefeedCheckpointsRecoverFromLeaseExpiration failed with artifacts on master @ 659fe193978a65187fcce81e985e14f9e2de9606:

=== RUN   TestRangefeedCheckpointsRecoverFromLeaseExpiration
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRangefeedCheckpointsRecoverFromLeaseExpiration4208452139
    test_log_scope.go:80: use -show-logs to present logs inline
    replica_rangefeed_test.go:1049: 
        	Error Trace:	replica_rangefeed_test.go:1049
        	Error:      	Received unexpected error:
        	            	request not evaluated locally; evaluated by s1 instead of local s2
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*TestServer).GetRangeLease
        	            	  | 	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:1323
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).FindRangeLeaseEx
        	            	  | 	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1051
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRangefeedCheckpointsRecoverFromLeaseExpiration
        	            	  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed_test.go:1048
        	            	  | testing.tRunner
        	            	  | 	/usr/local/go/src/testing/testing.go:1259
        	            	  | runtime.goexit
        	            	  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
        	            	Wraps: (2) request not evaluated locally; evaluated by s1 instead of local s2
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        	Test:       	TestRangefeedCheckpointsRecoverFromLeaseExpiration
    panic.go:642: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRangefeedCheckpointsRecoverFromLeaseExpiration4208452139
--- FAIL: TestRangefeedCheckpointsRecoverFromLeaseExpiration (1.40s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• GOFLAGS=-json

This test on roachdash | Improve this report!