Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/sql/copy/copy_test: TestCopyFromRetries failed #117912

Closed
cockroach-teamcity opened this issue Jan 18, 2024 · 1 comment · Fixed by #118135
Closed

pkg/sql/copy/copy_test: TestCopyFromRetries failed #117912

cockroach-teamcity opened this issue Jan 18, 2024 · 1 comment · Fixed by #118135
Assignees
@yuzefovich
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-sql-queries SQL Queries Team
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 18, 2024 •
edited by cockroach-jira-scripts
Loading

pkg/sql/copy/copy_test.TestCopyFromRetries failed on master @ c24aedd67a89ea14e91b085d37103127d693aa1a:

=== RUN   TestCopyFromRetries
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestCopyFromRetries2687829098
    test_log_scope.go:81: use -show-logs to present logs inline
    copy_in_test.go:630: -- test log scope end --
test logs left over in: outputs.zip/logTestCopyFromRetries2687829098
--- FAIL: TestCopyFromRetries (101.87s)
=== RUN   TestCopyFromRetries/retries_successfully_on_every_batch_(errors_before)
    test_server_shim.go:157: automatically injected an external process virtual cluster under test; see comment at top of test_server_shim.go for details.
    copy_in_test.go:626: 
        	Error Trace:	pkg/sql/copy/copy_test/pkg/sql/copy/copy_in_test.go:626
        	Error:      	Received unexpected error:
        	            	sql: database is closed
        	Test:       	TestCopyFromRetries/retries_successfully_on_every_batch_(errors_before)
    --- FAIL: TestCopyFromRetries/retries_successfully_on_every_batch_(errors_before) (79.82s)

Parameters:

  • attempt=1
  • run=7
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-35407
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Jan 18, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Jan 18, 2024
@github-project-automation github-project-automation bot moved this to Triage in SQL Queries Jan 18, 2024
@cockroach-teamcity
Copy link
Member Author

pkg/sql/copy/copy_test.TestCopyFromRetries failed with artifacts on master @ c24aedd67a89ea14e91b085d37103127d693aa1a:

	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x89
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x0?, {0x666dba8, 0xc00f358840}, 0x0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x30
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc00a362000, {0x666dba8, 0xc00f358840}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:797 +0x86e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc00a362108, {0x666dba8, 0xc00f358840}, {0x66b0f40?, 0xc007cc22d8})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:230 +0x808
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7f6d2c8b7ce8?, 0xc00a3624e8}, 0xc022c7c550)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc022c7cad8, {0x666dba8, 0xc008594a50}, {0x66a0610, 0xc00a362488})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1e8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc022c7cad8, {0x666dba8, 0xc008594a50})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1059 +0x1439
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:744 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc015471500, 0xc014f0ef90?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc012146620, {0x666dba8, 0xc004dd2e70}, {0x6666940, 0xc015471500}, 0xc0153a1800)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x666dba8?, 0xc004dd2e70?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 4553586
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Other goroutines holding locks:
goroutine 4557433 lock 0xc00f3c6a38
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4729 concurrency.(*lockTableImpl).verify ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4728 concurrency.(*lockTableImpl).verify ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:98 concurrency.verifyingLockTable.UpdateLocks ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:551 concurrency.(*managerImpl).OnLockUpdated ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:795 kvserver.(*Replica).handleReadWriteLocalEvalResult ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:233 kvserver.(*replicaStateMachine).ApplySideEffects ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 apply.mapCheckedCmdIter ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 apply.(*Task).applyOneBatch ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:250 apply.(*Task).ApplyCommittedEntries ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:453 kvserver.(*replicaStateMachine).moveStats ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:743 kvserver.(*Replica).handleRaftReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:688 kvserver.(*Store).processReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:420 kvserver.(*raftSchedulerShard).worker ???
GOROOT/src/sync/waitgroup.go:86 sync.(*WaitGroup).Done ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 stop.(*Stopper).RunAsyncTaskEx.func2 ???



=== RUN   TestCopyFromRetries/does_not_attempt_to_retry_if_disabled
=== RUN   TestCopyFromRetries/does_not_retry_inside_a_txn
=== RUN   TestCopyFromRetries/failure_in_atomic_transaction_does_not_retry

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@yuzefovich yuzefovich added skipped-test and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 18, 2024
@yuzefovich yuzefovich self-assigned this Jan 18, 2024
@yuzefovich yuzefovich moved this from Triage to Active in SQL Queries Jan 18, 2024
@yuzefovich yuzefovich mentioned this issue Jan 18, 2024
Merged
craig bot pushed a commit that referenced this issue Jan 19, 2024
@nvanbenschoten @rickystewart @yuzefovich
Merge #117840 #117899 #117940 #117950
2a98613 
117840: kv: promote expiration-based lease to epoch without sequence number change r=erikgrinaker a=nvanbenschoten

Fixes #117630.
Fixes #90656.
Fixes #98553.

Informs #61986.
Informs #115191.

This commit updates the post-lease transfer promotion of expiration-based leases to epoch-based leases to not change the sequence number of the lease. This avoids invalidating all requests proposed under the original expiration-based lease, which can lead to `RETRY_ASYNC_WRITE_FAILURE` errors.

The change accomplishes this by updating the `Lease.Equivalent` method to consider an expiration-based lease to be equivalent to an epoch-based lease that is held by the same replica and has the same start time. Doing so requires some care, because lease equivalency is checked below Raft and needs to remain deterministic across binary versions.

This change requires a cluster version check, so it cannot be backported.

Release note (bug fix): Improved an interaction during range lease transfers which could previously cause `RETRY_ASYNC_WRITE_FAILURE` errors to be returned to clients.

117899: backupccl: skip `TestBackupRestoreAppend` under `deadlock` r=rail a=rickystewart

These tests are likely to time out.

Epic: CRDB-8308
Release note: None

117940: backupccl,sql: skip a couple more tests under duress r=rail a=rickystewart

These tests are all timing out. For the failures that seem suspect in some way, I have filed GitHub issues.

Epic: CRDB-8308
Release note: None

117950: copy: skip TestCopyFromRetries for now r=yuzefovich a=yuzefovich

We recently expanded this test and it became flaky. Skip it until we stabilize it.

Informs: #117912.

Release note: None

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
@yuzefovich yuzefovich mentioned this issue Jan 22, 2024
Merged
@craig craig bot closed this as completed in 549c18e Jan 23, 2024
@github-project-automation github-project-automation bot moved this from Active to Done in SQL Queries Jan 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzefovich yuzefovich

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

copy: deflake TestCopyFromRetries yuzefovich/cockroach
2 participants
@yuzefovich @cockroach-teamcity