ccl/streamingccl/streamingest: TestTenantStreamingPauseResumeIngestion failed #107434
Assignees
Labels
branch-release-23.1
Used to mark GA and release blockers, technical advisories, and bugs for 23.1
C-test-failure
Broken test (automatically or manually discovered).
O-robot
Originated from a bot.
T-disaster-recovery
Milestone
Comments
cockroach-teamcity
added
branch-release-23.1
|
@lidorcarmel I think what is happening here is that the test runs to completion before the shared-process test tenant starts up on one of the nodes in the cluster. I know you are looking into another test in this same area so I've assigned to you for now. |
|
it doesn't repro locally, so I added a log fatal here (meaning failing without a retry): and we crash with: are we trying to stop tenant |
|
Fwiw i am.able to reproduce this locally under stress after about 10-15 runs |
|
I'm running: on the gce worker, and I get: |
|
ah! 400 runs didn't repro, but without the |
craig bot
pushed a commit
that referenced
this issue
Jul 27, 2023
107474: cli/zip: emit SQL table data using TSV by default r=abarganier a=knz Fixes #107473. Epic: CRDB-28893 This is a partial revert of 35738d4. It changes the default value of the `--format` flag back from JSON to TSV. Release note (backward-incompatible change): THIS RELEASE NOTE CANCELS THE CORRESPONDING PREVIOUS BACKWARD-INCOMPATIBLE CHANGE. New behavior, compatible with previous versions of CockroachDB: the command `cockroach debug zip` stores data retrieved from SQL tables in the remote cluster using the TSV format by default. Release note (cli change): The default value of the `--format` parameter to `cockroach debug zip` is `tsv`, like other CLI commands that can extract SQL data. 107489: sql: Delete invalid TestDropColumnAfterMutations test r=rafiss a=rimadeodhar This test checks the functionality for the following sequence of events: 1. A txn adds a constraint to a column on a table. 2. A separate txn drops the column. However, this interaction between the two txns has been made explicit by the PR #92289. Since this PR, step (2) will fail if the constraint in step (1) is in the process of being added. As a result, the elaborate set up and sequence of events being tested in TestDropColumnAfterMutations is no longer necessary. Release note: none Epic: none Fixes: #76843 107664: authors: add Angela Dietz to authors r=angeladietz a=angeladietz Release note: None Epic: None 107666: server: fix a race in tenant creation r=knz a=lidorcarmel Previously, scanTenantsForRunnableServices() was not holding the mutex when SELECTing for the existing tenant names, which means that the following may happen: - scanTenantsForRunnableServices() sees that only the system tenant exists - createServerEntryLocked() then adds another tenant while holding the mutex - scanTenantsForRunnableServices() takes the lock and stops the tenant that was just created because only the system tenant should be alive (which is wrong) This patch changes scanTenantsForRunnableServices() to take the mutex before SELECTing for the existing tenants in order to avoid the race. Epic: none Fixes: #107434 Fixes: #107343 Fixes: #107154 Release note: None 107673: opt: remove Metadata.AllUserDefinedFunctions r=mgartner a=mgartner The metadata method `AllUserDefinedFunctions` has been replaced with a new function `HasUserDefinedFunctions` which provides a simpler API without exposing the underlying UDF dependency map. The map is still available outside of the opt package via the `TestingUDFDeps` method which is designed for testing use only. Epic: None Release note: None 107714: roachtest: add warning to redacted github issue r=mgartner a=mgartner Epic: None Release note: None 107716: ui: extend search logic on insights page r=koorosh a=koorosh This change extends the number of fields where search is applied (instead of single transaction/ statement execution ID field). It makes possible to search for any available ID in Txn or statement insight. Release note (ui change): search is performed on all ID fields of transaction and statement insights. Resolves: #107253 Demo: https://github.com/cockroachdb/cockroach/assets/3106437/7fb56720-3ab2-4be4-9500-457707f6f01d Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net> Co-authored-by: rimadeodhar <rima@cockroachlabs.com> Co-authored-by: Angela Dietz <dietz@cockroachlabs.com> Co-authored-by: Lidor Carmel <lidor@cockroachlabs.com> Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com> Co-authored-by: Andrii Vorobiov <and.vorobiov@gmail.com>
lidorcarmel
added a commit
that referenced
this issue
Jul 28, 2023
Previously, scanTenantsForRunnableServices() was not holding the mutex when SELECTing for the existing tenant names, which means that the following may happen: - scanTenantsForRunnableServices() sees that only the system tenant exists - createServerEntryLocked() then adds another tenant while holding the mutex - scanTenantsForRunnableServices() takes the lock and stops the tenant that was just created because only the system tenant should be alive (which is wrong) This patch changes scanTenantsForRunnableServices() to take the mutex before SELECTing for the existing tenants in order to avoid the race. Epic: none Fixes: #107434 Release note: None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ccl/streamingccl/streamingest.TestTenantStreamingPauseResumeIngestion failed with artifacts on release-23.1 @ 5fe2141d9f7c5e429e3e589a4dd722aa12840e53:
Parameters:
TAGS=bazel,gss,deadlockHelp
See also: How To Investigate a Go Test Failure (internal)
This test on roachdash | Improve this report!
Jira issue: CRDB-30041
The text was updated successfully, but these errors were encountered: