[DocDB] Deadlock with yql partitions vtable and delete tables #16109
Assignees
Labels
Comments
hulien22
added
area/docdb
yugabyte-ci
added
kind/bug
|
Was not caught by TSAN deadlock detection in |
hulien22
added a commit
that referenced
this issue
Feb 17, 2023
Summary: Fixing lock order inversion with yql system.partitions building and table deletions: One thread doing deletes in catalog manager is holding CM's mutex_ and blocked on getting the yqlpartitions mutex_: ``` #5 std::lock_guard<std::shared_timed_mutex>::lock_guard (__m=..., this=<synthetic pointer>) at /cases/home/yugabyte/yb-software/yugabyte-2.12.5.0/linuxbrew-xxxxxxxxxxxxxxxxxxxxxxxx/Cellar/gcc/5.5.0_4/include/c++/5.5.0/mutex:386 #6 yb::master::YQLPartitionsVTable::RemoveFromCache (this=0x182091e0, table_id=...) at ../../src/yb/master/yql_partitions_vtable.cc:262 #7 0x00007f384c0931e5 in yb::master::CatalogManager::DeleteTableInternal (this=this@entry=0x275a000, req=req@entry=0x225a2138, resp=resp@entry=0x225a2160, rpc=rpc@entry=0x7f3815561cb0) at ../../src/yb/master/catalog_manager.cc:4831 ``` And another thread rebuilding the vtable has the yqlpartitions lock and is waiting on the CM mutex_: ``` #4 yb::NonRecursiveSharedLock<yb::rw_spinlock>::NonRecursiveSharedLock (this=0x7f3767405da0, mutex=...) at ../../src/yb/util/debug/lock_debug.h:41 #5 0x00007f384c04a4b1 in yb::master::CatalogManager::GetTables (this=0x275a000, mode=yb::master::GetTablesMode::kVisibleToClient) at ../../src/yb/master/catalog_manager.cc:5885 #6 0x00007f384c2148f2 in yb::master::YQLPartitionsVTable::GenerateAndCacheData (this=0x182091e0) at ../../src/yb/master/yql_partitions_vtable.cc:140 #7 0x00007f384c03c35d in yb::master::CatalogManager::RebuildYQLSystemPartitions (this=0x275a000) at ../../src/yb/master/catalog_manager.cc:10486 ``` For now keeping the default of generating the partitions cache on changes, due to create table perf concerns of the purely bg thread approach Test Plan: ``` ybd tsan --gtest_filter CppCassandraDriverTest.YQLPartitionsVtableCacheRefresh ``` Reviewers: bogdan, sergei, asrivastava Reviewed By: asrivastava Subscribers: kannan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D22943
hulien22
added a commit
that referenced
this issue
Feb 18, 2023
Summary: Original commit: 68fae2f / D22943 Fixing lock order inversion with yql system.partitions building and table deletions: One thread doing deletes in catalog manager is holding CM's mutex_ and blocked on getting the yqlpartitions mutex_: ``` ``` And another thread rebuilding the vtable has the yqlpartitions lock and is waiting on the CM mutex_: ``` ``` For now keeping the default of generating the partitions cache on changes, due to create table perf concerns of the purely bg thread approach Test Plan: ``` ybd tsan --gtest_filter CppCassandraDriverTest.YQLPartitionsVtableCacheRefresh ``` Reviewers: bogdan, sergei, asrivastava Reviewed By: asrivastava Subscribers: ybase, kannan Differential Revision: https://phabricator.dev.yugabyte.com/D23006
yugabyte-ci
added
priority/critical
hulien22
added a commit
that referenced
this issue
Mar 8, 2023
Summary: Original commit: 68fae2f / D22943 Fixing lock order inversion with yql system.partitions building and table deletions: One thread doing deletes in catalog manager is holding CM's mutex_ and blocked on getting the yqlpartitions mutex_: ``` ``` And another thread rebuilding the vtable has the yqlpartitions lock and is waiting on the CM mutex_: ``` ``` For now keeping the default of generating the partitions cache on changes, due to create table perf concerns of the purely bg thread approach Test Plan: ``` ybd tsan --gtest_filter CppCassandraDriverTest.YQLPartitionsVtableCacheRefresh ``` Reviewers: bogdan, sergei, asrivastava Reviewed By: asrivastava Subscribers: ybase, kannan Differential Revision: https://phabricator.dev.yugabyte.com/D23005
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-5514
Description
Lock order inversion causing deadlock.
Has catalog manager mutex_:
Has vtable mutex:
The text was updated successfully, but these errors were encountered: