[DocDB] Update blocking transaction info for waiting requests without re-running conflict resolution #16286
Assignees
Labels
area/docdb
YugabyteDB core features
kind/bug
This issue is a bug
priority/medium
Medium priority issue
Comments
robertsami
added
area/docdb
yugabyte-ci
added
kind/bug
basavaraj29
added a commit
that referenced
this issue
Mar 14, 2023
…oned workloads in use of WaitQueue and DeadlockDetector Summary: This diff enables wait-queues and deadlock detection for cross-region transactions. When wait-queues are enabled and wait-on conflict policy is set, each transaction enters the wait queue if it finds blocker transactions, post undergoing conflict resolution. The waiter txn records blocker_info (txn id, status tablet, conflicting subtxns etc) for each blocking transaction and registers itself with the deadlock detector. All this information is fetched from the tablet's transaction participant. In case of transaction promotion, all the involved transaction participants are notified of the updated status tablet location. The transaction gets promoted to global successfully only when all of the txn participants acknowledge this update, else the transaction is aborted. If successful, the old status tablet might be in kPending state until the commit time, at which it changes to kAborted state. In this diff, a `TransactionStausListerner` interface is introduced and that `WaitQueue` implements this interface. The wait-queue is notified on each transaction promotion. When notified, the wait queue submits a task of type `UpdateWaitersOnBlockerPromotion` to a background threadpool. When some thread gets a chance to execute the submitted task, it does the following - it checks if the transaction promoted was a waiter txn. If so, it is made to re-enter the wait queue by re-running conflict resolution. This ensures that the waiter transaction would definitely fetch the updated status tablet and the same is registered with the deadlock detector. - it forces all waiter transactions blocking on this promoted transaction to re-enter the wait queue. This ensures that the wait-for dependency is updated with the latest blocker txn's status tablet and the wait-for probes are forwarded to the right txn coordinator. There might be a control race encountered between a waiter txn entering the queue with the blocker's old status tablet and the wait queue process the promotion signal. The waiter wouldn't renter the queue if the signal gets processed before it entering the queue for the first time, and it follows that the deadlock detector wouldn't be aware of the latest wait-for dependencies. To prevent this, we check with the transaction participant on the latest status tablet of the blocker transaction on inserts to the wait-queue. This is done by acquiring a mutex, which is also acquired by while processing the promotion signal. Summarizing the changes, txn promotion could result in either a success (which leads to updated status tablet location) or a failure (aborted state)., In either case, the wait queue receives a signal and the waiter transactions waiting on the promoted/aborted blocker re-run conflict resolution (Currently wait-queue periodically polls for txn status, and hence aborted/committed cases are taken care of. Rob is working on a parallel diff where this too is being changed to a notification mechanism). Additionally, there is a backup mechanism in place where we force all waiter transactions in the wait-queue to re-run conflict resolution periodically. It follows that the deadlock detector would be updated with the latest wait-for dependencies periodically. User facing aspects - Since we don't execute the above logic of transaction promotion in-line with the promotion request, the shouldn't be any noticeable increase in latency for transaction promotion. The only concerning aspect would be that we re-run conflict resolution for the affected transactions when one of their blocker(s) get promoted. But this is necessary for maintaining up to date wait-for dependencies. And since the size of intentsdb should be considerably small, this shouldn't be much of a concern. Test Plan: Jenkins ``` ./yb_build.sh --cxx-test pgwrapper_geo_transactions_promotion-test --gtest_filter DeadlockDetectionWithTxnPromotionTest.TestBlockerPromotionWithDeadlock ./yb_build.sh --cxx-test pgwrapper_geo_transactions_promotion-test --gtest_filter DeadlockDetectionWithTxnPromotionTest.TestBlockerPromotionWithoutDeadlock ./yb_build.sh --cxx-test pgwrapper_geo_transactions_promotion-test --gtest_filter DeadlockDetectionWithTxnPromotionTest.TestDeadlockAmongstGlobalTransactions ./yb_build.sh --cxx-test pgwrapper_geo_transactions_promotion-test --gtest_filter DeadlockDetectionWithTxnPromotionTest.TestWaiterPromotionWithoutDeadlock ./yb_build.sh --cxx-test pgwrapper_geo_transactions_promotion-test --gtest_filter DeadlockDetectionWithTxnPromotionTest.TestWaiterPromotionWithDeadlock ./yb_build.sh --cxx-test pgwrapper_geo_transactions_promotion-test --gtest_filter DeadlockDetectionWithTxnPromotionTest.TestDelayedWaiterRegistrationInWaitQeue ./yb_build.sh --cxx-test pgwrapper_pg_wait_on_conflict-test --gtest_filter PgWaitQueuesTest.TestWaiterTxnReRunConflictResolution ``` Reviewers: esheng, sergei, rsami Reviewed By: rsami Subscribers: pjain, jenkins-bot, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D23215
this was complete in bdec10c This github issue is now tracking a better solution:
|
robertsami
changed the title
This was referenced Dec 6, 2023
|
The impact is not clear - whether it impacts fairness etc. |
|
Would be good to tackle this higher than the other backlog items. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-5712
Description
We can currently run into a case like the following:
At this point, we have a deadlock, but the coordinator for txn2 is not aware that txn2 is blocked on txn3, so we will not detect the deadlock
In bdec10c, we introduced a change to mitigate this issue by periodically re-running conflict resolution for waiting transactions, so their blocker list is eventually up-to-date at the coordinator
A better solution would be to keep the blocker list of each waiter up-to-date by forcing incoming requests to update waiters in the wait queue in real time
The text was updated successfully, but these errors were encountered: