[docdb] Investigate raft race between incoming write operations and background applies #3477
Assignees
Labels
area/docdb
YugabyteDB core features
kind/bug
This issue is a bug
priority/medium
Medium priority issue
Comments
rajukumaryb
added a commit
that referenced
this issue
Feb 7, 2020
…condition between insert and truncate; disable rocksdb flush on truncate Summary: - Prevent rocksdb instance from calling `ListenFilesChanged` callback after being detached from `Tablet` Full stack available at - #3288 (comment) Follow Up Work: #3476 - Prevent race between `Tablet::AcquireLocksAndPerformDocOperations() -> Tablet::StartDocWriteOperation()` and `Tablet::Truncate()` by incrementing `pending_op_counter_` Full stack available at - #3288 (comment) - Do not flush rocksdb memtable when user truncates table to prevent following crash on flush - ``` #0 0x000056371c706b00 in ?? () #1 0x00007f5229291099 in std::__invoke_impl<void, void (yb::tablet::Tablet::*&)(), yb::tablet::Tablet*&> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/7/bits/invoke.h:73 #2 std::__invoke<void (yb::tablet::Tablet::*&)(), yb::tablet::Tablet*&> (__fn=<optimized out>) at /usr/include/c++/7/bits/invoke.h:95 #3 std::_Bind<void (yb::tablet::Tablet::*(yb::tablet::Tablet*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (__args=..., this=<optimized out>) at /usr/include/c++/7/functional:467 #4 std::_Bind<void (yb::tablet::Tablet::*(yb::tablet::Tablet*))()>::operator()<, void>() (this=<optimized out>) at /usr/include/c++/7/functional:551 #5 std::_Function_handler<void (), std::_Bind<void (yb::tablet::Tablet::*(yb::tablet::Tablet*))()> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316 #6 0x00007f5225aee92c in std::function<void ()>::operator()() const (this=0x56371c5534f0) at /usr/include/c++/7/bits/std_function.h:706 #7 rocksdb::DBImpl::FilesChanged (this=this@entry=0x56371c552b00) at ../../src/yb/rocksdb/db/db_impl.cc:4359 #8 0x00007f5225b14b1b in rocksdb::DBImpl::BackgroundCallFlush (this=this@entry=0x56371c552b00, cfd=cfd@entry=0x0) at ../../src/yb/rocksdb/db/db_impl.cc:3285 ``` Full stack available at - #3288 (comment) - WORKAROUND for race condition (#3288 (comment)) by using `ScopedPendingOperation` in `Tablet::ShouldApplyWrite()` to prevent it from seeing a null `regular_db_` due to a concurrent `Tablet::Truncate()` Full stack available at - #3288 (comment) Follow Up Work: #3477 Test Plan: ./yb_build.sh debug --cxx-test pg_libpq-test --gtest_filter PgLibPqTest.ConcurrentInsertTruncateForeignKey ./yb_build.sh debug --cxx-test snapshot-txn-test --gtest_filter SnapshotTxnTest.MultiWriteWithRestart Reviewers: bogdan, sergei, mikhail Reviewed By: mikhail Subscribers: kannan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7808
yugabyte-ci
added
kind/bug
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-1383
See #3288 (comment) for background apply of
Truncateracing with a foregroundRaftConsensus::UpdateReplica.Is this really an out of order execution (as we were considering offline), or more just that, if Truncate was op id X, we are now applying it, while also processing some incoming operation at id X+1 and these can race on tablet/rocksdb access?
The text was updated successfully, but these errors were encountered: