Using ReadLockGuard/WriteLockGuard in Context might block user requests

[git-hulk]

Search before asking

• I had searched in the issues and found no similar issues.

Version

unstable

Minimal reproduce step

take backup from production (bgsave), rsync to test server, rut it and wait with ugly script:

while true; do timeout 1 redis-cli -p 6666 info > /dev/null || date; sleep 0.5; printf . ; done

What did you expect to see?

Compaction won't block user requests

What did you see instead?

User requests are blocked due to waiting for DB lock.

Anything Else?

root@kvrocks-novikov-0:/# gdb -batch -p 1 -ex "thread apply all bt"
[New LWP 7]
[New LWP 8]
[New LWP 9]
[New LWP 10]
[New LWP 26]
[New LWP 27]
[New LWP 28]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[New LWP 35]
[New LWP 36]
[New LWP 37]
[New LWP 38]
[New LWP 39]
[New LWP 40]
[New LWP 41]
[New LWP 42]
[New LWP 43]
[New LWP 44]
[New LWP 45]
[New LWP 1172]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=44, futex_word=0xffff7ed36c90) at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.

Thread 26 (Thread 0xffff5e3f6bc0 (LWP 1172) "rocksdb:low"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xffff5e3f5470) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xffff5e3f5470) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffff5e3f5470, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7dae1d0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa7834f10, cond=0xffff5e3f5448) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0xffff5e3f5448, mutex=0xffffa7834f10) at ./nptl/pthread_cond_wait.c:618
#5  0x0000aaaad6580b80 in rocksdb::port::CondVar::Wait() ()
#6  0x0000aaaad6502e4c in rocksdb::GenericRateLimiter::Request(long, rocksdb::Env::IOPriority, rocksdb::Statistics*) ()
#7  0x0000aaaad6502168 in rocksdb::RateLimiter::RequestToken(unsigned long, unsigned long, rocksdb::Env::IOPriority, rocksdb::Statistics*, rocksdb::RateLimiter::OpType) ()
#8  0x0000aaaad63f9360 in rocksdb::WritableFileWriter::WriteBuffered(rocksdb::IOOptions const&, char const*, unsigned long) ()
#9  0x0000aaaad63fc144 in rocksdb::WritableFileWriter::Flush(rocksdb::IOOptions const&) ()
#10 0x0000aaaad63fe48c in rocksdb::WritableFileWriter::Append(rocksdb::IOOptions const&, rocksdb::Slice const&, unsigned int) ()
#11 0x0000aaaad663071c in rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*) ()
#12 0x0000aaaad6631258 in rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType) ()
#13 0x0000aaaad6632200 in rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType) ()
#14 0x0000aaaad6634548 in rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&) ()
#15 0x0000aaaad6206fec in rocksdb::CompactionOutputs::AddToOutput(rocksdb::CompactionIterator const&, std::function<rocksdb::Status (rocksdb::CompactionOutputs&)> const&, std::function<rocksdb::Status (rocksdb::CompactionOutputs&, rocksdb::Status const&, rocksdb::Slice const&)> const&) ()
#16 0x0000aaaad620a468 in rocksdb::SubcompactionState::AddToOutput(rocksdb::CompactionIterator const&, std::function<rocksdb::Status (rocksdb::CompactionOutputs&)> const&, std::function<rocksdb::Status (rocksdb::CompactionOutputs&, rocksdb::Status const&, rocksdb::Slice const&)> const&) ()
#17 0x0000aaaad61e1584 in rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::SubcompactionState*) ()
#18 0x0000aaaad69052dc in execute_native_thread_routine ()
#19 0x0000ffffa7daee30 in start_thread (arg=0xffffa7073f27) at ./nptl/pthread_create.c:442
#20 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 25 (Thread 0xffff7e326bc0 (LWP 45) "compact-check"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xffffa79ad7a8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xffffa79ad7a8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffffa79ad7a8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7dae1d0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa79ad380, cond=0xffffa79ad780) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0xffffa79ad780, mutex=0xffffa79ad380) at ./nptl/pthread_cond_wait.c:618
#5  0x0000aaaad6580b80 in rocksdb::port::CondVar::Wait() ()
#6  0x0000aaaad640d38c in rocksdb::InstrumentedCondVar::Wait() ()
#7  0x0000aaaad6273b8c in rocksdb::DBImpl::RunManualCompaction(rocksdb::ColumnFamilyData*, int, int, rocksdb::CompactRangeOptions const&, rocksdb::Slice const*, rocksdb::Slice const*, bool, bool, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int*) ()
#8  0x0000aaaad6274628 in rocksdb::DBImpl::CompactRangeInternal(rocksdb::CompactRangeOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const*, rocksdb::Slice const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#9  0x0000aaaad627519c in rocksdb::DBImpl::CompactRange(rocksdb::CompactRangeOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const*, rocksdb::Slice const*) ()
#10 0x0000aaaad60e4ccc in engine::Storage::Compact(rocksdb::ColumnFamilyHandle*, rocksdb::Slice const*, rocksdb::Slice const*) ()
#11 0x0000aaaad615f1b4 in CompactionChecker::PickCompactionFilesForCf(engine::ColumnFamilyConfig const&) [clone .isra.0] ()
#12 0x0000aaaad60bc730 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<Server::Start()::{lambda()#2}>(char const*, Server::Start()::{lambda()#2})::{lambda()#1}> > >::_M_run() [clone .lto_priv.0] ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fbe7) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 24 (Thread 0xffff7ed36bc0 (LWP 44) "server-cron"):
#0  0x0000ffffa7de36b0 in __GI___clock_nanosleep (clock_id=<optimized out>, clock_id@entry=0, flags=flags@entry=0, req=0xffff7ed35d30, rem=0xffff7ed35d30) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
#1  0x0000ffffa7de861c in __GI___nanosleep (req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x0000aaaad60b2c04 in Server::cron() ()
#3  0x0000aaaad69052dc in execute_native_thread_routine ()
#4  0x0000ffffa7daee30 in start_thread (arg=0xffffe904fbe7) at ./nptl/pthread_create.c:442
#5  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 23 (Thread 0xffff7f746bc0 (LWP 43) "task-runner"):
#0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
#1  0x0000aaaad618fab8 in tbb::detail::r1::wait_bounded_queue_monitor(tbb::detail::r1::concurrent_monitor*, unsigned long, long, tbb::detail::d1::delegate_base&) ()
#2  0x0000aaaad603ccd4 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<TaskRunner::Start()::{lambda()#1}>(char const*, TaskRunner::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#3  0x0000aaaad69052dc in execute_native_thread_routine ()
#4  0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb47) at ./nptl/pthread_create.c:442
#5  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 22 (Thread 0xffff80156bc0 (LWP 42) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 21 (Thread 0xffff80b66bc0 (LWP 41) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 20 (Thread 0xffff81576bc0 (LWP 40) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 19 (Thread 0xffff81f86bc0 (LWP 39) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 18 (Thread 0xffff82996bc0 (LWP 38) "worker"):
#0  0x0000ffffa7e17c74 in __GI_epoll_pwait (epfd=83, events=0xffffa79b6200, maxevents=32, timeout=9311, set=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1  0x0000aaaad67344f8 in epoll_dispatch ()
#2  0x0000aaaad6728938 in event_base_loop.constprop ()
#3  0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#4  0x0000aaaad69052dc in execute_native_thread_routine ()
#5  0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#6  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 17 (Thread 0xffff833a6bc0 (LWP 37) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 16 (Thread 0xffff83db6bc0 (LWP 36) "worker"):
#0  0x0000ffffa7e17c74 in __GI_epoll_pwait (epfd=73, events=0xffffa79a4a00, maxevents=32, timeout=9311, set=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1  0x0000aaaad67344f8 in epoll_dispatch ()
#2  0x0000aaaad6728938 in event_base_loop.constprop ()
#3  0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#4  0x0000aaaad69052dc in execute_native_thread_routine ()
#5  0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#6  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 15 (Thread 0xffff847c6bc0 (LWP 35) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 14 (Thread 0xffff851d6bc0 (LWP 34) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 13 (Thread 0xffff85be6bc0 (LWP 33) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 12 (Thread 0xffff865f6bc0 (LWP 32) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 11 (Thread 0xffff9c7f6bc0 (LWP 31) "worker"):
#0  0x0000ffffa7e17c74 in __GI_epoll_pwait (epfd=47, events=0xffffa789bc00, maxevents=32, timeout=9311, set=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1  0x0000aaaad67344f8 in epoll_dispatch ()
#2  0x0000aaaad6728938 in event_base_loop.constprop ()
#3  0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#4  0x0000aaaad69052dc in execute_native_thread_routine ()
#5  0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#6  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 10 (Thread 0xffff9d7f6bc0 (LWP 30) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 9 (Thread 0xffffa01f6bc0 (LWP 29) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=3, futex_word=0xffffe90502fc) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=3, futex_word=futex_word@entry=0xffffe90502fc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502fc, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db5610 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:730
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 8 (Thread 0xffff983f6bc0 (LWP 28) "worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=false, abstime=0x0, op=393, expected=2, futex_word=0xffffe90502f8) at ./nptl/futex-internal.c:61
#1  __futex_abstimed_wait_common (cancel=false, private=private@entry=0, abstime=abstime@entry=0x0, clockid=clockid@entry=0, expected=expected@entry=2, futex_word=futex_word@entry=0xffffe90502f8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0xffffe90502f8, expected=expected@entry=2, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:128
#3  0x0000ffffa7db56e0 in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_common.c:829
#4  ___pthread_rwlock_wrlock (rwlock=0xffffe90502f0) at ./nptl/pthread_rwlock_wrlock.c:26
#5  0x0000aaaad5ef8b04 in engine::Context::~Context() [clone .lto_priv.0] ()
#6  0x0000aaaad60a388c in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) ()
#7  0x0000aaaad5f33424 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) ()
#8  0x0000aaaad671d5d8 in bufferevent_run_deferred_callbacks_unlocked ()
#9  0x0000aaaad67269e0 in event_process_active_single_queue ()
#10 0x0000aaaad6726e80 in event_process_active ()
#11 0x0000aaaad67289bc in event_base_loop.constprop ()
#12 0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 7 (Thread 0xffff973f6bc0 (LWP 27) "worker"):
#0  0x0000ffffa7e17c74 in __GI_epoll_pwait (epfd=27, events=0xffffa78e8600, maxevents=32, timeout=9311, set=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1  0x0000aaaad67344f8 in epoll_dispatch ()
#2  0x0000aaaad6728938 in event_base_loop.constprop ()
#3  0x0000aaaad60bcb34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() ()
#4  0x0000aaaad69052dc in execute_native_thread_routine ()
#5  0x0000ffffa7daee30 in start_thread (arg=0xffffe904fb17) at ./nptl/pthread_create.c:442
#6  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 6 (Thread 0xffff963f6bc0 (LWP 26) "kvrocks"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff963f6220, op=393, expected=0, futex_word=0xffffa782c578) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xffff963f6220, clockid=-1478838668, expected=0, futex_word=0xffffa782c578) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffffa782c578, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0xffff963f6220, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7dae4c0 in __pthread_cond_wait_common (abstime=0xffff963f6220, clockid=0, mutex=0xffffa782c508, cond=0xffffa782c550) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0xffffa782c550, mutex=0xffffa782c508, abstime=0xffff963f6220) at ./nptl/pthread_cond_wait.c:643
#5  0x0000aaaad6580c74 in rocksdb::port::CondVar::TimedWait(unsigned long) ()
#6  0x0000aaaad640d4e4 in rocksdb::InstrumentedCondVar::TimedWait(unsigned long) ()
#7  0x0000aaaad6326ac4 in rocksdb::Timer::Run() ()
#8  0x0000aaaad69052dc in execute_native_thread_routine ()
#9  0x0000ffffa7daee30 in start_thread (arg=0xffffe904d837) at ./nptl/pthread_create.c:442
#10 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 5 (Thread 0xffffa49f6bc0 (LWP 10) "rocksdb:high"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xffffa78f93d8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xffffa78f93d8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffffa78f93d8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7dae1d0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa78f9380, cond=0xffffa78f93b0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0xffffa78f93b0, mutex=0xffffa78f9380) at ./nptl/pthread_cond_wait.c:618
#5  0x0000aaaad665efd4 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#6  0x0000aaaad665f36c in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#7  0x0000aaaad69052dc in execute_native_thread_routine ()
#8  0x0000ffffa7daee30 in start_thread (arg=0xffffe904c787) at ./nptl/pthread_create.c:442
#9  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 4 (Thread 0xffffa59f6bc0 (LWP 9) "rocksdb:high"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xffffa78f93d8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xffffa78f93d8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffffa78f93d8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7dae1d0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa78f9380, cond=0xffffa78f93b0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0xffffa78f93b0, mutex=0xffffa78f9380) at ./nptl/pthread_cond_wait.c:618
#5  0x0000aaaad665efd4 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#6  0x0000aaaad665f36c in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#7  0x0000aaaad69052dc in execute_native_thread_routine ()
#8  0x0000ffffa7daee30 in start_thread (arg=0xffffe904c787) at ./nptl/pthread_create.c:442
#9  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 3 (Thread 0xffffa6666bc0 (LWP 8) "rocksdb:high"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xffffa78f93dc) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xffffa78f93dc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffffa78f93dc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7dae1d0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa78f9380, cond=0xffffa78f93b0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0xffffa78f93b0, mutex=0xffffa78f9380) at ./nptl/pthread_cond_wait.c:618
#5  0x0000aaaad665efd4 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#6  0x0000aaaad665f36c in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#7  0x0000aaaad69052dc in execute_native_thread_routine ()
#8  0x0000ffffa7daee30 in start_thread (arg=0xffffe904c787) at ./nptl/pthread_create.c:442
#9  0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 2 (Thread 0xffffa7076bc0 (LWP 7) "rocksdb:low"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffffa7073070, op=393, expected=0, futex_word=0xffffa7073120) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xffffa7073070, clockid=-1478838668, expected=0, futex_word=0xffffa7073120) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffffa7073120, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0xffffa7073070, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7dae4c0 in __pthread_cond_wait_common (abstime=0xffffa7073070, clockid=0, mutex=0xffffa7834f10, cond=0xffffa70730f8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0xffffa70730f8, mutex=0xffffa7834f10, abstime=0xffffa7073070) at ./nptl/pthread_cond_wait.c:643
#5  0x0000aaaad6580c74 in rocksdb::port::CondVar::TimedWait(unsigned long) ()
#6  0x0000aaaad6502d90 in rocksdb::GenericRateLimiter::Request(long, rocksdb::Env::IOPriority, rocksdb::Statistics*) ()
#7  0x0000aaaad6502168 in rocksdb::RateLimiter::RequestToken(unsigned long, unsigned long, rocksdb::Env::IOPriority, rocksdb::Statistics*, rocksdb::RateLimiter::OpType) ()
#8  0x0000aaaad63f9360 in rocksdb::WritableFileWriter::WriteBuffered(rocksdb::IOOptions const&, char const*, unsigned long) ()
#9  0x0000aaaad63fc144 in rocksdb::WritableFileWriter::Flush(rocksdb::IOOptions const&) ()
#10 0x0000aaaad63fe48c in rocksdb::WritableFileWriter::Append(rocksdb::IOOptions const&, rocksdb::Slice const&, unsigned int) ()
#11 0x0000aaaad663071c in rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*) ()
#12 0x0000aaaad6631258 in rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType) ()
#13 0x0000aaaad6632200 in rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType) ()
#14 0x0000aaaad6634548 in rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&) ()
#15 0x0000aaaad6206fec in rocksdb::CompactionOutputs::AddToOutput(rocksdb::CompactionIterator const&, std::function<rocksdb::Status (rocksdb::CompactionOutputs&)> const&, std::function<rocksdb::Status (rocksdb::CompactionOutputs&, rocksdb::Status const&, rocksdb::Slice const&)> const&) ()
#16 0x0000aaaad620a468 in rocksdb::SubcompactionState::AddToOutput(rocksdb::CompactionIterator const&, std::function<rocksdb::Status (rocksdb::CompactionOutputs&)> const&, std::function<rocksdb::Status (rocksdb::CompactionOutputs&, rocksdb::Status const&, rocksdb::Slice const&)> const&) ()
#17 0x0000aaaad61e1584 in rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::SubcompactionState*) ()
#18 0x0000aaaad61e4558 in rocksdb::CompactionJob::Run() ()
#19 0x0000aaaad6278400 in rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) ()
#20 0x0000aaaad62799f8 in rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) ()
#21 0x0000aaaad627a3b0 in rocksdb::DBImpl::BGWorkCompaction(void*) ()
#22 0x0000aaaad665f178 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#23 0x0000aaaad665f36c in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#24 0x0000aaaad69052dc in execute_native_thread_routine ()
#25 0x0000ffffa7daee30 in start_thread (arg=0xffffe904c787) at ./nptl/pthread_create.c:442
#26 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 1 (Thread 0xffffa84c1020 (LWP 1) "kvrocks"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=44, futex_word=0xffff7ed36c90) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=44, futex_word=0xffff7ed36c90) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffff7ed36c90, expected=44, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7db0738 in __pthread_clockjoin_ex (threadid=281472809528256, thread_return=0x0, clockid=0, abstime=0x0, block=<optimized out>) at ./nptl/pthread_join_common.c:102
#4  0x0000aaaad6905380 in std::thread::join() ()
#5  0x0000aaaad614b4c8 in Status util::ThreadOperationImpl<&std::thread::join>(std::thread&, char const*) [clone .constprop.0] ()
#6  0x0000aaaad60a4c98 in Server::Join() ()
#7  0x0000aaaad5e3c8ac in main ()
[Inferior 1 (process 1) detached]

Are you willing to submit a PR?

• I'm willing to submit a PR!

[git-hulk]

The DB lock was used to protect the db pointer from being freed while using. All threads currently can guarantee that, so I think we should not use the db lock in Context construction and destruction. cc @PokIsemaine

Can refer to the stack:

Thread 25 (Thread 0xffff7e326bc0 (LWP 45) "compact-check"):
#0  0x0000ffffa7dab694 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xffffa79ad7a8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xffffa79ad7a8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffffa79ad7a8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000ffffa7dae1d0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa79ad380, cond=0xffffa79ad780) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0xffffa79ad780, mutex=0xffffa79ad380) at ./nptl/pthread_cond_wait.c:618
#5  0x0000aaaad6580b80 in rocksdb::port::CondVar::Wait() ()
#6  0x0000aaaad640d38c in rocksdb::InstrumentedCondVar::Wait() ()
#7  0x0000aaaad6273b8c in rocksdb::DBImpl::RunManualCompaction(rocksdb::ColumnFamilyData*, int, int, rocksdb::CompactRangeOptions const&, rocksdb::Slice const*, rocksdb::Slice const*, bool, bool, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int*) ()
#8  0x0000aaaad6274628 in rocksdb::DBImpl::CompactRangeInternal(rocksdb::CompactRangeOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const*, rocksdb::Slice const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#9  0x0000aaaad627519c in rocksdb::DBImpl::CompactRange(rocksdb::CompactRangeOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const*, rocksdb::Slice const*) ()
#10 0x0000aaaad60e4ccc in engine::Storage::Compact(rocksdb::ColumnFamilyHandle*, rocksdb::Slice const*, rocksdb::Slice const*) ()
#11 0x0000aaaad615f1b4 in CompactionChecker::PickCompactionFilesForCf(engine::ColumnFamilyConfig const&) [clone .isra.0] ()
#12 0x0000aaaad60bc730 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<Server::Start()::{lambda()#2}>(char const*, Server::Start()::{lambda()#2})::{lambda()#1}> > >::_M_run() [clone .lto_priv.0] ()
#13 0x0000aaaad69052dc in execute_native_thread_routine ()
#14 0x0000ffffa7daee30 in start_thread (arg=0xffffe904fbe7) at ./nptl/pthread_create.c:442
#15 0x0000ffffa7e17adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

[git-hulk]

Related PR: #2332

[PokIsemaine]

Was it permanently blocked, or was it just a long blocking time?

I added the two locks because there was data race on the storage during testing.

Since it has been quite a while, I have forgotten the exact location. If it's convenient for you, could you try removing them and running the tests with data race checks(TSAN)?

[git-hulk]

@PokIsemaine Sure, you can take a look at #2597.