[DocDB] yb-master SIGSEGV in LookupTabletPeer during UpdateTransaction

[shamanthchandra-yb]

Jira Link: DB-9718

Description

Please find stress run report in JIRA description.

In one of the CDC testcase runs on version 2.16.9.0-b67, a master core dump was observed.

This appears to be a very isolated incident and not easily reproducible. I have run the same test case multiple times on the same build without encountering the issue. Also, this is a regression case running from some time across branches, and and this is the first occurrence of such an issue.

Here is the backtrace:

* thread #1, name = 'yb-master', stop reason = signal SIGSEGV
  * frame #0: 0x000055ff0528c56e yb-master`yb::tserver::LookupTabletPeer(yb::tserver::TabletPeerLookupIf*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [inlined] yb::tablet::RaftGroupStatePB std::__1::__cxx_atomic_load[abi:v15002]<yb::tablet::RaftGroupStatePB>(__a=0x00000000000000b0, __order=acquire) at atomic:958:12
    frame #1: 0x000055ff0528c56e yb-master`yb::tserver::LookupTabletPeer(yb::tserver::TabletPeerLookupIf*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [inlined] std::__1::__atomic_base<yb::tablet::RaftGroupStatePB, false>::load[abi:v15002](this=0x00000000000000b0, __m=acquire) const at atomic:1560:17
    frame #2: 0x000055ff0528c56e yb-master`yb::tserver::LookupTabletPeer(yb::tserver::TabletPeerLookupIf*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [inlined] yb::tablet::TabletPeer::state(this=0x0000000000000000) const at tablet_peer.h:290:19
    frame #3: 0x000055ff0528c56e yb-master`yb::tserver::LookupTabletPeer(yb::tserver::TabletPeerLookupIf*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [inlined] yb::Result<yb::tserver::TabletPeerTablet> yb::tserver::(anonymous namespace)::DoLookupTabletPeer<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(tablet_manager=<unavailable>, tablet_id="00000000000000000000000000000000") at service_util.cc:256:56
    frame #4: 0x000055ff0528c547 yb-master`yb::tserver::LookupTabletPeer(tablet_manager=<unavailable>, tablet_id="00000000000000000000000000000000") at service_util.cc:278:10
    frame #5: 0x000055ff05298887 yb-master`yb::tserver::TabletServiceImpl::UpdateTransaction(yb::tserver::UpdateTransactionRequestPB const*, yb::tserver::UpdateTransactionResponsePB*, yb::rpc::RpcContext) [inlined] yb::Result<yb::tserver::TabletPeerTablet> yb::tserver::LookupTabletPeerOrRespond<yb::tserver::UpdateTransactionResponsePB, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(tablet_manager=<unavailable>, tablet_id=<unavailable>, resp=0x000055ff198b4758, context=0x00007f80b61b3040) at service_util.h:156:37
    frame #6: 0x000055ff0529887f yb-master`yb::tserver::TabletServiceImpl::UpdateTransaction(this=0x000055ff19b5d000, req=0x000055ff198b4720, resp=0x000055ff198b4758, context=RpcContext @ 0x00007f80b61b3040) at tablet_service.cc:1073:24
    frame #7: 0x000055ff053216f5 yb-master`std::__1::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::__1::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] yb::tserver::TabletServerServiceIf::InitMethods(this=<unavailable>, req=<unavailable>, resp=<unavailable>, rpc_context=RpcContext @ 0x00007f80b61b2fe0)::$_9::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::tserver::UpdateTransactionRequestPB const*, yb::tserver::UpdateTransactionResponsePB*, yb::rpc::RpcContext)::operator()(yb::tserver::UpdateTransactionRequestPB const*, yb::tserver::UpdateTransactionResponsePB*, yb::rpc::RpcContext) const at tserver_service.service.cc:683:9
    frame #8: 0x000055ff053216bd yb-master`std::__1::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::__1::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) at local_call.h:127:7
    frame #9: 0x000055ff05321312 yb-master`std::__1::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::__1::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] yb::tserver::TabletServerServiceIf::InitMethods(this=<unavailable>, call=nullptr)::$_9::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const at tserver_service.service.cc:681:7
    frame #10: 0x000055ff05321295 yb-master`std::__1::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::__1::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] decltype(__f=<unavailable>, __args=<unavailable>)::$_9&>()(std::declval<std::__1::shared_ptr<yb::rpc::InboundCall>>())) std::__1::__invoke[abi:v15002]<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9&, std::__1::shared_ptr<yb::rpc::InboundCall>>(yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9&, std::__1::shared_ptr<yb::rpc::InboundCall>&&) at invoke.h:394:23
    frame #11: 0x000055ff05321274 yb-master`std::__1::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::__1::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] void std::__1::__invoke_void_return_wrapper<void, true>::__call<yb::tserver::TabletServerServiceIf::InitMethods(__args=<unavailable>, __args=<unavailable>)::$_9&, std::__1::shared_ptr<yb::rpc::InboundCall>>(yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9&, std::__1::shared_ptr<yb::rpc::InboundCall>&&) at invoke.h:479:9
    frame #12: 0x000055ff05321274 yb-master`std::__1::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::__1::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] std::__1::__function::__alloc_func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::__1::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator(this=<unavailable>, __arg=<unavailable>)[abi:v15002](std::__1::shared_ptr<yb::rpc::InboundCall>&&) at function.h:185:16
    frame #13: 0x000055ff05321274 yb-master`std::__1::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::__1::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator(this=<unavailable>, __arg=<unavailable>)(std::__1::shared_ptr<yb::rpc::InboundCall>&&) at function.h:359:12
    frame #14: 0x000055ff0532793f yb-master`yb::tserver::TabletServerServiceIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) [inlined] std::__1::__function::__value_func<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator(this=<unavailable>, __args=nullptr)[abi:v15002](std::__1::shared_ptr<yb::rpc::InboundCall>&&) const at function.h:512:16
    frame #15: 0x000055ff05327920 yb-master`yb::tserver::TabletServerServiceIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) [inlined] std::__1::function<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator(this=<unavailable>, __arg=nullptr)(std::__1::shared_ptr<yb::rpc::InboundCall>) const at function.h:1187:12
    frame #16: 0x000055ff05327920 yb-master`yb::tserver::TabletServerServiceIf::Handle(this=<unavailable>, call=<unavailable>) at tserver_service.service.cc:531:3
    frame #17: 0x000055ff0500186a yb-master`yb::rpc::ServicePoolImpl::Handle(this=0x000055ff1a01c240, incoming=nullptr) at service_pool.cc:270:19
    frame #18: 0x000055ff04f2b7ef yb-master`yb::rpc::InboundCall::InboundCallTask::Run(this=<unavailable>) at inbound_call.cc:253:13
    frame #19: 0x000055ff050104c3 yb-master`yb::rpc::(anonymous namespace)::Worker::Execute(this=0x000055ff198bc460) at thread_pool.cc:104:15
    frame #20: 0x000055ff054bc42c yb-master`yb::Thread::SuperviseThread(void*) [inlined] std::__1::__function::__value_func<void ()>::operator(this=0x000055ff19caa4d0)[abi:v15002]() const at function.h:512:16
    frame #21: 0x000055ff054bc419 yb-master`yb::Thread::SuperviseThread(void*) [inlined] std::__1::function<void ()>::operator(this=0x000055ff19caa4d0)() const at function.h:1187:12
    frame #22: 0x000055ff054bc419 yb-master`yb::Thread::SuperviseThread(arg=0x000055ff19caa480) at thread.cc:811:3
    frame #23: 0x00007f80c0bf2694 libpthread.so.0`start_thread(arg=0x00007f80b61bb700) at pthread_create.c:333
    frame #24: 0x00007f80c10f441d libc.so.6`__clone at clone.S:109

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[rthallamko3]

We suspect its a fallout of #21877 ? Lets create a new one if this happens again.

[shamanthchandra-yb]

@shishir2001-yb encountered this issue during a test run and has raised it in fresh GH: #23839. Looping this in for reference to the number of occurrences.