[YSQL] Postgres crash seen when running CDC with nemesis testcase

[shamanthchandra-yb]

Jira Link: DB-6026

Description

Run report: Check JIRA

thread #1

* thread #1, name = 'postgres', stop reason = signal SIGSEGV
  * frame #0: 0x00007f8ab8aafc35 libc.so.6`__memmove_avx_unaligned at memcpy-avx-unaligned.S:90
    frame #1: 0x00007f8ab962c88f libc++.so.1`std::__1::basic_streambuf<char, std::__1::char_traits<char>>::xsputn(char const*, long) + 111
    frame #2: 0x00007f8ab4ca94d3 libyb_util.so`std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::__put_character_sequence<char, std::__1::char_traits<char>>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*, unsigned long) [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::sputn[abi:v15003](this=0x00007f8aa9e08a78, __s="", __n=<unavailable>) at streambuf:232:14
    frame #3: 0x00007f8ab4ca94ca libyb_util.so`std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::__put_character_sequence<char, std::__1::char_traits<char>>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*, unsigned long) [inlined] std::__1::ostreambuf_iterator<char, std::__1::char_traits<char>> std::__1::__pad_and_output<char, std::__1::char_traits<char>>(__s=(__sbuf_ = 0x00007f8aa9e08a78), __ob=<unavailable>, __op=<unavailable>, __oe="", __iob=<unavailable>, __fl=<unavailable>) at locale:1451:26
    frame #4: 0x00007f8ab4ca93c7 libyb_util.so`std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::__put_character_sequence<char, std::__1::char_traits<char>>(__os=0x00007f8aa9e08a70, __str=<unavailable>, __len=<unavailable>) at ostream:763:17
    frame #5: 0x00007f8ab4d718b0 libyb_util.so`yb::PrometheusWriter::FlushSingleEntry(std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, long) [inlined] std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::operator<<<char, std::__1::char_traits<char>, std::__1::allocator<char>>(__os=0x00007f8aa9e08a70, __str=Summary Unavailable) at ostream:1086:12
    frame #6: 0x00007f8ab4d71884 libyb_util.so`yb::PrometheusWriter::FlushSingleEntry(this=0x00007f8aa9e08848, attr=<unavailable>, name=<unavailable>, value=106) at metrics_writer.cc:59:40
    frame #7: 0x00007f8ab4d7449c libyb_util.so`yb::PrometheusWriter::WriteSingleEntry(this=0x00007f8aa9e08848, attr=0x00007f8ab6431ba8, name="handler_latency_yb_ysqlserver_SQLProcessor_SelectStmt_count", value=106, aggregation_function=kSum) at metrics_writer.cc:108:12
    frame #8: 0x00007f8ab63f9a81 libserver_process.so`yb::PgPrometheusMetricsHandler(req=<unavailable>, resp=<unavailable>) at pgsql_webserver_wrapper.cc:321:5
    frame #9: 0x00007f8ab6418c20 libserver_process.so`yb::Webserver::Impl::RunPathHandler(yb::Webserver::Impl::PathHandler const&, sq_connection*, sq_request_info*) [inlined] std::__1::__function::__value_func<void (yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*)>::operator(this=0x0000564f08846510, __args=0x00007f8aa9e0ab90, __args=0x00007f8aa9e0aa60)[abi:v15003](yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*&&) const at function.h:512:16
    frame #10: 0x00007f8ab6418c07 libserver_process.so`yb::Webserver::Impl::RunPathHandler(yb::Webserver::Impl::PathHandler const&, sq_connection*, sq_request_info*) [inlined] std::__1::function<void (yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*)>::operator(this= Function = yb::PgPrometheusMetricsHandler(yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*) , __arg=0x00007f8aa9e0ab90, __arg=0x00007f8aa9e08a60)(yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*) const at function.h:1197:12
    frame #11: 0x00007f8ab6418c07 libserver_process.so`yb::Webserver::Impl::RunPathHandler(this=0x0000564f08792600, handler=<unavailable>, connection=0x0000564f088a0000, request_info=<unavailable>) at webserver.cc:599:5
    frame #12: 0x00007f8ab64186bd libserver_process.so`yb::Webserver::Impl::BeginRequestCallback(this=0x0000564f08792600, connection=0x0000564f088a0000, request_info=0x0000564f088a0000) at webserver.cc:531:10
    frame #13: 0x00007f8ab6425178 libserver_process.so`process_new_connection + 4184
    frame #14: 0x00007f8ab6423e28 libserver_process.so`worker_thread + 232
    frame #15: 0x00007f8ab9334694 libpthread.so.0`start_thread(arg=0x00007f8aa9e1a700) at pthread_create.c:333
    frame #16: 0x00007f8ab8a7141d libc.so.6`__clone at clone.S:109

thread #2

  thread #2, stop reason = signal 0
    frame #0: 0x00007f8ab8a685cd libc.so.6`poll at syscall-template.S:84
    frame #1: 0x00007f8ab64239cc libserver_process.so`master_thread + 572
    frame #2: 0x00007f8ab9334694 libpthread.so.0`start_thread(arg=0x00007f8aaa61b700) at pthread_create.c:333
    frame #3: 0x00007f8ab8a7141d libc.so.6`__clone at clone.S:109

thread #3

  thread #3, stop reason = signal 0
    frame #0: 0x00007f8ab5ebe9b6 libyb_docdb_encoding.so`__do_fini + 22
    frame #1: 0x00007f8aba8c432a
    frame #2: 0x00007f8ab89c0969 libc.so.6`__run_exit_handlers(status=<unavailable>, listp=0x00007f8ab8d215c0, run_list_atexit=true) at exit.c:82
    frame #3: 0x00007f8ab89c09b5 libc.so.6`__GI_exit(status=<unavailable>) at exit.c:104
    frame #4: 0x0000564f046797c2 postgres`proc_exit(code=<unavailable>) at ipc.c:157:2
    frame #5: 0x0000564f0485ccbf postgres`errfinish(dummy=<unavailable>) at elog.c:801:3
    frame #6: 0x0000564f045d99dd postgres`bgworker_die(postgres_signal_arg=<unavailable>) at bgworker.c:672:2
    frame #7: 0x00007f8ab933cba0 libpthread.so.0`__restore_rt
    frame #8: 0x00007f8ab8a719f3 libc.so.6`epoll_wait at syscall-template.S:84
    frame #9: 0x0000564f0467b7e5 postgres`WaitEventSetWait [inlined] WaitEventSetWaitBlock(set=0x0000564f08833e88, cur_timeout=-1, occurred_events=0x00007ffe83256498, nevents=1) at latch.c:1062:7
    frame #10: 0x0000564f0467b7d7 postgres`WaitEventSetWait(set=0x0000564f08833e88, timeout=-1, occurred_events=<unavailable>, nevents=1, wait_event_info=<unavailable>) at latch.c:1014:8
    frame #11: 0x0000564f0467b263 postgres`WaitLatchOrSocket(latch=0x00007f8ab39ab88c, wakeEvents=<unavailable>, sock=-1, timeout=<unavailable>, wait_event_info=117440512) at latch.c:399:7
    frame #12: 0x0000564f0467b160 postgres`WaitLatch(latch=<unavailable>, wakeEvents=<unavailable>, timeout=<unavailable>, wait_event_info=<unavailable>) at latch.c:353:9 [artificial]
    frame #13: 0x00007f8ab40d86a0 yb_pg_metrics.so`webserver_worker_main(unused=<unavailable>) at yb_pg_metrics.c:375:3
    frame #14: 0x0000564f045d986c postgres`StartBackgroundWorker at bgworker.c:841:2
    frame #15: 0x0000564f045f25ac postgres`maybe_start_bgworkers [inlined] do_start_bgworker(rw=0x0000564f08866000) at postmaster.c:6021:4
    frame #16: 0x0000564f045f254f postgres`maybe_start_bgworkers at postmaster.c:6247:9
    frame #17: 0x0000564f045eef00 postgres`PostmasterMain(argc=23, argv=0x0000564f087403c0) at postmaster.c:1429:2
    frame #18: 0x0000564f044f4a4f postgres`PostgresServerProcessMain(argc=23, argv=0x0000564f087403c0) at main.c:234:3
    frame #19: 0x0000564f041be702 postgres`main + 34
    frame #20: 0x00007f8ab89ab825 libc.so.6`__libc_start_main(main=(postgres`main), argc=23, argv=0x00007ffe83256dd8, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007ffe83256dc8) at libc-start.c:289
    frame #21: 0x0000564f041be619 postgres`_start at start.S:108

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[sushantrmishra]

Possible duplicate of #18948 . We should reverify once fix for #18948 has landed.