postgres process crashes when running initdb for GCC 9 release build on CentOS 7

[mbautin]

Core was generated by `~/code/yugabyte-db/build/release-gcc9-dynamic-ninja/pos'.
Program terminated with signal 6, Aborted.
#0  0x00007fe7104d1387 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-324.el7_9.x86_64 libatomic-4.8.5-44.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 nss-softokn-freebl-3.53.1-6.el7_9.x86_64
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) where
#0  0x00007fe7104d1387 in raise () from /lib64/libc.so.6
#1  0x00007fe7104d2a78 in abort () from /lib64/libc.so.6
#2  0x00007fe70f8b2fac in tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) [clone .cold] () from /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210520175418-8aebe87f79-centos7-gcc9/installed/uninstrumented/lib/libtcmalloc.so.4
#3  0x00007fe70f8b4799 in (anonymous namespace)::InvalidFree(void*) () from /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210520175418-8aebe87f79-centos7-gcc9/installed/uninstrumented/lib/libtcmalloc.so.4
#4  0x00007fe70e8689f3 in ~percpu_rwlock (this=0x2b166e0, __in_chrg=<optimized out>) at ../../src/yb/util/rw_semaphore.h:73
#5  yb::rpc::Messenger::~Messenger (this=0x2b16680, __in_chrg=<optimized out>) at ../../src/yb/rpc/messenger.cc:577
#6  0x00007fe70e868d42 in yb::rpc::Messenger::~Messenger (this=0x2b16680, __in_chrg=<optimized out>) at ../../src/yb/rpc/messenger.cc:577
#7  0x00007fe7122901ca in operator() (this=0x2b2eb48, __ptr=<optimized out>) at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/unique_ptr.h:75
#8  ~unique_ptr (this=0x2b2eb48, __in_chrg=<optimized out>) at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/unique_ptr.h:292
#9  ~MessengerHolder (this=0x2b2eb40, __in_chrg=<optimized out>) at ../../src/yb/yql/pggate/pggate.h:488
#10 yb::pggate::PgApiImpl::~PgApiImpl (this=0x2b2e9c0, __in_chrg=<optimized out>) at ../../src/yb/yql/pggate/pggate.cc:221
#11 0x00007fe712290422 in yb::pggate::PgApiImpl::~PgApiImpl (this=0x2b2e9c0, __in_chrg=<optimized out>) at ../../src/yb/yql/pggate/pggate.cc:221
#12 0x00007fe712289c5c in YBCDestroyPgGate () at ../../src/yb/yql/pggate/ybc_pggate.cc:109
#13 0x000000000082b6e7 in proc_exit (code=code@entry=0) at ../../../../../../../src/postgres/src/backend/storage/ipc/ipc.c:152
#14 0x0000000000549a99 in BootstrapModeMain () at ../../../../../../src/postgres/src/backend/bootstrap/bootstrap.c:551
#15 AuxiliaryProcessMain () at ../../../../../../src/postgres/src/backend/bootstrap/bootstrap.c:442
#16 0x0000000000713a3b in PostgresServerProcessMain (argc=6, argv=0x2a935b0) at ../../../../../../src/postgres/src/backend/main/main.c:226
#17 0x0000000000713ab2 in main ()
(gdb) 

[mbautin]

The above crash fails with gperftools 2.7.0. When disabling tcmalloc, the crash does not occur. Also, previously, I've already seen this error and found that it was fixed by upgrading gperftools 2.8.0, but then we had to downgrade gperftools to 2.7.0 due to gperftools/gperftools#1204 and gperftools/gperftools#1227. Perhaps it makes sense to upgrade to gperftools 2.8.1 now.

[mbautin]

This could be related to the postgres process not linking libtcmalloc directly, so some memory allocations might be happening before tcmalloc is initialized.

Here is a potentially relevant issue: gperftools/gperftools#1066