Test failure ICountDownLatchTest.testLatch

[ihsandemir]

ICountDownLatchTest.testLatch test sometimes fails for build cpp-linux-nightly-64-STATIC-Release.
https://hazelcast-l337.ci.cloudbees.com/job/cpp-linux-nightly-64-STATIC-Release/64/console

The test causes segmentation fault. When examined the core stack trace, I noticed that the test terminates before the thread finishes. The thread finishes when the await request gets the response from the server. One scenario in mind:

1. The latch is decremented 20th time at the server side becoming 0.
2. The response to await comes to the client before the response to countdown is received.

Hence, this may cause the test termination where the latch pointer becomes invalid memory and the latch thread accessing the invalid memory crashes.

Stack trace:
Program terminated with signal 11, Segmentation fault.
#0 0x000000000065ef70 in hazelcast::util::LockGuard::~LockGuard() ()

(gdb) info threads
7 Thread 0x2b3c94137250 (LWP 24821) 0x0000003388807c65 in pthread_join () from /lib64/libpthread.so.0
6 Thread 28071 0x0000003387cce172 in select () from /lib64/libc.so.6
5 Thread 28072 0x0000003387cce172 in select () from /lib64/libc.so.6
4 Thread 28073 0x000000338880b280 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 28074 0x000000338880de11 in recv () from /lib64/libpthread.so.0
2 Thread 28079 0x000000338880b280 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
1 Thread 28082 0x000000000065ef70 in hazelcast::util::LockGuard::~LockGuard() ()
(gdb) bt
#0 0x000000000065ef70 in hazelcast::util::LockGuard::~LockGuard() ()
#1 0x00000000006d0b9e in hazelcast::client::connection::CallFuture::get(long) ()
#2 0x00000000006d1115 in hazelcast::client::connection::CallFuture::get() ()
#3 0x0000000000628afe in hazelcast::client::proxy::ProxyImpl::invoke(std::auto_ptrhazelcast::client::protocol::ClientMessage, int) ()
#4 0x000000000061e78b in hazelcast::client::ICountDownLatch::countDown() ()
#5 0x0000000000527338 in hazelcast::client::test::testLatchThread(hazelcast::util::ThreadArgs&) ()
#6 0x000000000065ea93 in hazelcast::util::Thread::controlledThread(void*) ()
#7 0x000000338880683d in start_thread () from /lib64/libpthread.so.0
#8 0x0000003387cd4fdd in clone () from /lib64/libc.so.6

(gdb) thread 2
[Switching to thread 2 (Thread 28079)]#0 0x000000338880b280 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x000000338880b280 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000065b993 in hazelcast::util::ConditionVariable::waitFor(hazelcast::util::Mutex&, long) ()
#2 0x000000000065de85 in hazelcast::util::Thread::interruptibleSleep(int) ()
#3 0x00000000006ba30e in hazelcast::client::spi::PartitionService::runListener(hazelcast::util::Thread*) ()
#4 0x000000000065ea93 in hazelcast::util::Thread::controlledThread(void*) ()
#5 0x000000338880683d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003387cd4fdd in clone () from /lib64/libc.so.6

(gdb) thread 3
[Switching to thread 3 (Thread 28074)]#0 0x000000338880de11 in recv () from /lib64/libpthread.so.0
(gdb) bt
#0 0x000000338880de11 in recv () from /lib64/libpthread.so.0
#1 0x000000000065675e in hazelcast::client::Socket::receive(void*, int, int) const ()
#2 0x000000000065fd9c in hazelcast::util::ByteBuffer::readFrom(hazelcast::client::Socket const&, int, int) ()
#3 0x00000000006cc697 in hazelcast::client::connection::Connection::readBlocking() ()
#4 0x0000000000617f6e in hazelcast::client::connection::ClusterListenerThread::listenMembershipEvents() ()
#5 0x000000000061848c in hazelcast::client::connection::ClusterListenerThread::run(hazelcast::util::Thread*) ()
#6 0x000000000065ea93 in hazelcast::util::Thread::controlledThread(void*) ()
#7 0x000000338880683d in start_thread () from /lib64/libpthread.so.0
#8 0x0000003387cd4fdd in clone () from /lib64/libc.so.6

(gdb) thread 4
[Switching to thread 4 (Thread 28073)]#0 0x000000338880b280 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x000000338880b280 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000065b993 in hazelcast::util::ConditionVariable::waitFor(hazelcast::util::Mutex&, long) ()
#2 0x000000000065de85 in hazelcast::util::Thread::interruptibleSleep(int) ()
#3 0x0000000000613356 in hazelcast::client::connection::HeartBeater::run(hazelcast::util::Thread*) ()
#4 0x000000000065ea93 in hazelcast::util::Thread::controlledThread(void*) ()
#5 0x000000338880683d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003387cd4fdd in clone () from /lib64/libc.so.6

(gdb) thread 5
[Switching to thread 5 (Thread 28072)]#0 0x0000003387cce172 in select () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003387cce172 in select () from /lib64/libc.so.6
#1 0x00000000006161ce in hazelcast::client::connection::OutSelector::listenInternal() ()
#2 0x0000000000614330 in hazelcast::client::connection::IOSelector::listen() ()
#3 0x000000000065ea93 in hazelcast::util::Thread::controlledThread(void*) ()
#4 0x000000338880683d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003387cd4fdd in clone () from /lib64/libc.so.6

(gdb) thread 6
[Switching to thread 6 (Thread 28071)]#0 0x0000003387cce172 in select () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003387cce172 in select () from /lib64/libc.so.6
#1 0x0000000000616a57 in hazelcast::client::connection::InSelector::listenInternal() ()
#2 0x0000000000614330 in hazelcast::client::connection::IOSelector::listen() ()
#3 0x000000000065ea93 in hazelcast::util::Thread::controlledThread(void*) ()
#4 0x000000338880683d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003387cd4fdd in clone () from /lib64/libc.so.6

(gdb) thread 7
[Switching to thread 7 (Thread 0x2b3c94137250 (LWP 24821))]#0 0x0000003388807c65 in pthread_join () from /lib64/libpthread.so.0
(gdb) bt
#0 0x0000003388807c65 in pthread_join () from /lib64/libpthread.so.0
#1 0x000000000065e21e in hazelcast::util::Thread::~Thread() ()
#2 0x0000000000529ac1 in hazelcast::client::test::ICountDownLatchTest_testLatch_Test::TestBody() ()
#3 0x00000000006fc664 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const*) ()
#4 0x0000000000704906 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const*) ()
#5 0x00000000006f23fb in testing::Test::Run() ()
#6 0x00000000006f26e6 in testing::TestInfo::Run() ()
#7 0x00000000006f2807 in testing::TestCase::Run() ()
#8 0x00000000006f510d in testing::internal::UnitTestImpl::RunAllTests() ()
#9 0x00000000006fc6f4 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl, bool (testing::internal::UnitTestImpl::)(), char const*) ()
#10 0x00000000007043a9 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl, bool (testing::internal::UnitTestImpl::_)(), char const*) ()
#11 0x00000000006f0fe7 in testing::UnitTest::Run() ()
#12 0x000000000052ac7a in main ()

(gdb)

[ihsandemir]

We do not see this failure anymore by the way.

The issue #339 fix may be a reason.

Closing the issue since it is not observed any more in the nightly builds.