Not able to scale down a docker cluster gracefully

[schoudhury]

Hitting an assert on scaling down a docker container based cluster.

• Create a new docker local cluster with 3 nodes
• Run the load tester against it
• Add a new node using

./yb-docker-ctl add_node

• Now remove a node

./yb-docker-ctl remove_node 4

All the nodes go down. The error from any one container:

F0321 18:27:46.960845   111 net_util.cc:203] Check failed: _s.ok() Bad status: Network error (yb/util/net/net_util.cc:195): Unable to resolve address yb-tserver-n4, getaddrinfo returned -2 (EAI_NONAME): Name or service not known
Fatal failure details written to /mnt/disk0/yb-data/tserver/logs/yb-tserver.FATAL.details.2018-03-21T18_27_46.pid1.txt
*** Check failure stack trace: ***
    @     0x7f4549f388eb  DumpStackTraceAndExit (yb/util/logging.cc:154)
    @     0x7f454910a0cc 
    @     0x7f454910bfdc 
    @     0x7f4549109c29 
    @     0x7f454910ca9e 
    @     0x7f4549f68249  ResolveInetAddresses<yb::HostPort::ResolveAddresses(std::vector<boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> >*) const::<lambda(sockaddr_in*)> > (yb/util/net/net_util.cc:203)
    @     0x7f4549f68249  yb::HostPort::ResolveAddresses(std::vector<boost::asio::ip::basic_endpoint<boost::asio::ip::tcp>, std::allocator<boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> > >*) const (yb/util/net/net_util.cc:226)
    @     0x7f454ebbcac8  CreateConsensusServiceProxyForHost (yb/consensus/consensus_peers.cc:457)
    @     0x7f454ebbcd74  yb::consensus::RpcPeerProxyFactory::NewProxy(yb::consensus::RaftPeerPB const&, gscoped_ptr<yb::consensus::PeerProxy, yb::DefaultDeleter<yb::consensus::PeerProxy> >*) (yb/consensus/consensus_peers.cc:477)
    @     0x7f454ebd4787  yb::consensus::LeaderElection::LeaderElection(yb::consensus::RaftConfigPB const&, yb::consensus::PeerProxyFactory*, yb::consensus::VoteRequestPB const&, std::unique_ptr<yb::consensus::VoteCounter, std::default_delete<yb::consensus::VoteCounter> >, yb::MonoDelta, yb::StronglyTypedBool<yb::consensus::TEST_SuppressVoteRequest_Tag>, yb::Callback<void (yb::consensus::ElectionResult const&)>) (yb/consensus/leader_election.cc:201)
    @     0x7f454ebf48a3  yb::consensus::RaftConsensus::DoStartElection(yb::consensus::Consensus::ElectionMode, bool, yb::OpIdPB const&, std::string const&, yb::StronglyTypedBool<yb::consensus::TEST_SuppressVoteRequest_Tag>) (yb/consensus/raft_consensus.cc:534)
    @     0x7f454ebe3f8b  yb::consensus::Consensus::StartElection(yb::consensus::Consensus::ElectionMode, bool, yb::OpIdPB const&, std::string const&, yb::StronglyTypedBool<yb::consensus::TEST_SuppressVoteRequest_Tag>) (yb/consensus/consensus.h:198)
    @     0x7f454ebe3f8b  yb::consensus::RaftConsensus::ReportFailureDetected(std::string const&, yb::Status const&) (yb/consensus/raft_consensus.cc:782)
    @     0x7f4549f245be  yb::Callback<void (std::string const&, yb::Status const&)>::Run(std::string const&, yb::Status const&) const (yb/gutil/callback.h:494)
    @     0x7f4549f245be  yb::TimedFailureDetector::CheckForFailures(yb::MonoTime const&) (yb/util/failure_detector.cc:131)
    @     0x7f4549f2302c  yb::RandomizedFailureMonitor::RunThread() (yb/util/failure_detector.cc:225)
    @     0x7f4549f970d5  std::function<void ()>::operator()() const (gcc/5.5.0/include/c++/5.5.0/functional:2267)
    @     0x7f4549f970d5  yb::Thread::SuperviseThread(void*) (yb/util/thread.cc:602)
    @     0x7f4546c7d693  start_thread (/tmp/glibc-20180109-1462-njgus6/glibc-2.23/nptl/pthread_create.c:333)
    @     0x7f45469bf3cc  (unknown) (../sysdeps/unix/sysv/linux/x86_64/clone.S:109)
    @ 0xffffffffffffffff 
    @     0x7f454ebd4788 yb::consensus::LeaderElection::LeaderElection()
    @     0x7f454ebf48a4 yb::consensus::RaftConsensus::DoStartElection()
    @     0x7f454ebe3f8c yb::consensus::RaftConsensus::ReportFailureDetected()
    @     0x7f4549f245bf yb::TimedFailureDetector::CheckForFailures()
    @     0x7f4549f2302d yb::RandomizedFailureMonitor::RunThread()
    @     0x7f4549f970d6 yb::Thread::SuperviseThread()
    @     0x7f4546c7d694 start_thread
    @     0x7f45469bf3cd __clone
    @                0x0 (unknown)

Version:

[root@8f69d3501259 yugabyte]# ./bin/yb-tserver --version
version 0.9.7.0 build 9 build_type RELEASE

[rkarthik007]

@ttyusupov - is this related to your recent fix? Do we just need to publish a new container?

[ttyusupov]

Yes, looks like #97 which has been already fixed.