Node gets stuck in a loop while shutting down

[evan-forbes]

When attempting to shut down my node while not producing blocks, it gets stuck in a loop and the process have to be forcefully stopped.

10:03PM INF stopping service impl=MConn{51.178.133.224:43798} module=p2p peer={"Hostname":"51.178.133.224","NodeID":"322542cec82814d8903de2259b1d4d97026bcb75","Path":"","Port":43798,"Protocol":"mconn"} service=MConnection
10:03PM INF Ignoring inbound connection: error while adding peer err="duplicate ID<322542cec82814d8903de2259b1d4d97026bcb75>" id=322542cec82814d8903de2259b1d4d97026bcb75 module=p2p
10:03PM INF starting service impl=MConn{88.99.249.251:53706} module=p2p peer={"Hostname":"88.99.249.251","NodeID":"ec072065bd4c6126a5833c97c8eb2d4382db85be","Path":"","Port":53706,"Protocol":"mconn"} service=MConnection
10:03PM INF stopping service impl=MConn{88.99.249.251:53706} module=p2p peer={"Hostname":"88.99.249.251","NodeID":"ec072065bd4c6126a5833c97c8eb2d4382db85be","Path":"","Port":53706,"Protocol":"mconn"} service=MConnection
10:03PM INF Ignoring inbound connection: error while adding peer err="duplicate ID<ec072065bd4c6126a5833c97c8eb2d4382db85be>" id=ec072065bd4c6126a5833c97c8eb2d4382db85be module=p2p
10:03PM INF starting service impl=MConn{51.178.133.224:43832} module=p2p peer={"Hostname":"51.178.133.224","NodeID":"322542cec82814d8903de2259b1d4d97026bcb75","Path":"","Port":43832,"Protocol":"mconn"} service=MConnection
10:03PM INF stopping service impl=MConn{51.178.133.224:43832} module=p2p peer={"Hostname":"51.178.133.224","NodeID":"322542cec82814d8903de2259b1d4d97026bcb75","Path":"","Port":43832,"Protocol":"mconn"} service=MConnection
10:03PM INF Ignoring inbound connection: error while adding peer err="duplicate ID<322542cec82814d8903de2259b1d4d97026bcb75>" id=322542cec82814d8903de2259b1d4d97026bcb75 module=p2p

[fkbenjamin]

I just encountered the same behaviour on our validator:

May 28 00:09:46 celestia-val01 celestia-appd[286985]: 12:09AM INF stopping service impl=MConn{5.161.116.198:56726} module=p2p peer={"Hostname":"5.161.116.198","NodeID":"11c2351aece63b26e4bf0ea28463e08ff26c0869","Path":"","Port":56726,"Protocol":"mconn"} service=MConnection
May 28 00:09:46 celestia-val01 celestia-appd[286985]: 12:09AM INF Inbound Peer rejected err="filtered CONN<5.161.116.198:56726>: duplicate CONN<5.161.116.198:56726>" module=p2p numPeers=148
May 28 00:09:46 celestia-val01 celestia-appd[286985]: 12:09AM INF starting service impl=MConn{51.178.133.224:42232} module=p2p peer={"Hostname":"51.178.133.224","NodeID":"322542cec82814d8903de2259b1d4d97026bcb75","Path":"","Port":42232,"Protocol":"mconn"} service=MConnection
May 28 00:09:46 celestia-val01 celestia-appd[286985]: 12:09AM INF stopping service impl=MConn{51.178.133.224:42232} module=p2p peer={"Hostname":"51.178.133.224","NodeID":"322542cec82814d8903de2259b1d4d97026bcb75","Path":"","Port":42232,"Protocol":"mconn"} service=MConnection
May 28 00:09:46 celestia-val01 celestia-appd[286985]: 12:09AM INF Ignoring inbound connection: error while adding peer err="duplicate ID<322542cec82814d8903de2259b1d4d97026bcb75>" id=322542cec82814d8903de2259b1d4d97026bcb75 module=p2p
May 28 00:09:46 celestia-val01 systemd[1]: celestia-appd.service: State 'stop-sigterm' timed out. Killing.

[evan-forbes]

this was fixed in tendermint/tendermint#8651 and should be fixed after we upgrade.

@fkbenjamin if you would like to test, please use the latest commit of the evan/profile-constantly branch

Let's wait to close this until we merge an upgrade, however