Peers disconnect frequently despite running default software #1368

evan-forbes · 2024-06-03T07:27:09Z

There's been an issue for as long as I can remember w/ Comet where peers often disconnect from each other despite all peers operating using unmodified software. For example, in testground we see a large number of peer disconnections despite all nodes in the experiment running stock software. Its worth noting that the default timeouts are being used (45 seconds for receiving a pong)

Here are all of the different errors that resulted in a peer getting disconnected from a single testground experiment

reason send packet msg: write tcp 10.44.240.2:26656->10.33.224.1:53580: write: connection reset by peer reason send packet msg: write tcp 10.44.240.2:26656->10.32.192.3:50992: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.44.240.2:51992->10.36.240.3:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason recv routine: unexpected EOF reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.41.192.2:50130->10.39.0.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.42.176.2:37776->10.32.224.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.46.0.2:57696->10.34.64.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.0.2:50328->10.47.136.3:26656: use of closed network connection reason send packet msg: write tcp 10.46.0.2:57776->10.42.176.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.46.0.2:53824->10.45.72.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.0.2:47756->10.43.64.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.0.2:37012->10.39.160.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.0.2:59820->10.34.64.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.46.0.2:26656->10.47.176.3:57254: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.36.112.3:26656->10.37.208.2:36530: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.36.112.3:53644->10.36.160.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:33182->10.47.232.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.45.72.2:34296: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.36.112.3:33104->10.45.32.0:26656: use of closed network connection reason send packet msg: write tcp 10.36.112.3:42448->10.43.32.0:26656: use of closed network connection reason send packet msg: write tcp 10.36.112.3:38758->10.34.160.0:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:36818->10.35.96.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:38824->10.46.64.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.36.240.3:51578: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:50936->10.34.64.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.41.128.2:53860: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:39604->10.32.192.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:36534->10.38.64.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:51350->10.46.0.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.43.64.2:59764: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.32.0.4:60026: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.38.224.1:60170: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:47088->10.37.112.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.36.112.3:26656->10.43.48.2:48816: use of closed network connection reason switch: reason send packet msg: write tcp 10.36.112.3:26656->10.44.240.2:57366: use of closed network connection reason switch: reason send packet msg: write tcp 10.36.112.3:26656->10.36.160.2:36728: use of closed network connection reason switch: reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.42.16.3:47964: write: connection reset by peer reason send packet msg: write tcp 10.35.224.2:26656->10.38.64.2:33562: write: broken pipe reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.45.16.1:47522: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.41.192.2:37518: write: connection reset by peer reason send packet msg: write tcp 10.35.224.2:26656->10.34.64.1:46558: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.46.64.2:39144: write: connection reset by peer reason recv routine: unexpected EOF reason send packet msg: write tcp 10.35.224.2:59816->10.33.224.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.42.208.2:58870: write: connection reset by peer reason send packet msg: write tcp 10.35.224.2:26656->10.45.32.0:37068: use of closed network connection reason send packet msg: write tcp 10.35.224.2:34722->10.37.160.1:26656: use of closed network connection reason send packet msg: write tcp 10.35.224.2:26656->10.38.224.1:56240: use of closed network connection reason send packet msg: write tcp 10.35.224.2:54492->10.43.32.0:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.39.0.2:40520: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:38492->10.32.96.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:42964->10.47.176.3:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:35110->10.40.208.2:26656: use of closed network connection reason switch: reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.37.112.2:55134: use of closed network connection reason send packet msg: write tcp 10.35.224.2:42602->10.35.96.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.46.0.2:46380: use of closed network connection reason switch: reason switch: reason send packet msg: write tcp 10.35.224.2:58130->10.44.240.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:42052->10.34.64.1:26656: use of closed network connection reason send packet msg: write tcp 10.35.224.2:26656->10.45.0.2:50006: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.47.192.3:47734: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:37478->10.38.64.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:47272->10.37.32.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.34.160.0:43626: use of closed network connection reason switch: reason send packet msg: write tcp 10.33.224.1:52514->10.47.232.2:26656: write: connection reset by peer reason recv routine: unexpected EOF reason send packet msg: write tcp 10.33.224.1:47346->10.38.64.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.33.224.1:54898->10.39.0.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.33.224.1:26656->10.33.96.2:60568: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.33.224.1:45160->10.32.224.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason recv routine: unexpected EOF reason send packet msg: write tcp 10.41.80.3:39498->10.32.224.1:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.41.80.3:26656->10.45.16.1:49578: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason recv routine: unexpected EOF reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.33.96.2:47350->10.46.64.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.33.96.2:49634->10.41.192.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.33.96.2:26656->10.39.160.1:42156: write: connection reset by peer reason send packet msg: write tcp 10.33.96.2:26656->10.46.0.2:49232: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.37.208.2:26656->10.43.64.2:55936: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.37.208.2:41988->10.32.192.3:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.37.208.2:43280->10.42.176.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.37.208.2:26656->10.47.136.3:40000: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.34.64.1:53704->10.39.96.3:26656: write: connection reset by peer reason send routine: pong timeout reason recv routine: unexpected EOF reason send routine: pong timeout reason send packet msg: write tcp 10.34.64.1:26656->10.36.240.3:35330: write: connection reset by peer reason send packet msg: write tcp 10.34.64.1:56166->10.37.224.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.34.64.1:26656->10.47.136.3:42874: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.34.64.1:26656->10.41.192.2:37630: write: connection reset by peer reason recv routine: unexpected EOF reason send packet msg: write tcp 10.43.48.2:37520->10.43.64.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.43.48.2:26656->10.40.208.2:53286: write: connection reset by peer reason send packet msg: write tcp 10.43.48.2:45092->10.40.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.43.48.2:26656->10.45.80.2:44282: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.45.96.2:26656->10.35.96.1:49800: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:36382->10.32.0.4:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:26656->10.41.80.3:50112: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:26656->10.33.224.1:33322: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:26656->10.33.96.2:38352: write: broken pipe reason send packet msg: write tcp 10.42.16.3:26656->10.36.240.3:45140: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:35048->10.41.128.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:47920->10.42.176.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:26656->10.45.16.1:56478: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:56504->10.41.128.2:26656: write: broken pipe reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:59102->10.47.136.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:26656->10.37.112.2:36540: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:56866->10.36.112.3:26656: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:26656->10.45.32.0:58130: use of closed network connection reason send packet msg: write tcp 10.42.16.3:26656->10.38.224.1:34574: use of closed network connection reason send packet msg: write tcp 10.42.16.3:58970->10.46.0.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:36032->10.34.64.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.39.208.2:59066: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:33664->10.35.96.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:47438->10.32.224.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.45.80.2:49130: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:43106->10.37.224.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.45.0.2:55386: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.47.136.3:56628: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:48146->10.46.64.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.45.96.2:53978: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.34.160.0:40976: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:37404->10.43.64.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:48292->10.37.208.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:53744->10.44.0.1:26656: use of closed network connection reason switch: reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.37.32.1:53654->10.45.0.2:26656: write: connection reset by peer reason send routine: pong timeout reason recv routine: unexpected EOF reason send packet msg: write tcp 10.45.16.1:47740->10.40.208.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.45.16.1:26656->10.42.176.2:37026: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.32.0.4:26656->10.45.96.2:52746: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.32.0.4:56564->10.44.0.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.47.232.2:36624->10.37.32.1:26656: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.47.232.2:49128->10.40.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.47.232.2:45816->10.36.240.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.208.2:42522->10.35.224.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.32.96.2:51540->10.46.0.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:50418->10.47.136.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:26656->10.36.160.2:41668: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:26656->10.36.160.2:35294: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:26656->10.46.0.2:52264: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:57978->10.43.48.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason recv routine: read tcp 10.45.32.0:40298->10.43.32.0:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:38690->10.42.176.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:37068->10.35.224.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:57506->10.43.48.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:36120->10.32.224.1:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:49678->10.37.224.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:58130->10.42.16.3:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.47.232.2:46144: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.32.192.3:48558: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.41.80.3:56200: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.40.224.2:38002: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.37.32.1:49330: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.33.224.1:40688: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.36.112.3:33104: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:43602->10.34.160.0:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:55438->10.38.64.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:55276->10.41.192.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.44.0.1:43330: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.46.0.2:42710: use of closed network connection reason switch: reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.33.96.2:37696: use of closed network connection reason recv routine: read tcp 10.45.32.0:26656->10.44.240.2:49324: use of closed network connection reason switch: reason send packet msg: write tcp 10.37.224.2:60512->10.47.136.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.37.224.2:55082->10.40.208.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.37.224.2:52052->10.47.136.3:26656: write: broken pipe reason send packet msg: write tcp 10.41.128.2:26656->10.32.96.2:39690: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.46.64.2:50028->10.47.192.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.64.2:26656->10.45.72.2:47956: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.46.64.2:26656->10.36.160.2:45260: write: connection reset by peer reason send packet msg: write tcp 10.46.64.2:26656->10.41.192.2:47322: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.39.160.1:26656->10.32.0.4:50354: write: connection reset by peer reason send packet msg: write tcp 10.45.72.2:53496->10.35.224.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.45.72.2:26656->10.34.64.1:36118: use of closed network connection reason send packet msg: write tcp 10.45.72.2:26656->10.40.208.2:60702: write: connection reset by peer reason send packet msg: write tcp 10.45.72.2:58804->10.35.96.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.45.72.2:34476->10.43.64.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.45.72.2:50080->10.37.208.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.44.0.1:26656->10.40.224.2:42714: write: connection reset by peer reason send packet msg: write tcp 10.44.0.1:26656->10.36.160.2:49030: write: connection reset by peer reason send packet msg: write tcp 10.44.0.1:26656->10.42.16.3:57810: use of closed network connection reason send packet msg: write tcp 10.44.0.1:26656->10.32.224.1:33820: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.47.192.3:26656->10.46.0.2:56442: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.47.192.3:48878->10.45.16.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.43.64.2:26656->10.34.64.1:48716: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.43.64.2:26656->10.45.96.2:50944: write: broken pipe reason send routine: pong timeout reason send packet msg: write tcp 10.45.0.2:33580->10.47.232.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.45.0.2:60898->10.47.176.3:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.45.0.2:39358->10.35.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.45.0.2:26656->10.33.224.1:51416: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:59852->10.35.224.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:44188->10.41.80.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:26656->10.36.160.2:57224: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:34992->10.44.0.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:26656->10.42.176.2:42348: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:26656->10.45.16.1:44282: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:37428->10.36.112.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:50588->10.39.160.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:41956->10.37.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:59904->10.34.64.1:26656: write: connection reset by peer reason recv routine: unexpected EOF reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:26656->10.41.128.2:58036: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:26656->10.38.224.1:39242: use of closed network connection reason send packet msg: write tcp 10.39.0.2:36122->10.40.224.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:48788->10.37.32.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:54566->10.36.160.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:59248->10.40.208.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:58192->10.39.96.3:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:51744->10.34.160.0:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.39.0.2:56630->10.43.32.0:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:40520->10.35.224.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:26656->10.41.192.2:56706: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:47396->10.46.0.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:26656->10.37.160.1:58696: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:26656->10.47.192.3:36646: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:35992->10.43.48.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.45.80.2:37624->10.47.136.3:26656: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.45.80.2:26656->10.39.0.2:48146: write: connection reset by peer reason send packet msg: write tcp 10.45.80.2:33910->10.37.32.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.35.96.1:26656->10.33.96.2:57562: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.36.240.3:26656->10.47.136.3:43226: write: connection reset by peer reason send packet msg: write tcp 10.36.240.3:38992->10.45.96.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.40.224.2:43406->10.44.0.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.40.224.2:36240->10.39.96.3:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.40.224.2:26656->10.33.96.2:33540: use of closed network connection reason send packet msg: write tcp 10.40.224.2:43308->10.41.80.3:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.40.224.2:26656->10.32.224.1:37638: write: connection reset by peer reason send packet msg: write tcp 10.40.224.2:32790->10.38.64.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.38.64.2:26656->10.36.240.3:42908: write: connection reset by peer reason send packet msg: write tcp 10.38.64.2:26656->10.41.80.3:44398: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.36.160.2:47852->10.42.176.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.160.2:56098->10.47.176.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.160.2:26656->10.45.0.2:51810: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.36.160.2:26656->10.37.32.1:51438: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.36.160.2:40256->10.37.224.2:26656: write: broken pipe reason send packet msg: write tcp 10.36.160.2:40050->10.32.192.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.160.2:39410->10.41.192.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.36.160.2:26656->10.45.72.2:37374: write: connection reset by peer reason send routine: pong timeout reason recv routine: unexpected EOF reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.39.96.3:26656->10.45.72.2:48104: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.40.208.2:36954->10.36.160.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.40.208.2:26656->10.42.16.3:42716: write: connection reset by peer reason send packet msg: write tcp 10.40.208.2:43250->10.32.0.4:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.208.2:26656->10.36.112.3:34088: write: connection reset by peer reason send packet msg: write tcp 10.42.208.2:57556->10.42.16.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.208.2:26656->10.43.64.2:34844: write: connection reset by peer reason send packet msg: write tcp 10.42.208.2:57336->10.33.96.2:26656: use of closed network connection reason send packet msg: write tcp 10.42.208.2:26656->10.45.16.1:33824: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.47.136.3:52072->10.47.232.2:26656: use of closed network connection reason send packet msg: write tcp 10.47.136.3:44618->10.33.96.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.47.136.3:44418->10.36.160.2:26656: use of closed network connection reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.32.224.1:34264->10.35.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.224.1:39494->10.47.136.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.224.1:47612->10.35.96.1:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.32.224.1:38234->10.37.208.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.32.224.1:26656->10.36.240.3:38766: write: connection reset by peer reason send packet msg: write tcp 10.32.224.1:36570->10.45.72.2:26656: write: broken pipe reason send packet msg: write tcp 10.32.224.1:35292->10.34.64.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.224.1:44620->10.44.240.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout

AC

find the cause of most if not all of the above issues and fix them. This would likely involve re-running these traces in knuu considering that is the future of our testing infra, and then addressing them. Considering all nodes in the network tests are behaving honestly and with adequate resources, in theory they should not disconnect.

The text was updated successfully, but these errors were encountered:

evan-forbes · 2024-06-10T14:16:44Z

It could be the case that nodes are being overwhelmed, in which case debugging bottlenecks in the p2p stack would also address the issues here. Therefore, it makes sense to debug that before moving forward with this.

blocked by #1362

evan-forbes added T:Bug Type: Bug (confirmed) WS: Big Blonks 🔭 Improving consensus critical gossiping protocols labels Jun 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Peers disconnect frequently despite running default software #1368

Peers disconnect frequently despite running default software #1368

evan-forbes commented Jun 3, 2024 •

edited

Loading

evan-forbes commented Jun 10, 2024 •

edited

Loading

Peers disconnect frequently despite running default software #1368

Peers disconnect frequently despite running default software #1368

Comments

evan-forbes commented Jun 3, 2024 • edited Loading

AC

evan-forbes commented Jun 10, 2024 • edited Loading

evan-forbes commented Jun 3, 2024 •

edited

Loading

evan-forbes commented Jun 10, 2024 •

edited

Loading