remote_endpoint: Transport endpoint is not connected exception

[pnx]

Version: 3.2.4

I know this is for wax. but the code should be the same. Nodeos throws an exception in the SHiP plugin. I think this type of errors should be recoverable. but the exception is never caught and stops the node.

info  2023-07-19T22:01:49.938 net-0     net_plugin.cpp:2949           handle_message       ] ["172.16.8.131:9676:blk - 8d40b83" - 7 172.16.6.67:9676] received time_message
info  2023-07-19T22:01:50.024 net-1     net_plugin.cpp:2949           handle_message       ] ["peer1-wax.eosphere.io:9876 - 994a65d" - 13 158.69.7.206:9876] received time_message
info  2023-07-19T22:01:50.187 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667139},"last_irreversible":{"block_num":256666809},"this_block":{"block_num":256667139}} to send queue
info  2023-07-19T22:01:50.187 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block a427932978747d86... #256667139 @ 2023-07-19T22:01:50.500 signed by bountyblokbp [trxs: 54, lib: 256666809, confirmed: 0, net: 7176, cpu: 23347, elapsed: 7601, time: 11039, latency: -312 ms]
info  2023-07-19T22:01:50.372 net-0     net_plugin.cpp:2949           handle_message       ] ["eosn-wax-seed54:9876 - 2a41144" - 14 216.66.68.30:9876] received time_message
info  2023-07-19T22:01:50.405 net-0     net_plugin.cpp:2949           handle_message       ] ["0.0.0.0:9876 - 080681a" - 10 135.125.160.57:36715] received time_message
info  2023-07-19T22:01:50.698 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667140},"last_irreversible":{"block_num":256666809},"this_block":{"block_num":256667140}} to send queue
info  2023-07-19T22:01:50.698 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block 00875a3d8de8ca3e... #256667140 @ 2023-07-19T22:01:51.000 signed by bountyblokbp [trxs: 70, lib: 256666809, confirmed: 0, net: 9552, cpu: 31456, elapsed: 13926, time: 18698, latency: -301 ms]
info  2023-07-19T22:01:51.189 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667141},"last_irreversible":{"block_num":256666809},"this_block":{"block_num":256667141}} to send queue
info  2023-07-19T22:01:51.189 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block 2e2f36a793e51b9d... #256667141 @ 2023-07-19T22:01:51.500 signed by bountyblokbp [trxs: 65, lib: 256666809, confirmed: 0, net: 8296, cpu: 28302, elapsed: 8203, time: 11728, latency: -310 ms]
info  2023-07-19T22:01:51.694 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667142},"last_irreversible":{"block_num":256666809},"this_block":{"block_num":256667142}} to send queue
info  2023-07-19T22:01:51.694 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block f7765b412c3072a6... #256667142 @ 2023-07-19T22:01:52.000 signed by bountyblokbp [trxs: 74, lib: 256666809, confirmed: 0, net: 9456, cpu: 32234, elapsed: 9727, time: 13577, latency: -305 ms]
info  2023-07-19T22:01:51.965 net-1     net_plugin.cpp:2949           handle_message       ] ["api.waxsweden.org - 955f8e1" - 2 172.16.6.21:9676] received time_message
info  2023-07-19T22:01:52.024 net-1     net_plugin.cpp:2949           handle_message       ] ["0.0.0.0:10301 - 066eae1" - 12 148.251.181.48:10301] received time_message
info  2023-07-19T22:01:52.193 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667143},"last_irreversible":{"block_num":256666809},"this_block":{"block_num":256667143}} to send queue
info  2023-07-19T22:01:52.193 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block 60184fbe79dc81f9... #256667143 @ 2023-07-19T22:01:52.500 signed by bountyblokbp [trxs: 37, lib: 256666809, confirmed: 0, net: 15504, cpu: 20390, elapsed: 6261, time: 9029, latency: -306 ms]
info  2023-07-19T22:01:52.714 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667144},"last_irreversible":{"block_num":256666809},"this_block":{"block_num":256667144}} to send queue
info  2023-07-19T22:01:52.714 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block cf4d19d594bc5f99... #256667144 @ 2023-07-19T22:01:53.000 signed by bountyblokbp [trxs: 107, lib: 256666809, confirmed: 0, net: 16232, cpu: 51460, elapsed: 18243, time: 24733, latency: -285 ms]
info  2023-07-19T22:01:52.732 net-0     net_plugin.cpp:2949           handle_message       ] ["eosn-wax-seed54:9876 - 2a41144" - 14 216.66.68.30:9876] received time_message
info  2023-07-19T22:01:52.868 net-1     net_plugin.cpp:1862           recv_handshake       ] ["peer1-wax.eosphere.io:9876 - 994a65d" - 13 158.69.7.206:9876] handshake lib 256666809, head 256667143, head id 60184fbe79dc81f9.. sync 4, head 256667144, lib 256666809
info  2023-07-19T22:01:52.979 net-0     net_plugin.cpp:1817           recv_handshake       ] ["peer1-wax.eosphere.io:9876 - 994a65d" - 13 158.69.7.206:9876] handshake lib 256666809, head 256667144, head id cf4d19d594bc5f99.. sync 0, lib 256666809
info  2023-07-19T22:01:53.111 net-1     net_plugin.cpp:1839           recv_handshake       ] ["api.waxsweden.org - 6a26be1" - 3934 172.16.0.37:34654] handshake lib 250929815, head 250930147, head id a3af5c7274a75afa.. sync 2, head 256667144, lib 256666809
info  2023-07-19T22:01:53.116 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667145},"last_irreversible":{"block_num":256666809},"this_block":{"block_num":256667145}} to send queue
info  2023-07-19T22:01:53.116 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block 68ebbf9957d78e71... #256667145 @ 2023-07-19T22:01:53.500 signed by bountyblokbp [trxs: 5, lib: 256666809, confirmed: 0, net: 632, cpu: 2396, elapsed: 1078, time: 1955, latency: -383 ms]
info  2023-07-19T22:01:53.178 net-1     net_plugin.cpp:2949           handle_message       ] [""172.16.8.132:9671:trx" - cb7bde4" - 6 172.16.0.13:9671] received time_message
info  2023-07-19T22:01:53.674 net-0     net_plugin.cpp:2949           handle_message       ] ["peer1-wax.eosphere.io:9876 - 994a65d" - 13 158.69.7.206:9876] received time_message
info  2023-07-19T22:01:53.739 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667146},"last_irreversible":{"block_num":256666809},"this_block":{"block_num":256667146}} to send queue
info  2023-07-19T22:01:53.743 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block 9b3063339c45a0f3... #256667146 @ 2023-07-19T22:01:54.000 signed by eosdacserver [trxs: 60, lib: 256666821, confirmed: 240, net: 8248, cpu: 28206, elapsed: 8307, time: 11727, latency: -256 ms]
info  2023-07-19T22:01:54.250 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256667147}} to send queue
info  2023-07-19T22:01:54.250 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666810}} to send queue
info  2023-07-19T22:01:54.250 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block d515cc4c79e78988... #256667147 @ 2023-07-19T22:01:54.500 signed by eosdacserver [trxs: 86, lib: 256666821, confirmed: 0, net: 11280, cpu: 39876, elapsed: 11369, time: 17007, latency: -249 ms]
info  2023-07-19T22:01:54.251 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666811}} to send queue
info  2023-07-19T22:01:54.251 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666812}} to send queue
info  2023-07-19T22:01:54.251 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666813}} to send queue
info  2023-07-19T22:01:54.252 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666814}} to send queue
info  2023-07-19T22:01:54.252 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666815}} to send queue
info  2023-07-19T22:01:54.357 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666816}} to send queue
info  2023-07-19T22:01:54.452 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666817}} to send queue
info  2023-07-19T22:01:54.535 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666818}} to send queue
info  2023-07-19T22:01:54.679 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667147},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666819}} to send queue
info  2023-07-19T22:01:54.749 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667148},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256667148}} to send queue
info  2023-07-19T22:01:54.749 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block 19ab553c338368c5... #256667148 @ 2023-07-19T22:01:55.000 signed by eosdacserver [trxs: 79, lib: 256666821, confirmed: 0, net: 10640, cpu: 41278, elapsed: 13466, time: 17808, latency: -250 ms]
info  2023-07-19T22:01:54.798 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667148},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666820}} to send queue
info  2023-07-19T22:01:54.888 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667148},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256666821}} to send queue
info  2023-07-19T22:01:54.924 net-0     net_plugin.cpp:2949           handle_message       ] ["ed8.s - 5204350" - 16 172.16.0.126:46536] received time_message
info  2023-07-19T22:01:55.262 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667149},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256667149}} to send queue
info  2023-07-19T22:01:55.263 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block 9d4cfdddc3a8ed63... #256667149 @ 2023-07-19T22:01:55.500 signed by eosdacserver [trxs: 106, lib: 256666821, confirmed: 0, net: 15760, cpu: 55874, elapsed: 16493, time: 23204, latency: -236 ms]
info  2023-07-19T22:01:55.746 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667150},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256667150}} to send queue
info  2023-07-19T22:01:55.746 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block b342c0974610703c... #256667150 @ 2023-07-19T22:01:56.000 signed by eosdacserver [trxs: 79, lib: 256666821, confirmed: 0, net: 10216, cpu: 41268, elapsed: 10861, time: 15368, latency: -253 ms]
info  2023-07-19T22:01:56.247 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667151},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256667151}} to send queue
info  2023-07-19T22:01:56.247 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block ab7b6947d63db3f6... #256667151 @ 2023-07-19T22:01:56.500 signed by eosdacserver [trxs: 69, lib: 256666821, confirmed: 0, net: 9248, cpu: 40122, elapsed: 12768, time: 17056, latency: -252 ms]
info  2023-07-19T22:01:56.613 net-0     net_plugin.cpp:2949           handle_message       ] ["hrm.sen - cbce06f" - 15 172.16.0.27:37950] received time_message
info  2023-07-19T22:01:56.746 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667152},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256667152}} to send queue
info  2023-07-19T22:01:56.746 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block bc3bdd22c2f13ed2... #256667152 @ 2023-07-19T22:01:57.000 signed by eosdacserver [trxs: 73, lib: 256666821, confirmed: 0, net: 10472, cpu: 36063, elapsed: 10735, time: 14959, latency: -253 ms]
info  2023-07-19T22:01:57.256 nodeos    state_history_plugin.c:338    send_update          ] pushing result {"head":{"block_num":256667153},"last_irreversible":{"block_num":256666821},"this_block":{"block_num":256667153}} to send queue
info  2023-07-19T22:01:57.256 nodeos    producer_plugin.cpp:531       on_incoming_block    ] Received block 08d5536a52885f12... #256667153 @ 2023-07-19T22:01:57.500 signed by eosdacserver [trxs: 89, lib: 256666821, confirmed: 0, net: 14000, cpu: 47534, elapsed: 13516, time: 19155, latency: -243 ms]
info  2023-07-19T22:01:57.535 SHiP-0    state_history_plugin.c:150    start                ] incoming connection
info  2023-07-19T22:01:57.535 net-1     net_plugin.cpp:2949           handle_message       ] ["eosn-wax-seed54:9876 - 2a41144" - 14 216.66.68.30:9876] received time_message
error 2023-07-19T22:01:57.535 SHiP-0    state_history_plugin.c:376    callback             ] async_accept: Connection reset by peer
libc++abi: terminating with uncaught exception of type boost::wrapexcept<boost::system::system_error>: remote_endpoint: Transport endpoint is not connected

Link to the reported line: https://github.com/AntelopeIO/leap/blob/v3.2.4/plugins/state_history_plugin/state_history_plugin.cpp#L376

Same but for wax-leap: https://github.com/cc32d9/wax-leap/blob/v3.2.4wax01/plugins/state_history_plugin/state_history_plugin.cpp#L376

[coachbjork]

Same error crashed another 3.2.4 node today.

This node is not running state history, but is one of our transaction api nodes.

plugin = eosio::chain_plugin
plugin = eosio::chain_api_plugin
plugin = eosio::db_size_api_plugin
plugin = eosio::net_api_plugin
plugin = eosio::producer_api_plugin
plugin = eosio::producer_plugin

libc++abi: terminating with uncaught exception of type boost::wrapexcept<boost::system::system_error>: remote_endpoint: Transport endpoint is not connected

[heifner]

Fixed in 4.0: #1175

[ericpassmore]

Added LGTM tag, and targeting this to merge into the 3.2 branch. Will put out a new release with the fix.