4.0.3: p2p connections get dropped frequently

[matthewdarwin]

Looking at nodeos logs on 4.0.3, p2p connections between 4.0.3 nodes are getting dropped frequently. Downgrading to 3.2.3 on a single node seems to improve things. Exact scenario is not clear.

This was noticed because BP node is producing blocks with low number of transactions. When downgrading to 3.2.3 on the BP node, the transactions went back to the number expected.

[heifner]

Unclear if related, but the logs show a lot of duplicate connection closing. This seems to be because of: eosnetworkfoundation/mandel#756 which has been there since 3.1.

[matthewdarwin]

The nodes are configured to connect to each other, so there should be a duplicate dropped connection. We've had it set up that way for 4+ years.

[heifner]

• Add producer to Block #317926534 trx idle: 283us out of 200046us, success: 3, 593us, fail: 86, 197687us, transient: 0, 0us, other: 1483us log output.

[matthewdarwin]

Also add these idle metrics to prometheus in 5.0. I'll create a separate issue for that.

[matthewdarwin]

Kevin can add more details, but the low number of transactions seems to be because read-mode = head as default in 4.0. read-mode = speculative seems to be better. Consider if documentation or other action is appropriate.

[matthewdarwin]

There is still investigation needed on the p2p connetions dropping, but probably less urgent, and not related to 4.0 specifically (ie the issue pre-dates 4.0).

[heifner]

It appears something is severing the connection between a couple of your nodes. There is no indication that this is related to nodeos. It appears to happen with both 4.0.x and 3.2.x. Adding some additional paranoid shutdown of socket changes and make sure a re-connect happens quickly once determined that connection is dead.

Recommend decreasing your p2p-keepalive-interval-ms setting. The default is 10000 (10 seconds). Recommend reducing that to 5000 so that dead connections are determined quicker.