Discussion thread for EIP-2464: eth/65: transaction annoucements and retrievals

[karalabe]

Abstract

The eth network protocol has two ways to propagate a newly mined block: it can be broadcast to a peer in its entirety (via NewBlock (0x07) in eth/64 and prior) or it can be announced only (via NewBlockHashes (0x01)). This duality allows nodes to do the high-bandwidth broadcasting (10s-100s KB) for a logarithmic number of peers; and the low-bandwidth announcing (10s-100s B) for the remaining linear number of peers. The logarithmic broadcast is enough to reach all well connected nodes, but the linear announce is needed to get across degenerate topologies. This works well.

The eth protocol, however, does not have a similar dual mechanism for propagating transactions, so nodes need to rely on broadcasting (via Transactions (0x02)). To cater for degenerate topologies, transactions cannot be broadcast logarithmically, rather they need to be transferred linearly to all peers. With N peers, each node will transfer the same transaction N times (counting both directions), whereas 1 would be enough in a perfect world. This is a significant waste.

A similar issue arises when a new network connection is made between two nodes, as they need to sync up their transaction pools, but the pool is just a soup of dangling transactions. Without a way to deduplicate transactions remotely, each node is forced to naively transfer their entire list of transactions to the other side. With pools containing thousands of transactions, a naive transfer amounts to 10s-100s MB, most of which is useless. There is no better way, however.

This EIP introduces two additional message types into the eth protocol (releasing a new version, eth/65): NewPooledTransactionHashes (0x08) to announce a set of transactions without their content; and GetPooledTransactions (0x09) to request a batch of transactions by their announced hash. This permits reducing the bandwidth used for transaction propagation from linear complexity in the number of peers to logarithmic one; and also reducing the initial transaction exchange from 10s-100s MB to len(pool) * 32B ~= 128KB.

#2464

[lithp]

In case you find this useful, I was writing my own EIP for this and tried to collect some data for it:

I picked a random hour and weighed all the transactions included in blocks from that hour. The first block was 9039541 and the last was 9039766. All the transactions from that period consumed 5348965 bytes (5.1MB). If I compress each transaction in its own message they consume a total of 3316386 bytes (3.16MB). Averaged over the hour these two block represent, that comes out to ~921 bytes/sec of transactions.

Because nodes attempt not to send transactions to nodes which have already announced that transaction, each connection between two nodes will have each transaction sent over it about once. There is the occasional race where both nodes simultaneously send the same transaction to each other but this should be rare.

Geth nodes connect to up to 25 nodes, so a maxed-out geth node spends at least 921*25 -> 22.5 KB of data each second on transaction propagation. This is an under-estimate, because there is some number of transactions which are propagated and which never make it into the chain. I'm working on an estimate for how common that is.

Just for fun, I looked up Google Cloud Engine's egress pricing, $0.12/GB. So, a geth node which didn't participate in transaction propagation would save ~$10/month. This EIP won't cut the required bandwidth to 0 but across all 8000 ethereum nodes in the network this change could save the network up to $80k each month.

[holiman]

cc @sorpaas any comments from Parity?

[karalabe]

To add our own metrics, here's a chart of the last 7 days from one of our bootnodes wrt transaction propagation (other bootnodes look the same):

Apart from a weird spike on Sunday, the bandwidth used by ~250 eth peers is 300-400KB/s downstream and 400-500KB/s upstream, so lets say roughly 800KB/s in both directions, or around 3.2KB/s per peer (3.5x of @lithp's numbers, but I'm just "reading" the charts, so it may be off a bit).

With my numbers (and Geth's default of 50 peers, it was raised in 1.9.0), it's 160KB/s wasted on shuffling transactions (seems consistent with a plain full node - not bootnode - charts):

160KB/s would actually accumulate to $48/mo at the GCE $0.12/GB price range, or $384K across 8000 Geth nodes.

[atoulme]

Hey folks, is this EIP the only EIP tied to eth/65? I ask to understand the scope of work. If this is the only EIP for this version, then it would be reasonable to flip to eth/65 as soon as this is implemented. Please correct me if I'm missing something.

[holiman]

Yes, this is it. Geth already shipped it, so it's out in the wild already :)

[holiman]

Although, you probably shouldn't "flip to", but rather add support for 65. 

[atoulme]

Right, sorry, bad wording.

[timbeiko]

Thanks to @atoulme's work, Besu now supports eth/65 starting from v1.4.3 😄

[uri-bloXroute]

Not sure I'm following what's going on here... was this implemented in geth and besu and being used on mainnet? I thought it wasn't, but I'm not sure how to interpret @holiman and @timbeiko's comments about being in the wild

[lightclient]

@uri-bloXroute yes Geth has eth/65, see ethereum/go-ethereum#20234. Not sure about Besu.

[holiman]

Yup, we have it (thought my comment was pretty clear(?) :) )The other comment sounds to me pretty clear that Besu has it too. On mainnet, yes.

[uri-bloXroute]

I think what threw me off was that the EIP is still in "draft" status, but I haven't took an active role in EIPs in the past, so I guess the flow is discuss -> implement -> testnet -> try in mainnet -> update protocol?

[timbeiko]

@uri-bloXroute because this EIP doesn't change the consensus rules, clients can ship it (incl. on mainnet) whenever they have it implemented.

[MicahZoltu]

@uri-bloXroute @timbeiko The proper process is to move this to Last Call and then start doing client implementations. For a Core EIP:

• Draft means "looking for initial feedback and thoughts"
• Last Call means "we think this is ready to get into clients and discuss among All Core Devs"
• Accepted (very roughly) means that "implementations exist and the EIP is effectively immutable"
• Final means it is scheduled for inclusion in a future hard fork

Sadly, that process is rarely followed and people often start doing implementations of EIPs in draft form. I suspect this is because the EIP process has historically been a bit slow and people don't want to wait for a Last Call merge before they move forward.

@karalabe If you think that this EIP is "done" (you aren't actively iterating on it anymore) then you can submit a PR to move it into Last Call to signal to other client developers that it is ready for broader discussion and implementation attempts.

[karalabe]

@MicahZoltu We got a confirm from all core devs at some ancient meeting that the EIP is good to go before rolling it out. No idea why it's still in Draft mode, should be Final really, not even Last Call. It's not gonna change.

[karalabe]

Looking through the networking EIPs https://eips.ethereum.org/networking there is 7 in draft mode, but in reality all but the last have been shipped a very long time ago by possibly almost all clients and are final.

[MicahZoltu]

@karalabe We should definitely resolve that. Doing so is a 2 step process:

1. Submit a PR to EIP-2464: eth/65: transaction announcements and retrievals #2464 changing its status to Last Call and adding a review-period-end: 2020-08-01 line to the header.
2. Come back on 2020-08-01 and submit a PR to remove the review-period-end line and change the status to Accepted or Final (if it is already live in Ethereum then it should be Final.

If you mention me in those two PRs, I'll promise to prioritize getting them merged ASAP. 😄

Same for all of the other networking EIPs. I would like to get stuff moved out of Draft and into more appropriate statuses.

[github-actions]

There has been no activity on this issue for two months. It will be closed in a week if no further activity occurs. If you would like to move this EIP forward, please respond to any outstanding feedback or add a comment indicating that you have addressed all required feedback and are ready for a review.

[github-actions]

This issue was closed due to inactivity. If you are still pursuing it, feel free to reopen it and respond to any feedback or request a review in a comment.