-
Notifications
You must be signed in to change notification settings - Fork 264
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sending messages slows to a crawl when connecting more peers. #1362
Comments
The thinking behind this issue stems from tracing the entire lifecycle of a block part. Below we can see a block part start its journey at the bottom with the block proposer. From there, it is gossiped to the proposer's peers, and then to their peers and so on. Blue indicates that its the first time a peer received a block part. here's a zoomed in image of the above If we measure the transit time (time from calling We can see the first two hops, the first in blue from the proposer to its peers, then green from those peers to the next take a while. When we look at larger blocks, we almost universally see very very long times for the proposer to send each part to the first peer, with significantly shorter times after that. Notice the enormous blue When we look at traces of when we are sending and receiving the block parts overlayed with bandwidth, we see that most of the wait in on the send side. As soon as the nodes receive the block part, they are processing it. We can also see that we aren't using even close to all of our allocated bandwdith per peer this differs from when there are only two nodes in the network, where we see we can utilize as much bandwidth as we have allocated (up to 100MB/s) We can replicate the bug here:
I'm in the middle of digging deeper into the above to figure out exactly where the latency is coming from. Its possible that we have a bit of a "buffer hernia", where we are adding to the buffers at the configured send and receive rate, but those buffers are not actually being emptied by tcp. If this is occuring, and we don't actually have a mechanism to stop adding to the buffer when the tcp buffer is full, then this could explain why we see very late block part messages and huge delays from block parts when proposing new blocks. |
after debugging more today, it looks like we are unable to clear the buffer on the receiver side, which is forcing the send side to send data slowly. It could just be the buffer being too small on the receiver side. Will run more experiments tmrw! |
to update this further, the buffers in question here are if we add buffer to receive side to ensure that we never block the tcp connection, we don't see an improvement. When we look at tcp traces from a realistic network, we can confirm that we have plenty of room in the receive window. Unfortunately, its only during local benchmarks where we could hit the receive limit. when we change the tcp congestion control algo, we get a significant performance increase depending on the amount of network latency and packets dropped. More analysis is needed over the tcp packet traces of both algos. We should look at the window buffer sizes and the unacknowledged packets we only ever see this issues when we have network latency and we have many (10-20) peers (aka a realistic network). If we can replicate this using a tcp benchmark that uses identical or similar code here, then we can get additional confirmation there. Given that network latency and multiple peers are required to hit the issue, it would make sense that this is where the issue lies. |
…tiaorg#1362) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4.2.1 to 5.0.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v4.2.1...v5.0.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sergio Mena <sergio@informal.systems>
In the network tests where we run only two nodes, proposers are able to send block parts and message at a very high rate (100MB/s).
However when we increase the number of nodes, this stops being the case. Proposers are only able to send block parts at a fraction of the allocated bandwidth. The next steps are to recreate the issue locally by using a simple reactor and multiple local tcp connection with latency.
Before we can move forward with closing this issue, we must be able to at least prove that one aspect of the stack is causing the delay in sending messages. Once we know this, we can reevaluate the next path forward.
The text was updated successfully, but these errors were encountered: