Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chainxchg write response timeout #5254

Merged
merged 1 commit into from
Jan 5, 2021

Conversation

turuslan
Copy link
Contributor

Description
Chainxchg is slow. Requests with depth >3 are timed out (60 seconds by default). High CPU usage on server side.
We profiled chainxchg methods and found a bottleneck.

Problem
From profiling we found that the longest operation is response writing.
Response is marshalled by MarshalCBOR directly to network stream.
MarshalCBOR makes a lot of small writes with average size of 1 byte (because it writes each CBOR header separately).
Each of these writes is treated like network packet.
Each packet is encrypted and wrapped by underlying security and multiplexing protocols, which adds overhead to packet size.
A lot of small writes with large overhead are causing timeouts.

Solution
Buffered writes.
With this patch we could make requests with larger depth (e.g. 100) without timing out.
Would be good to have this patch applied on mainnet nodes.

Copy link
Contributor

@Kubuxu Kubuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great find.
I was quite sure we did it already, but maybe it was on another stream.

@magik6k magik6k merged commit a15080b into filecoin-project:master Jan 5, 2021
bibibong pushed a commit to EpiK-Protocol/go-epik that referenced this pull request Jan 9, 2021
@turuslan turuslan deleted the fix/chainxchg-buffered branch January 11, 2021 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants