-
Notifications
You must be signed in to change notification settings - Fork 314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: make add_piece taking less time #1707
Conversation
`add_piece` operates in 64 bytes pieces. It's faster to operate in bigger chunks. This commits increases the buffer size to 4KiB. This makes adding a 32GiB piece about more than 2x faster. On the hardware I used it goes down from 14min to about 6min.
@@ -7,11 +7,13 @@ use rayon::prelude::{ParallelIterator, ParallelSlice}; | |||
|
|||
use crate::{constants::DefaultPieceHasher, pieces::piece_hash}; | |||
|
|||
const BUFFER_SIZE: usize = 4096; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you tried bumping it even further, for example, up to 1MiB-8MiB?
Even with the 4KiB, it is just 64 SHA invocations, the branch predictor is probably just warming up at that point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've tried 8MiB it didn't make a difference.
I think yet another difference would make it if the hashing would be done in parallel. But a simple "par chunker" didn't really help, as then the pieces are to small. One need to implement it manually. I decided that it's not worth it and that this improvement is already good enough for how simple it is.
Can we check if lotus is using this method? I recall it not being used, but may be wrong there. In any case, it should help speed up our big tests. |
Why is that important? It's not a breaking change, it just improves things. So if Lotus would use it, it would also be beneficial for them. |
I just wanted to know if we'd see the speed-ups outside of tests. |
add_piece
operates in 64 bytes pieces. It's faster to operate in bigger chunks. This commits increases the buffer size to 4KiB. This makes adding a 32GiB piece about more than 2x faster. On the hardware I used it goes down from 14min to about 6min.