-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disk-based cache for the downloader #306
Conversation
Performance impact: As the number of blobs increases, sampling time increases, when mining threads per shard is 24.
On the other hand, the time statistics in the |
After using an extend GetSample API on billy, the performance issue disappears. Changes to billy can be reviewed here: |
Updates on performance test based on the above comments The test has been done with build with commit 264bf03
|
This PR is purposed to solve the issue #298.
To hold the downloaded blobs in es-node in a scenario like long-time finalization, the blob cache uses https://github.com/holiman/billy as the disk-backed solution. Billy is a very simplistic data store with features like:
Since blob transactions can contain between 1-6 blobs, the stored items in the blob cache here are expected to have sizes of 128KB ~ 128KB * 6 buckets.
A similar implementation reference is the blobpool of geth: ethereum/go-ethereum#26940
Other changes include:
Tests should verify that sampling from cached blobs can work as fast as from storage file, and the mining tx confirmed successfully.