-
Notifications
You must be signed in to change notification settings - Fork 987
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize historical event scanning #1130
Comments
Really we should do the bloom filter check locally by first pulling in all block headers, which we probably already do anyways. |
Expanding on the previous comment, scanning logs through
Proposed SolutionThe block ingestor would always keep a complete and consistent chain of block headers in the DB. This way, we can scan the bloom filter locally and find the relevant block pointers. Note that we don't really use the logs returned from A challenge is how to get all the headers in the database. By batching
|
This part of the issue we can solve while still using |
Just now we're hitting a (so far undocumented) limit in Alchemy, where expensive So splitting these calls up by contract addresses (or by smaller groups of addresses) and then merging the results afterwards would be a good next step. |
Treat it the same as the Infura log limit, and reduce the range. I still want to do #1130 and try separating the log calls, but this is a quick way for affected subgraphs make progress again, and is desirable anyways.
Treat it the same as the Infura log limit, and reduce the range. I still want to do #1130 and try separating the log calls, but this is a quick way for affected subgraphs make progress again, and is desirable anyways.
Do you want to request a feature or report a bug?
Performance bug.
What is the current behavior?
Scanning for common events like
Transfer
is currently slow. Scanning for events from many data sources (e.g. created by templates) at once is also slow. Both used in combination... you can guess for yourself.The reason for this is that all event signatures that we are scanning for are hashed together for the
eth_getLogs
filter. The more event signatures are hashed together, the less efficient the filter is. Also, as soon as there is more than one contract / data source involved, the filter we're building is no longer tied to a single contract address and therefor much less efficient.What is the expected behavior?
Scanning for any event should be fast, regardless of how many data sources there are. It should not take days to handle
Transfer
events.The text was updated successfully, but these errors were encountered: