Pagination / Streaming for Event Logs

[3lpsy]

Is your feature request related to a problem? Please describe.
As event logs can yield more results than an RPC will allows (for example, infura failing when the response has over 10k items), looping over a stream of event logs based on a filter or providing a paginated API would be very useful.

Describe the solution you'd like

Gnosis has a potential solution here where past_logs_pages/stream can return a filter based log stream.

I think something similar would be the easiest implementation. The draft implementation could start by adding a get_logs_pages method on the provider that accepts a Filter object. However, I am open to feedback / suggestions in terms of how it's designed.

Additional context
Understanding the nuances of how eth_getLogs is handled by the RPC provider may be required and may introduce additional complexity. Below is a list of use cases to research to determine how they differ and how to best handle their use cases:

• eth on infura
• eth on alchemy
• polygon on infura
• polygon on alchemy
• bsc on infura

[3lpsy]

Some additional thoughts and notes:

Most probably knew this but I didn't. eth_getLogs doesn't appear to support pagination at the RPC level. There was an attempt but it apparenttly didnt' get any traction.

The de facto approach appears to be use the block range as a substitute for a page_limit. This is unfortunate, but seems to be the only available solution.

There are edge cases where a single block may contain >1000 (or an RPCs max responses) events. Infura actually bumped their response limit up to 10k for this reason. I am basing this off of this thread from 2019. While dynamically resizing the block range down may be a useful feature, I'm not sure if accomadating this edge case where a single block has greater than 1000 responses (or the limit of the RPC) is worth it as it would require splitting topics or addresses.

So a starting solution could have the user pass in a filter and a page_range (though i prefer to call it a block_range as it's more explicit what it is). Then the provider's get_logs_pages method could create a new filter based off the filter's FilterBlockOption with a new to_block and then iterate upon it. A nice feature in this case would be catching an too many responses error and attempting to downsize the range to handle this. Though then we have to determine whether to keep the downsized range for the remainder of the stream or move it back to the original value.

[meetmangukiya]

Graph protocol has some neat dynamics around elastic block ranges when they do the indexing https://github.com/graphprotocol/graph-node/blob/38d2a8428c92ae837c1d324e15de1c433267e19c/graph/src/blockchain/polling_block_stream.rs#L352-L375

[gakonst]

@meetmangukiya WDYT about taking this issue?