Iterate one storage map over blocks

[mmyyrroonn]

We have a storage map with huge amount of keys. Is it possible to iterate one storage map over blocks? I cannot store the PrefixIterator into the DB since it cannot be encoded. Meanwhile, fields of PrefixIterator are private. I cannot build a new PrefixIterator with a given previous_key.

[gui1117]

I agree, I think we should create 1 or 2 method on PrefixIterator.

• fn start_after_key(self, raw_previous_key: Vec<u8>) -> Self which allow for the iterator to start from a specific previous_key.
• and maybe fn last_key(&self) -> Vec<u8> to get the last previous_key. (but this can be computed from the last fetched key as well).

[shawntabrizi]

@KiChjang maybe an interesting idea for you

[xlc]

We already doing this in offchain worker and built an iterator for this purpose https://github.com/open-web3-stack/open-runtime-module-library/blob/443ee91bc2ca5f1fc155c0378eef6e89b67e2e97/utilities/src/iterator.rs#L102

[KiChjang]

Oh yes, this is definitely useful not just for pagination, but for multiblock consensus mechanisms as well. We should definitely do this.

[kianenigma]

We have a storage map with huge amount of keys. Is it possible to iterate one storage map over blocks? I cannot store the PrefixIterator into the DB since it cannot be encoded. Meanwhile, fields of PrefixIterator are private. I cannot build a new PrefixIterator with a given previous_key.

until a nicer abstraction is being built, you can simply store the last key in storage and use sp_io::storage::next_key. See an example here:

https://github.com/paritytech/substrate/pull/8197/files#diff-847a61d013ce4aedef41be8e70a2b09d96eb3ac5d5ce3d3da45e03915ab1fa90R908