Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tracking Issue] Flat storage for resharding V3 #12174

Open
15 of 25 tasks
Trisfald opened this issue Sep 30, 2024 · 2 comments
Open
15 of 25 tasks

[Tracking Issue] Flat storage for resharding V3 #12174

Trisfald opened this issue Sep 30, 2024 · 2 comments
Assignees

Comments

@Trisfald
Copy link
Contributor

Trisfald commented Sep 30, 2024

Part of #11881

Goal

Support resharding V3 for Flat Storage. On shard layout, Flat Storage must be able to take a shard and split it into two children.
The entire process will happen in the background with no strict time requirement. Additional processing resources consumption should be minimal because, in the meantime, the node will continue to apply blocks and chunks.

Sub-tasks

  • Implementation
    • Support for splitting a Flat storage shard
      • Handle all Flat storage key-value pairs
        • Simple keys
        • Receipts-like keys
      • Possibility to interrupt resharding
      • Background processing
      • Creating and apply deltas on children shards
      • Handle parent shard deletion
    • Add observability
      • Log progress of background task
      • Log progress of catchup
      • Time shard deletions operations
      • Metric for status of flat storage (?)
      • Metric for progress for split / catchup
    • Integration with ReshardingManager
    • Handle parent flat storage deltas split
    • Better limits on background task iteration, like key size
    • Trigger memtrie rebuild
  • Tests

    • Unit tests
      • Basic flat storage resharder functionality
      • Splitting simple keys
      • More complex resharding with other types of keys
      • Catchup children
      • Parent flat state deltas handling
    • Integration tests
      • To be covered with test loop resharding test
    • Test with forknet: resharding, restart
      • To be done for resharding as a whole
  • Nice to have

    • Refactoring to avoid sending an instance of FlatStorageResharder inside actor messages.
@Trisfald
Copy link
Contributor Author

merged #12164 tackling the introduction of FlatStorageResharder and simple key splitting

@Trisfald
Copy link
Contributor Author

merged #12223 to handle all key types

github-merge-queue bot pushed a commit that referenced this issue Oct 17, 2024
Adding log entries telling the time spent on the longer operations
happening during flat storage resharding.

Part of flat storage resharding issue (#12174).
github-merge-queue bot pushed a commit that referenced this issue Oct 18, 2024
…2246)

This PR improves the way batches are handled in the background task that
splits a shard. In particular, I'm re-using the good old `batch_size`
and `batch_delay` to throttle processing.

Part of #12174
github-merge-queue bot pushed a commit that referenced this issue Oct 21, 2024
Implementation of flat storage deltas split for flat storage resharding.

Before this change, only flat storage key pairs at flat head height were
split during resharding. Now flat storage deltas are part of the
splitting process as well.
On a high level, it works by concatenating the iterator over flat
storage and the iterator over deltas for all block heights between chain
head an flat storage head.

Part of #12174
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant