Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quick flat state removal for non-tracked shard #8332

Closed
pugachAG opened this issue Jan 11, 2023 · 4 comments · Fixed by #9109
Closed

Quick flat state removal for non-tracked shard #8332

pugachAG opened this issue Jan 11, 2023 · 4 comments · Fixed by #9109
Assignees
Labels
A-storage Area: storage and databases T-core Team: issues relevant to the core team

Comments

@pugachAG
Copy link
Contributor

pugachAG commented Jan 11, 2023

The original idea was to use delete_range introduced in #8200 when cleaning flat state. That doesn't work due to sub-accounts for fixed shards.

With #8670 we can just remove all keys with a given shard_uid prefix.

@pugachAG pugachAG self-assigned this Jan 11, 2023
@pugachAG pugachAG mentioned this issue Jan 11, 2023
26 tasks
@pugachAG
Copy link
Contributor Author

@Longarithm let's consider the following layout with 3 shards:

ShardLayoutV1{
  fixed_shards: ["alice.near"],
  boundary_accounts: ["bob.near"],
  ..
}

What would the range we can use when cleaning shard with index 1 (with all keys before bob.near) without removing any subaccounts of alice.near (because those belong to shard with index 0)?

@Longarithm
Copy link
Member

Hm I didn't realize that there is a plan to put subaccounts of a fixed account to the same shard. This is confusing because resharding of such shards will be computation-heavy. Asked in Zulip https://near.zulipchat.com/#narrow/stream/295558-pagoda.2Fcore/topic/fixed.20shards/near/320919706

@Longarithm
Copy link
Member

Discussed offline: it is fine to ignore subaccounts feature because fixing single account is enough. Though we don't want to drop fixed accounts completely because in theory they could allow us to quickly move some large account to a separate shard.
We should mention this in comments to avoid future confusions.

@pugachAG
Copy link
Contributor Author

Update: we've decided not to implement it for now.

This requires removing subaccounts support for accounts in fixed_shards which is a pretty significant change. Also this would be the first place in our codebase where we start relying on trie keys belonging to a single shard forming ranges, which introduces more assumptions to the systems and makes it prone to breaking in case of future changes. Considering that we don't find the underlying use case to be significant enough, so we are not proceeding with this for now.

@walnut-the-cat walnut-the-cat added A-storage Area: storage and databases T-core Team: issues relevant to the core team and removed T-storage labels Mar 13, 2023
@pugachAG pugachAG linked a pull request May 26, 2023 that will close this issue
near-bulldozer bot pushed a commit that referenced this issue May 26, 2023
Use range deletion in `clean_state` when removing values from `FlatState` column. Addresses #8332
nikurt pushed a commit that referenced this issue May 31, 2023
Use range deletion in `clean_state` when removing values from `FlatState` column. Addresses #8332
nikurt pushed a commit to nikurt/nearcore that referenced this issue Jun 8, 2023
Use range deletion in `clean_state` when removing values from `FlatState` column. Addresses near#8332
nikurt pushed a commit to nikurt/nearcore that referenced this issue Jun 8, 2023
Use range deletion in `clean_state` when removing values from `FlatState` column. Addresses near#8332
nikurt pushed a commit that referenced this issue Jun 13, 2023
Use range deletion in `clean_state` when removing values from `FlatState` column. Addresses #8332
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage Area: storage and databases T-core Team: issues relevant to the core team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants