You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Goal here is to see how fast "solution 2" would be, i.e. (1) prefix trie nodes with account ID (or prefix), and (2) during resharding, iterate through the trie from the root, copy everything except for subtrees that correspond to a single account, in which case we shallow copy the roots of those subtrees.
Part (1) requires a db migration (to remove shard UID and to add account prefix), but even without a db migration, we can already estimate the speed of (2), because the trie structure itself is not changing. The amount of trie iteration we would need to do before and after the db migration would be the same.
So, the way to do this can be, to make an offline tool that iterates the trie of a specific shard, stopping at any account subtrees, and see how long that takes. The sequential random reads should be the majority of the cost. We can also include writes in there - just as an estimate - there's no way to write correct resharded data because of the existing shard UID prefix.
The point is that if this takes only a few minutes, it would probably be preferrable to solution 1 (restructure trie) because this is much simpler. Solution 1 is neat but also requires maintaining the previous code, and that can be ugly.
The text was updated successfully, but these errors were encountered:
This is step 1 of the following plan to benchmark how long does it take to perform shallow trie iteration. For more details about the goal please see #9101.
1. Add a neard subcommand that does full trie iteration.
2. Extend it to actually measure and report the time it takes.
3. Extend it with shallow iteration.
Once all three steps are done it will be possible to benchmark the full iteration, the shallow iteration and compare the time it takes for each.
This is step 1 of the following plan to benchmark how long does it take to perform shallow trie iteration. For more details about the goal please see #9101.
1. Add a neard subcommand that does full trie iteration.
2. Extend it to actually measure and report the time it takes.
3. Extend it with shallow iteration.
Once all three steps are done it will be possible to benchmark the full iteration, the shallow iteration and compare the time it takes for each.
Goal here is to see how fast "solution 2" would be, i.e. (1) prefix trie nodes with account ID (or prefix), and (2) during resharding, iterate through the trie from the root, copy everything except for subtrees that correspond to a single account, in which case we shallow copy the roots of those subtrees.
Part (1) requires a db migration (to remove shard UID and to add account prefix), but even without a db migration, we can already estimate the speed of (2), because the trie structure itself is not changing. The amount of trie iteration we would need to do before and after the db migration would be the same.
So, the way to do this can be, to make an offline tool that iterates the trie of a specific shard, stopping at any account subtrees, and see how long that takes. The sequential random reads should be the majority of the cost. We can also include writes in there - just as an estimate - there's no way to write correct resharded data because of the existing shard UID prefix.
The point is that if this takes only a few minutes, it would probably be preferrable to solution 1 (restructure trie) because this is much simpler. Solution 1 is neat but also requires maintaining the previous code, and that can be ugly.
The text was updated successfully, but these errors were encountered: