refactor: implement append-only tree

[jjyr]

Abstract

This PR took the idea from Patricia merkle tree to implement an append-only tree, which allows users access history tree and generate merkle proof.

Key features:

This PR offers the following key features:

• Internal nodes use hash(node) as the store key to ensure they are not overridden. All internal node histories are persisted in storage, although we will need to introduce a pruning mechanism in a future PR to delete unused histories.
• By using different roots allows users to access the history state and generate history merkle proof.
• Further reduction in DB access, compared to previous.

This PR implements the append-only tree with some significant changes, including:

1. Deleting the BranchKey and using hash(node) as the internal node store key. The get operation’s complexity changes from O(1) to O(log n) as we walk down from the root to find the internal node and leaf.
2. Removing the trie feature and related code. MergeValue::MergeWithZero already contains all the information the code requires.

Break changes

This update has the following break changes:

1. The merkle root calculation remains unchanged, meaning that you can verify the merkle proof generated by this PR using an older verifier.
2. The internal storage structure has changed, which means that users need to re-insert every leaf from the old SMT into the append-only tree to perform the data migration.
3. To access the history in the migrated SMT, users need to replay the history update operation in order on the new SMT.

Benchmark

The micro-benchmark shows this PR causes the regression of get(+3000%!!) and update(2%~40%) operations.

But please look at the IO counter. this PR significantly reduces the get_branch_counter to 1/3~1/4, and slightly reduces the insert_branch_counter, which means in the real world, the PR may have better performance.

Master branch with Trie

random update 100 keys, store counters: Counters { get_branch_counter: 1736, get_leaf_counter: 0, insert_branch_counter: 657, insert_leaf_counter: 100, remove_branch_counter: 0, remove_leaf_counter: 0 }
random update 10000 keys, store counters: Counters { get_branch_counter: 370773, get_leaf_counter: 0, insert_branch_counter: 131730, insert_leaf_counter: 10000, remove_branch_counter: 0, remove_leaf_counter: 0 }
random update_all 100 keys, store counters: Counters { get_branch_counter: 1709, get_leaf_counter: 0, insert_branch_counter: 650, insert_leaf_counter: 100, remove_branch_counter: 0, remove_leaf_counter: 0 }
random update_all 10000 keys, store counters: Counters { get_branch_counter: 371414, get_leaf_counter: 0, insert_branch_counter: 131971, insert_leaf_counter: 10000, remove_branch_counter: 0, remove_leaf_counter: 0 }

Append only tree

Gnuplot not found, disabling plotting
SMT update/100 time: [466.91 us 466.98 us 467.08 us]
change: [+1.0634% +1.1025% +1.1405%] (p = 0.00 < 0.05)
Performance has regressed.
SMT update/10000 time: [119.13 ms 119.44 ms 119.88 ms]
change: [+43.702% +44.593% +45.317%] (p = 0.00 < 0.05)
Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) low mild

SMT update_all/100 time: [432.03 us 432.19 us 432.33 us]
change: [+1.4337% +1.4805% +1.5244%] (p = 0.00 < 0.05)
Performance has regressed.
SMT update_all/10000 time: [109.48 ms 109.64 ms 109.79 ms]
change: [+40.561% +40.893% +41.266%] (p = 0.00 < 0.05)
Performance has regressed.

SMT get/5000 time: [1.1593 us 1.1606 us 1.1616 us]
change: [+3324.1% +3329.9% +3335.4%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) low mild
1 (10.00%) high mild
SMT get/10000 time: [1.2946 us 1.2970 us 1.3014 us]
change: [+3721.6% +3743.4% +3773.4%] (p = 0.00 < 0.05)
Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe

SMT generate merkle proof
time: [40.111 us 40.532 us 40.968 us]
change: [-83.837% -83.228% -82.616%] (p = 0.00 < 0.05)
Performance has improved.

SMT verify merkle proof time: [217.05 us 220.74 us 223.52 us]
change: [-9.6139% -8.4413% -7.2963%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) low mild

Gnuplot not found, disabling plotting
Running benches/store_counter_benchmark.rs (target/release/deps/store_counter_benchmark-3a7f5b4f6f563002)
random update 100 keys, store counters: Counters { get_branch_counter: 464, get_leaf_counter: 0, insert_branch_counter: 563, insert_leaf_counter: 100, remove_branch_counter: 0, remove_leaf_counter: 0 }
random update 10000 keys, store counters: Counters { get_branch_counter: 111784, get_leaf_counter: 0, insert_branch_counter: 121783, insert_leaf_counter: 10000, remove_branch_counter: 0, remove_leaf_counter: 0 }
random update_all 100 keys, store counters: Counters { get_branch_counter: 473, get_leaf_counter: 0, insert_branch_counter: 572, insert_leaf_counter: 100, remove_branch_counter: 0, remove_leaf_counter: 0 }
random update_all 10000 keys, store counters: Counters { get_branch_counter: 111764, get_leaf_counter: 0, insert_branch_counter: 121763, insert_leaf_counter: 10000, remove_branch_counter: 0, remove_leaf_counter: 0 }