Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/merkledb: serializable hashing type #3176

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

x/merkledb: serializable hashing type #3176

wants to merge 3 commits into from

Conversation

darioush
Copy link
Contributor

@darioush darioush commented Jul 5, 2024 •
edited

Why this should be merged

Prep work for eth compatible serialization work (draft here)

How this works

Adds an interface "SerializeableID" which provides either a hash (ids.ID) or a serialization (to writer or bytes)
The bytes form is used in proof nodes.

The hasher interface can decode the ID from either a reader or bytes (counterpart to above).
Additionally the hasher must return the empty hash as a serializable ID. (used in addChildWithID? not 100% sure about this).

How this was tested

CI

@darioush darioush changed the title x/merkledb: serializeable hashing type x/merkledb: serializable hashing type Jul 5, 2024
@StephenButtolph
Copy link
Contributor

I think the relevant benchmarks for this are:

Benchmark_EncodeDBNode:

goos: darwin
goarch: arm64
pkg: github.com/ava-labs/avalanchego/x/merkledb
                             │   old.txt   │               new.txt                │
                             │   sec/op    │   sec/op     vs base                 │
_EncodeDBNode/empty_node-12    24.84n ± 1%   66.09n ± 2%  +166.01% (p=0.000 n=10)
_EncodeDBNode/has_value-12     36.24n ± 1%   78.60n ± 1%  +116.89% (p=0.000 n=10)
_EncodeDBNode/1_child-12       161.9n ± 1%   220.8n ± 1%   +36.38% (p=0.000 n=10)
_EncodeDBNode/2_children-12    235.8n ± 1%   297.9n ± 1%   +26.29% (p=0.000 n=10)
_EncodeDBNode/16_children-12   1.400µ ± 1%   1.579µ ± 1%   +12.79% (p=0.000 n=10)
geomean                        136.9n        222.0n        +62.14%

                             │  old.txt   │                new.txt                │
                             │    B/op    │    B/op      vs base                  │
_EncodeDBNode/empty_node-12    2.000 ± 0%   26.000 ± 0%  +1200.00% (p=0.000 n=10)
_EncodeDBNode/has_value-12     8.000 ± 0%   32.000 ± 0%   +300.00% (p=0.000 n=10)
_EncodeDBNode/1_child-12       48.00 ± 0%    72.00 ± 0%    +50.00% (p=0.000 n=10)
_EncodeDBNode/2_children-12    96.00 ± 0%   120.00 ± 0%    +25.00% (p=0.000 n=10)
_EncodeDBNode/16_children-12   640.0 ± 0%    664.0 ± 0%     +3.75% (p=0.000 n=10)
geomean                        34.26         86.25        +151.77%

                             │  old.txt   │               new.txt               │
                             │ allocs/op  │ allocs/op   vs base                 │
_EncodeDBNode/empty_node-12    1.000 ± 0%   2.000 ± 0%  +100.00% (p=0.000 n=10)
_EncodeDBNode/has_value-12     1.000 ± 0%   2.000 ± 0%  +100.00% (p=0.000 n=10)
_EncodeDBNode/1_child-12       1.000 ± 0%   2.000 ± 0%  +100.00% (p=0.000 n=10)
_EncodeDBNode/2_children-12    1.000 ± 0%   2.000 ± 0%  +100.00% (p=0.000 n=10)
_EncodeDBNode/16_children-12   1.000 ± 0%   2.000 ± 0%  +100.00% (p=0.000 n=10)
geomean                        1.000        2.000       +100.00%

Benchmark_DecodeDBNode:

goos: darwin
goarch: arm64
pkg: github.com/ava-labs/avalanchego/x/merkledb
                             │   old.txt   │               new.txt               │
                             │   sec/op    │   sec/op     vs base                │
_DecodeDBNode/empty_node-12    255.8n ± 2%   308.0n ± 2%  +20.41% (p=0.000 n=10)
_DecodeDBNode/has_value-12     290.8n ± 1%   350.6n ± 1%  +20.60% (p=0.000 n=10)
_DecodeDBNode/1_child-12       500.6n ± 2%   588.9n ± 2%  +17.65% (p=0.000 n=10)
_DecodeDBNode/2_children-12    642.5n ± 3%   746.2n ± 1%  +16.14% (p=0.000 n=10)
_DecodeDBNode/16_children-12   2.467µ ± 3%   3.193µ ± 6%  +29.41% (p=0.000 n=10)
geomean                        567.8n        685.6n       +20.76%

                             │   old.txt    │               new.txt                │
                             │     B/op     │     B/op      vs base                │
_DecodeDBNode/empty_node-12      48.00 ± 0%     80.00 ± 0%  +66.67% (p=0.000 n=10)
_DecodeDBNode/has_value-12       56.00 ± 0%     88.00 ± 0%  +57.14% (p=0.000 n=10)
_DecodeDBNode/1_child-12         216.0 ± 0%     264.0 ± 0%  +22.22% (p=0.000 n=10)
_DecodeDBNode/2_children-12      288.0 ± 0%     352.0 ± 0%  +22.22% (p=0.000 n=10)
_DecodeDBNode/16_children-12   1.400Ki ± 0%   1.682Ki ± 0%  +20.08% (p=0.000 n=10)
geomean                          188.8          257.2       +36.27%

                             │  old.txt   │               new.txt               │
                             │ allocs/op  │ allocs/op   vs base                 │
_DecodeDBNode/empty_node-12    1.000 ± 0%   2.000 ± 0%  +100.00% (p=0.000 n=10)
_DecodeDBNode/has_value-12     2.000 ± 0%   3.000 ± 0%   +50.00% (p=0.000 n=10)
_DecodeDBNode/1_child-12       4.000 ± 0%   6.000 ± 0%   +50.00% (p=0.000 n=10)
_DecodeDBNode/2_children-12    6.000 ± 0%   9.000 ± 0%   +50.00% (p=0.000 n=10)
_DecodeDBNode/16_children-12   19.00 ± 0%   36.00 ± 0%   +89.47% (p=0.000 n=10)
geomean                        3.908        6.507        +66.48%

Benchmark_SHA256_HashNode:

goos: darwin
goarch: arm64
pkg: github.com/ava-labs/avalanchego/x/merkledb
                                │   old.txt   │               new.txt                │
                                │   sec/op    │    sec/op     vs base                │
_SHA256_HashNode/empty_node-12    70.15n ± 2%   109.95n ± 2%  +56.74% (p=0.000 n=10)
_SHA256_HashNode/has_value-12     83.68n ± 1%   122.70n ± 1%  +46.62% (p=0.000 n=10)
_SHA256_HashNode/has_key-12       71.64n ± 1%   112.85n ± 1%  +57.51% (p=0.000 n=10)
_SHA256_HashNode/1_child-12       140.5n ± 1%    182.9n ± 1%  +30.14% (p=0.000 n=10)
_SHA256_HashNode/2_children-12    199.6n ± 2%    247.4n ± 1%  +23.97% (p=0.000 n=10)
_SHA256_HashNode/16_children-12   981.7n ± 0%   1076.0n ± 2%   +9.61% (p=0.000 n=10)
geomean                           150.4n         205.0n       +36.26%

                                │   old.txt   │           new.txt            │
                                │    B/op     │    B/op     vs base          │
_SHA256_HashNode/empty_node-12    0.00 ± 0%     32.00 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/has_value-12     0.00 ± 0%     32.00 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/has_key-12       0.00 ± 0%     32.00 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/1_child-12       0.00 ± 0%     32.00 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/2_children-12    0.00 ± 0%     32.00 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/16_children-12   0.00 ± 0%     32.00 ± 0%  ? (p=0.000 n=10)
geomean                                     ¹   32.00       ?
¹ summaries must be >0 to compute geomean

                                │   old.txt    │           new.txt            │
                                │  allocs/op   │ allocs/op   vs base          │
_SHA256_HashNode/empty_node-12    0.000 ± 0%     1.000 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/has_value-12     0.000 ± 0%     1.000 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/has_key-12       0.000 ± 0%     1.000 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/1_child-12       0.000 ± 0%     1.000 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/2_children-12    0.000 ± 0%     1.000 ± 0%  ? (p=0.000 n=10)
_SHA256_HashNode/16_children-12   0.000 ± 0%     1.000 ± 0%  ? (p=0.000 n=10)
geomean                                      ¹   1.000       ?
¹ summaries must be >0 to compute geomean

Benchmark_HashChangedNodes:

goos: darwin
goarch: arm64
pkg: github.com/ava-labs/avalanchego/x/merkledb
                            │   old.txt   │               new.txt                │
                            │   sec/op    │    sec/op     vs base                │
_HashChangedNodes/1-12        320.5n ± 1%   367.5n ±  3%  +14.65% (p=0.000 n=10)
_HashChangedNodes/10-12       2.656µ ± 1%   3.082µ ±  1%  +16.04% (p=0.000 n=10)
_HashChangedNodes/100-12      26.93µ ± 1%   30.97µ ±  1%  +15.00% (p=0.000 n=10)
_HashChangedNodes/1000-12     296.3µ ± 1%   365.4µ ±  1%  +23.29% (p=0.000 n=10)
_HashChangedNodes/10000-12    3.034m ± 3%   4.537m ±  1%  +49.51% (p=0.000 n=10)
_HashChangedNodes/100000-12   56.56m ± 5%   72.08m ± 11%  +27.44% (p=0.000 n=10)
geomean                       102.6µ        127.0µ        +23.76%

                            │  old.txt   │                   new.txt                   │
                            │    B/op    │      B/op       vs base                     │
_HashChangedNodes/1-12        144.0 ± 0%       176.0 ± 0%       +22.22% (p=0.000 n=10)
_HashChangedNodes/10-12       144.0 ± 0%       496.0 ± 0%      +244.44% (p=0.000 n=10)
_HashChangedNodes/100-12      144.0 ± 0%      3600.0 ± 0%     +2400.00% (p=0.000 n=10)
_HashChangedNodes/1000-12     144.0 ± 0%     36784.0 ± 0%    +25444.44% (p=0.000 n=10)
_HashChangedNodes/10000-12    144.0 ± 0%    345265.0 ± 0%   +239667.36% (p=0.000 n=10)
_HashChangedNodes/100000-12   144.0 ± 0%   3794608.0 ± 0%  +2635044.44% (p=0.000 n=10)
geomean                       144.0          15.36Ki         +10823.20%

                            │  old.txt   │                   new.txt                    │
                            │ allocs/op  │    allocs/op     vs base                     │
_HashChangedNodes/1-12        3.000 ± 0%        4.000 ± 0%       +33.33% (p=0.000 n=10)
_HashChangedNodes/10-12       3.000 ± 0%       14.000 ± 0%      +366.67% (p=0.000 n=10)
_HashChangedNodes/100-12      3.000 ± 0%      111.000 ± 0%     +3600.00% (p=0.000 n=10)
_HashChangedNodes/1000-12     3.000 ± 0%     1148.000 ± 0%    +38166.67% (p=0.000 n=10)
_HashChangedNodes/10000-12    3.000 ± 0%    10788.000 ± 0%   +359500.00% (p=0.000 n=10)
_HashChangedNodes/100000-12   3.000 ± 0%   118580.000 ± 0%  +3952566.67% (p=0.000 n=10)
geomean                       3.000             457.2         +15138.64%

@StephenButtolph
Copy link
Contributor

Benchmark_HashChangedNodes Is really the e2e hashing benchmark, which seems to be +23.76%

@darioush
Copy link
Contributor Author

darioush commented Jul 5, 2024 

goos: darwin
goarch: arm64
pkg: github.com/ava-labs/avalanchego/x/merkledb
                            │   old.txt   │               new.txt               │
                            │   sec/op    │   sec/op     vs base                │
_HashChangedNodes/1-12        182.9n ± 1%   206.8n ± 0%  +13.07% (p=0.000 n=10)
_HashChangedNodes/10-12       1.911µ ± 0%   2.122µ ± 2%  +11.04% (p=0.000 n=10)
_HashChangedNodes/100-12      19.78µ ± 0%   21.22µ ± 0%   +7.27% (p=0.000 n=10)
_HashChangedNodes/1000-12     229.1µ ± 2%   246.9µ ± 2%   +7.79% (p=0.000 n=10)
_HashChangedNodes/10000-12    2.324m ± 1%   2.512m ± 1%   +8.08% (p=0.000 n=10)
_HashChangedNodes/100000-12   42.72m ± 4%   40.89m ± 2%   -4.28% (p=0.000 n=10)
geomean                       73.47µ        78.62µ        +7.01%

                            │  old.txt   │               new.txt               │
                            │    B/op    │    B/op     vs base                 │
_HashChangedNodes/1-12        144.0 ± 0%   144.0 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/10-12       144.0 ± 0%   144.0 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/100-12      144.0 ± 0%   144.0 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/1000-12     144.0 ± 0%   144.0 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/10000-12    144.0 ± 0%   144.0 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/100000-12   144.0 ± 0%   144.0 ± 0%       ~ (p=1.000 n=10) ¹
geomean                       144.0        144.0       +0.00%
¹ all samples are equal

                            │  old.txt   │               new.txt               │
                            │ allocs/op  │ allocs/op   vs base                 │
_HashChangedNodes/1-12        3.000 ± 0%   3.000 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/10-12       3.000 ± 0%   3.000 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/100-12      3.000 ± 0%   3.000 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/1000-12     3.000 ± 0%   3.000 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/10000-12    3.000 ± 0%   3.000 ± 0%       ~ (p=1.000 n=10) ¹
_HashChangedNodes/100000-12   3.000 ± 0%   3.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                       3.000        3.000       +0.00%
¹ all samples are equal

@darioush
Copy link
Contributor Author

darioush commented Jul 5, 2024

trying to minimize allocs ^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@StephenButtolph StephenButtolph Awaiting requested review from StephenButtolph StephenButtolph will be requested when the pull request is marked ready for review StephenButtolph is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
Platform Engineering Group
Status: Backlog 🗄️
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@darioush @StephenButtolph