Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datastore: revamp bench suite #1733

Merged
merged 5 commits into from
Mar 30, 2023
Merged

Datastore: revamp bench suite #1733

merged 5 commits into from
Mar 30, 2023

Conversation

teh-cmc
Copy link
Member

@teh-cmc teh-cmc commented Mar 29, 2023

This reworks the datastore benchmark suite to actually cover all of the important stuff, and make it scalable to more parameters in the future (without breaking existing graphs).
This is a prerequisite for #1619 and in particular #1727 and its upcoming follow ups.

Needs to be merged in separately as it breaks existing benchmarks.


Benchmark suite Current: 29626cd
datastore/num_rows=1000/num_instances=1000/packed=false/insert/default 12625677 ns/iter (± 607274)
datastore/num_rows=1000/num_instances=1000/packed=false/insert/bucketsz=0 14646870 ns/iter (± 1133610)
datastore/num_rows=1000/num_instances=1000/packed=false/insert/bucketsz=2 13995839 ns/iter (± 782248)
datastore/num_rows=1000/num_instances=1000/packed=false/insert/bucketsz=32 13002161 ns/iter (± 984171)
datastore/num_rows=1000/num_instances=1000/packed=false/insert/bucketsz=2048 12813783 ns/iter (± 906175)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/default 12206704 ns/iter (± 599724)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/bucketsz=0 14043694 ns/iter (± 609431)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/bucketsz=2 13270571 ns/iter (± 772281)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/bucketsz=32 11689675 ns/iter (± 452193)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/bucketsz=2048 11165205 ns/iter (± 474775)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/default 1856 ns/iter (± 20)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/bucketsz=0 1873 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/bucketsz=2 1865 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/bucketsz=32 1867 ns/iter (± 2)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/bucketsz=2048 1854 ns/iter (± 33)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/default 1869 ns/iter (± 18)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/bucketsz=0 1877 ns/iter (± 2)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/bucketsz=2 1872 ns/iter (± 6)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/bucketsz=32 1885 ns/iter (± 3)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/bucketsz=2048 1883 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/default 286 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/default 438 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/bucketsz=0 281 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/bucketsz=0 444 ns/iter (± 6)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/bucketsz=2 280 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/bucketsz=2 443 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/bucketsz=32 282 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/bucketsz=32 443 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/bucketsz=2048 282 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/bucketsz=2048 439 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/default 281 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/default 437 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/bucketsz=0 280 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/bucketsz=0 444 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/bucketsz=2 281 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/bucketsz=2 447 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/bucketsz=32 281 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/bucketsz=32 444 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/bucketsz=2048 282 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/bucketsz=2048 438 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/range/default 12517856 ns/iter (± 924973)
datastore/num_rows=1000/num_instances=1000/packed=false/range/bucketsz=0 2216396 ns/iter (± 9037)
datastore/num_rows=1000/num_instances=1000/packed=false/range/bucketsz=2 2188885 ns/iter (± 10207)
datastore/num_rows=1000/num_instances=1000/packed=false/range/bucketsz=32 1957717 ns/iter (± 8345)
datastore/num_rows=1000/num_instances=1000/packed=false/range/bucketsz=2048 1864529 ns/iter (± 6538)
datastore/num_rows=1000/num_instances=1000/packed=true/range/default 12318758 ns/iter (± 392622)
datastore/num_rows=1000/num_instances=1000/packed=true/range/bucketsz=0 2212547 ns/iter (± 6209)
datastore/num_rows=1000/num_instances=1000/packed=true/range/bucketsz=2 2178050 ns/iter (± 7677)
datastore/num_rows=1000/num_instances=1000/packed=true/range/bucketsz=32 1920285 ns/iter (± 12660)
datastore/num_rows=1000/num_instances=1000/packed=true/range/bucketsz=2048 1857672 ns/iter (± 4359)

@teh-cmc teh-cmc added ⛃ re_datastore affects the datastore itself 🚀 performance Optimization, memory use, etc labels Mar 29, 2023
@teh-cmc teh-cmc marked this pull request as draft March 29, 2023 18:52
@teh-cmc teh-cmc marked this pull request as ready for review March 29, 2023 19:56
@emilk emilk self-requested a review March 30, 2023 08:27
@emilk emilk merged commit 680f791 into main Mar 30, 2023
@emilk emilk deleted the cmc/datastore/revamp_benchmarks branch March 30, 2023 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🚀 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants