Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a split by constant to the datafusion opener #1995

Merged
merged 3 commits into from
Jan 17, 2025

Conversation

joseph-isaacs
Copy link
Member

No description provided.

@joseph-isaacs joseph-isaacs force-pushed the ji/create-large-split-on-read branch from 36caccb to abc952f Compare January 17, 2025 14:10
@joseph-isaacs joseph-isaacs added the benchmark Run benchmarks on this branch label Jan 17, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Jan 17, 2025
Copy link
Contributor

Benchmarks: TPC-H

Table of Results
name PR 7551a7c base b18e1dc ratio (PR/base) unit
tpch_q01/arrow 557190482 5.49388e+08 1.0142 ns
tpch_q01/parquet 779324722 7.59197e+08 1.02651 ns
tpch_q01/vortex-file-compressed 532171045 5.03753e+08 1.05641 ns
tpch_q02/arrow 140953835 1.41255e+08 0.997867 ns
tpch_q02/parquet 175026404 1.75132e+08 0.999399 ns
tpch_q02/vortex-file-compressed 147789313 1.40731e+08 1.05015 ns
tpch_q03/arrow 178010881 1.73033e+08 1.02877 ns
tpch_q03/parquet 389018347 3.76372e+08 1.0336 ns
tpch_q03/vortex-file-compressed 259075674 2.27004e+08 1.14128 ns
tpch_q04/arrow 179138748 1.74033e+08 1.02934 ns
tpch_q04/parquet 220341198 2.17827e+08 1.01154 ns
tpch_q04/vortex-file-compressed 205575509 1.81632e+08 1.13182 ns
tpch_q05/arrow 330782306 3.16257e+08 1.04593 ns
tpch_q05/parquet 509852180 4.85737e+08 1.04965 ns
tpch_q05/vortex-file-compressed 372323586 3.57107e+08 1.04261 ns
tpch_q06/arrow 26115096 2.44588e+07 1.06772 ns
tpch_q06/parquet 151860788 1.47863e+08 1.02704 ns
tpch_q06/vortex-file-compressed 96364762 6.98004e+07 1.38058 ns
tpch_q07/arrow 620871431 6.21599e+08 0.998829 ns
tpch_q07/parquet 775225275 7.52926e+08 1.02962 ns
tpch_q07/vortex-file-compressed 664733404 6.25153e+08 1.06331 ns
tpch_q08/arrow 276297428 2.63023e+08 1.05047 ns
tpch_q08/parquet 549848671 5.41925e+08 1.01462 ns
tpch_q08/vortex-file-compressed 364346011 3.35115e+08 1.08723 ns
tpch_q09/arrow 484255540 4.74061e+08 1.02151 ns
tpch_q09/parquet 793715269 7.58355e+08 1.04663 ns
tpch_q09/vortex-file-compressed 595474177 5.71515e+08 1.04192 ns
tpch_q10/arrow 272220467 2.61035e+08 1.04285 ns
tpch_q10/parquet 507233424 5.07448e+08 0.999577 ns
tpch_q10/vortex-file-compressed 301893422 2.85982e+08 1.05564 ns
tpch_q11/arrow 136524948 1.38302e+08 0.98715 ns
tpch_q11/parquet 147614229 1.46051e+08 1.0107 ns
tpch_q11/vortex-file-compressed 131059355 1.28091e+08 1.02317 ns
tpch_q12/arrow 184721770 1.80475e+08 1.02353 ns
tpch_q12/parquet 336510134 3.23373e+08 1.04063 ns
tpch_q12/vortex-file-compressed 278811361 2.74054e+08 1.01736 ns
tpch_q13/arrow 169285557 1.62546e+08 1.04146 ns
tpch_q13/parquet 309036590 2.95806e+08 1.04473 ns
tpch_q13/vortex-file-compressed 192126687 1.77001e+08 1.08545 ns
tpch_q14/arrow 36659017 3.5699e+07 1.02689 ns
tpch_q14/parquet 233831678 2.34873e+08 0.995565 ns
tpch_q14/vortex-file-compressed 102937582 8.78606e+07 1.1716 ns
tpch_q15/arrow 73086216 6.50878e+07 1.12289 ns
tpch_q15/parquet 331455795 3.17974e+08 1.0424 ns
tpch_q15/vortex-file-compressed 178544144 1.46883e+08 1.21555 ns
tpch_q16/arrow 97636486 9.69282e+07 1.00731 ns
tpch_q16/parquet 113223034 1.12332e+08 1.00793 ns
tpch_q16/vortex-file-compressed 105524448 1.02083e+08 1.03372 ns
tpch_q17/arrow 631237789 5.97869e+08 1.05581 ns
tpch_q17/parquet 677465074 6.88035e+08 0.984637 ns
tpch_q17/vortex-file-compressed 580439059 5.96062e+08 0.97379 ns
tpch_q18/arrow 1141950861 1.10576e+09 1.03273 ns
tpch_q18/parquet 1341571593 1.30709e+09 1.02638 ns
tpch_q18/vortex-file-compressed 1164726540 1.13947e+09 1.02216 ns
tpch_q19/arrow 155678364 1.50865e+08 1.03191 ns
tpch_q19/parquet 424064091 4.15398e+08 1.02086 ns
tpch_q19/vortex-file-compressed 209559010 1.62702e+08 1.28799 ns
tpch_q20/arrow 180577338 1.72817e+08 1.04491 ns
tpch_q20/parquet 317094847 3.15017e+08 1.0066 ns
tpch_q20/vortex-file-compressed 240165419 2.09762e+08 1.14494 ns
tpch_q21/arrow 980126404 9.79017e+08 1.00113 ns
tpch_q21/parquet 1115535995 1.08338e+09 1.02968 ns
tpch_q21/vortex-file-compressed 1048214487 9.72092e+08 1.07831 ns
tpch_q22/arrow 77707290 7.70116e+07 1.00903 ns
tpch_q22/parquet 109755284 1.06453e+08 1.03102 ns
tpch_q22/vortex-file-compressed 83814617 8.13618e+07 1.03015 ns

Copy link
Contributor

Benchmarks: random_access

Table of Results
name PR 7551a7c base b18e1dc ratio (PR/base) unit
random-access/vortex-tokio-local-disk 2.27562e+06 2.28984e+06 0.993791 ns
random-access/vortex-local-fs 2.44962e+06 2.48018e+06 0.987679 ns
random-access/parquet-tokio-local-disk 2.1927e+08 2.17281e+08 1.00915 ns

Copy link
Contributor

Benchmarks: datafusion

Table of Results
name PR 7551a7c base b18e1dc ratio (PR/base) unit
arrow/planning 944740 945299 0.999409 ns
arrow/exec 1.9682e+06 2.00799e+06 0.980182 ns
vortex-pushdown-compressed/planning 581303 587123 0.990087 ns
vortex-pushdown-compressed/exec 2.71985e+06 2.69972e+06 1.00745 ns
vortex-pushdown-uncompressed/planning 584494 587501 0.994882 ns
vortex-pushdown-uncompressed/exec 1.56959e+06 1.55063e+06 1.01223 ns
vortex-nopushdown-compressed/planning 949289 947849 1.00152 ns
vortex-nopushdown-compressed/exec 3.16956e+06 3.11015e+06 1.0191 ns
vortex-nopushdown-uncompressed/planning 952030 947238 1.00506 ns
vortex-nopushdown-uncompressed/exec 5.10927e+06 5.18186e+06 0.985991 ns

@joseph-isaacs joseph-isaacs merged commit 97c9f35 into develop Jan 17, 2025
21 checks passed
@joseph-isaacs joseph-isaacs deleted the ji/create-large-split-on-read branch January 17, 2025 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants