Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-6887][VL] Daily Update Velox Version (2024_10_03) #7406

Merged
merged 5 commits into from
Oct 8, 2024
Merged

[GLUTEN-6887][VL] Daily Update Velox Version (2024_10_03) #7406

merged 5 commits into from
Oct 8, 2024
+7 −2

Conversation

GlutenPerfBot
Copy link
Contributor

Upstream Velox's New Commits:

b6a7cd260 by aditi-pandit, Make PlanNode enum to string conversions code consistent (11144)
0dc5609cf by Jimmy Lu, Add sorted indices as output parameter for hasOverlappingRanges (11148)
71e1ca722 by Jialiang Tan, Don't reclaim from a hash build/probe operators if it has exceeded max spill level limit (11123)
181a3138c by Orri Erling, Add basic group by and status returning without unified page faults (10944)
532a54898 by Jimmy Lu, Handle missing fields when read flatmap as struct (11106)
65f7b5a27 by Brady Kilpatrick, Revert D63554289: Support custom comparison in Variants
3e9dfd118 by Brady Kilpatrick, Revert D63565430: Support custom comparison in Histogram Aggregation
813cfb874 by Brady Kilpatrick, Revert D63570186: Support custom comparison in Map Aggregations
003721b28 by Brady Kilpatrick, Revert D63645720: Support custom comparison in Multi Map Aggregation
f94aa6a92 by MacVincent Agha-Oko, Refactor Confusing Dwrf Reader RowNumber Method (11029)
bd2b072ae by mwish, Typo fix in TreeOfLosers.h (11125)
bfb8ebe92 by Xiaoxuan Meng, Remove the use task_bucketed_writer_count config (11138)
7ad64b337 by Kevin Wilfong, Support custom comparison in Multi Map Aggregation (11133)
811fbde65 by Daniel Hunte, update slice funtion to take BIGINT (11113)

[GLUTEN-6887][VL] Daily Update Velox Version (2024_10_03)
8434a33 
Signed-off-by: glutenperfbot <glutenperfbot@intel-internal.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 2, 2024

#6887

@FelixYBW
Copy link
Contributor

FelixYBW commented Oct 4, 2024 •
edited
Loading

error caused by PR: 532a54898 by Jimmy Lu, Handle missing fields when read flatmap as struct (11106)

2024-10-03T22:21:30.6316503Z - vectorized reader: missing all struct fields *** FAILED ***
2024-10-03T22:21:30.6316614Z   Results do not match for query:
2024-10-03T22:21:30.6318573Z   Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
2024-10-03T22:21:30.6318786Z   Timezone Env:
2024-10-03T22:21:30.6318870Z
2024-10-03T22:21:30.6318972Z   == Parsed Logical Plan ==
2024-10-03T22:21:30.6319069Z   Relation [_1#718302] parquet
2024-10-03T22:21:30.6319153Z
2024-10-03T22:21:30.6319253Z   == Analyzed Logical Plan ==
2024-10-03T22:21:30.6319354Z   _1: struct<_3:int,_4:bigint>
2024-10-03T22:21:30.6319447Z   Relation [_1#718302] parquet
2024-10-03T22:21:30.6319529Z
2024-10-03T22:21:30.6319628Z   == Optimized Logical Plan ==
2024-10-03T22:21:30.6319722Z   Relation [_1#718302] parquet
2024-10-03T22:21:30.6319803Z
2024-10-03T22:21:30.6319897Z   == Physical Plan ==
2024-10-03T22:21:30.6320003Z   VeloxColumnarToRowExec
2024-10-03T22:21:30.6321227Z   +- ^(24203) NativeFileScan parquet [_1#718302] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/spark-93a54e63-7b43-434c-bda1-04f72a3b4f1e], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_1:struct<_3:int,_4:bigint>>
2024-10-03T22:21:30.6321309Z
2024-10-03T22:21:30.6321402Z   == Results ==
2024-10-03T22:21:30.6321477Z
2024-10-03T22:21:30.6321568Z   == Results ==
2024-10-03T22:21:30.6321769Z   !== Correct Answer - 3 ==   == Spark Answer - 3 ==
2024-10-03T22:21:30.6321940Z   !struct<>                   struct<_1:struct<_3:int,_4:bigint>>
2024-10-03T22:21:30.6322061Z   ![null]                     [[null,null]]
2024-10-03T22:21:30.6322170Z   ![null]                     [[null,null]]
2024-10-03T22:21:30.6322344Z   ![null]                     [[null,null]] (QueryTest.scala:244)

@github-actions github-actions bot added the CORE works for Gluten Core label Oct 8, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 8, 2024

Run Gluten Clickhouse CI

@zhouyuan
fix ut
53524c9 
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 8, 2024

Run Gluten Clickhouse CI

@zhouyuan zhouyuan merged commit 910cb58 into apache:main Oct 8, 2024
46 checks passed
@GlutenPerfBot
Copy link
Contributor Author

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_10_08_2024_time.csv log/native_master_10_07_2024_3ca5f778f6_time.csv difference percentage
q1 14.17 14.37 0.195 101.38%
q2 14.79 15.32 0.533 103.60%
q3 3.19 4.81 1.625 150.96%
q4 71.98 71.77 -0.209 99.71%
q5 10.62 10.19 -0.430 95.95%
q6 3.87 2.37 -1.493 61.39%
q7 6.18 6.00 -0.182 97.06%
q8 3.30 5.30 1.999 160.58%
q9 26.66 24.21 -2.455 90.79%
q10 8.50 8.33 -0.170 97.99%
q11 36.15 36.96 0.807 102.23%
q12 1.38 1.45 0.064 104.60%
q13 6.56 6.22 -0.339 94.84%
q14a 47.56 47.79 0.227 100.48%
q14b 40.73 41.06 0.330 100.81%
q15 2.44 2.56 0.117 104.79%
q16 47.06 47.24 0.180 100.38%
q17 4.73 4.65 -0.075 98.41%
q18 6.57 6.64 0.076 101.16%
q19 2.00 3.30 1.298 164.88%
q20 1.40 2.35 0.953 168.12%
q21 2.39 1.39 -1.000 58.12%
q22 7.97 7.81 -0.160 98.00%
q23a 106.97 103.15 -3.826 96.42%
q23b 125.24 125.99 0.745 100.60%
q24a 107.36 110.95 3.583 103.34%
q24b 114.79 112.76 -2.033 98.23%
q25 4.22 4.07 -0.150 96.45%
q26 4.07 4.18 0.116 102.86%
q27 5.17 4.69 -0.479 90.74%
q28 32.06 31.99 -0.070 99.78%
q29 11.68 10.47 -1.209 89.65%
q30 5.05 5.41 0.361 107.15%
q31 7.41 6.74 -0.676 90.88%
q32 1.26 1.24 -0.022 98.23%
q33 4.48 4.17 -0.310 93.09%
q34 3.82 3.90 0.089 102.33%
q35 9.56 7.61 -1.945 79.65%
q36 5.03 5.01 -0.014 99.72%
q37 4.75 4.73 -0.019 99.61%
q38 13.31 13.41 0.097 100.73%
q39a 3.16 3.06 -0.103 96.73%
q39b 3.03 3.08 0.048 101.60%
q40 3.97 3.80 -0.175 95.60%
q41 0.63 0.62 -0.014 97.87%
q42 1.00 0.85 -0.156 84.45%
q43 4.50 4.53 0.034 100.75%
q44 9.37 9.27 -0.093 99.01%
q45 3.12 3.31 0.191 106.13%
q46 3.62 3.68 0.053 101.47%
q47 17.58 18.93 1.355 107.71%
q48 4.89 5.05 0.161 103.29%
q49 6.95 6.89 -0.054 99.23%
q50 21.75 21.43 -0.323 98.51%
q51 9.59 9.92 0.333 103.47%
q52 1.11 1.03 -0.083 92.53%
q53 2.17 2.03 -0.143 93.43%
q54 3.59 3.59 0.006 100.17%
q55 1.04 0.99 -0.047 95.44%
q56 4.04 4.02 -0.025 99.37%
q57 10.37 11.02 0.651 106.28%
q58 2.43 2.57 0.137 105.62%
q59 11.13 10.70 -0.426 96.17%
q60 3.99 3.96 -0.028 99.29%
q61 3.92 3.82 -0.106 97.31%
q62 4.31 4.36 0.053 101.22%
q63 2.23 2.16 -0.074 96.69%
q64 64.68 59.92 -4.760 92.64%
q65 18.38 20.20 1.820 109.90%
q66 4.17 4.35 0.185 104.43%
q67 416.07 407.28 -8.799 97.89%
q68 3.77 3.38 -0.383 89.83%
q69 5.52 6.31 0.791 114.33%
q70 10.41 10.25 -0.159 98.47%
q71 2.20 2.31 0.107 104.84%
q72 224.01 218.43 -5.579 97.51%
q73 2.49 2.41 -0.084 96.64%
q74 22.92 22.72 -0.201 99.13%
q75 24.82 24.48 -0.343 98.62%
q76 12.06 11.19 -0.867 92.81%
q77 2.08 2.11 0.031 101.49%
q78 49.32 54.61 5.292 110.73%
q79 3.66 3.62 -0.038 98.95%
q80 11.33 11.65 0.318 102.81%
q81 4.57 4.59 0.019 100.42%
q82 7.08 7.39 0.314 104.44%
q83 1.60 1.64 0.039 102.41%
q84 3.72 2.78 -0.944 74.62%
q85 7.63 6.85 -0.784 89.72%
q86 4.21 4.15 -0.056 98.67%
q87 13.74 13.40 -0.337 97.54%
q88 17.00 17.72 0.726 104.27%
q89 3.16 2.93 -0.230 92.73%
q90 2.85 2.62 -0.228 92.01%
q91 2.07 2.16 0.095 104.60%
q92 1.21 1.18 -0.026 97.82%
q93 39.79 38.88 -0.907 97.72%
q94 25.51 26.20 0.687 102.69%
q9 89.20 88.55 -0.643 99.28%
q5 2.63 2.68 0.052 101.98%
q96 17.90 17.68 -0.225 98.74%
q97 1.83 1.95 0.127 106.95%
q98 11.49 11.30 -0.193 98.32%
q99 11.49 11.30 -0.193 98.32%
total 2215.04 2197.16 -17.879 99.19%

@GlutenPerfBot
Copy link
Contributor Author

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_10_08_2024_time.csv log/native_master_10_07_2024_3ca5f778f6_time.csv difference percentage
q1 52.48 52.01 -0.462 99.12%
q2 30.79 30.42 -0.362 98.82%
q3 52.77 54.19 1.428 102.71%
q4 43.11 42.19 -0.921 97.86%
q5 104.61 109.12 4.517 104.32%
q6 12.15 12.15 0.005 100.04%
q7 113.24 111.44 -1.807 98.40%
q8 115.80 114.50 -1.301 98.88%
q9 172.68 171.77 -0.911 99.47%
q10 68.84 67.48 -1.355 98.03%
q11 26.06 26.67 0.613 102.35%
q12 32.04 29.67 -2.375 92.59%
q13 51.81 51.51 -0.298 99.42%
q14 20.47 22.50 2.029 109.91%
q15 46.84 49.49 2.651 105.66%
q16 16.51 18.11 1.596 109.67%
q17 124.60 127.13 2.533 102.03%
q18 198.54 200.81 2.264 101.14%
q19 34.47 27.10 -7.365 78.63%
q20 39.32 40.47 1.157 102.94%
q21 328.66 332.05 3.392 101.03%
q22 16.00 15.90 -0.095 99.40%
total 1701.76 1706.69 4.932 100.29%

@FelixYBW FelixYBW mentioned this pull request Oct 8, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhouyuan zhouyuan zhouyuan approved these changes

Assignees
No one assigned
Labels
BUILD CORE works for Gluten Core VELOX
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@GlutenPerfBot @FelixYBW @zhouyuan