Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Support create temporary function for native hive udf #6829

Merged
merged 6 commits into from
Aug 25, 2024

Conversation

marin-ma
Copy link
Contributor

@marin-ma marin-ma commented Aug 14, 2024

  1. Create a native Velox udf, name it to the full name of the hive udf (package + classname) e.g. org.apache.spark.sql.hive.execution.UDFStringString
  2. Register the hive udf by create temporary function with an arbitrary function name. e.g. create temporary function hive_string_string as 'org.apache.spark.sql.hive.execution.UDFStringString'
  3. Execute the sql with the temporary function. The execution will be offloaded to the native hive udf if validation passed, or fallback to the original Java Hive udf.

@github-actions github-actions bot added CORE works for Gluten Core VELOX labels Aug 14, 2024
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link

Run Gluten Clickhouse CI

2 similar comments
Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

1 similar comment
@marin-ma
Copy link
Contributor Author

Run Gluten Clickhouse CI

@FelixYBW
Copy link
Contributor

Works perfectly! Thank you.

Can you update the document as well?

@marin-ma
Copy link
Contributor Author

Can you update the document as well?

@FelixYBW Will update in the next pr with removing the native registration.

@marin-ma marin-ma merged commit d4d7241 into apache:main Aug 25, 2024
45 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_08_25_2024_time.csv log/native_master_08_24_2024_a575395a60_time.csv difference percentage
q1 13.90 13.95 0.044 100.32%
q2 14.48 14.11 -0.373 97.43%
q3 2.87 2.47 -0.407 85.84%
q4 72.17 70.88 -1.292 98.21%
q5 9.02 9.27 0.251 102.79%
q6 4.12 3.92 -0.197 95.22%
q7 6.60 6.53 -0.069 98.95%
q8 5.23 5.48 0.245 104.68%
q9 24.75 24.00 -0.751 96.97%
q10 8.47 9.85 1.379 116.29%
q11 39.05 38.36 -0.691 98.23%
q12 1.33 1.29 -0.041 96.95%
q13 6.54 6.45 -0.085 98.71%
q14a 45.29 44.95 -0.345 99.24%
q14b 44.33 43.69 -0.643 98.55%
q15 2.70 2.53 -0.168 93.77%
q16 47.27 45.47 -1.800 96.19%
q17 5.33 4.93 -0.396 92.56%
q18 6.87 7.35 0.480 106.98%
q19 2.31 2.54 0.226 109.77%
q20 1.58 1.47 -0.111 92.96%
q21 1.26 1.03 -0.225 82.13%
q22 7.77 7.65 -0.125 98.39%
q23a 103.18 103.70 0.527 100.51%
q23b 130.28 127.16 -3.120 97.60%
q24a 116.35 109.84 -6.518 94.40%
q24b 110.84 111.26 0.413 100.37%
q25 4.27 4.57 0.309 107.24%
q26 3.13 3.31 0.189 106.04%
q27 4.04 4.09 0.051 101.26%
q28 34.34 29.61 -4.727 86.23%
q29 11.79 9.46 -2.330 80.23%
q30 5.28 5.10 -0.182 96.55%
q31 7.74 7.26 -0.484 93.75%
q32 1.23 1.20 -0.028 97.72%
q33 4.19 4.26 0.070 101.68%
q34 4.10 4.01 -0.084 97.96%
q35 10.54 7.79 -2.756 73.86%
q36 4.37 4.66 0.285 106.52%
q37 4.68 4.58 -0.099 97.89%
q38 14.28 15.16 0.879 106.16%
q39a 3.70 3.08 -0.616 83.34%
q39b 3.09 2.70 -0.392 87.32%
q40 3.90 3.91 0.009 100.23%
q41 0.68 0.62 -0.063 90.72%
q42 0.96 0.88 -0.084 91.24%
q43 4.78 4.86 0.070 101.47%
q44 10.08 9.77 -0.311 96.92%
q45 3.31 3.24 -0.063 98.09%
q46 3.98 3.72 -0.260 93.46%
q47 18.69 18.69 0.004 100.02%
q48 5.21 5.25 0.042 100.80%
q49 8.84 8.53 -0.312 96.47%
q50 21.41 21.80 0.391 101.82%
q51 10.15 9.48 -0.668 93.42%
q52 1.03 1.17 0.144 114.06%
q53 2.68 2.38 -0.299 88.84%
q54 4.26 3.94 -0.313 92.65%
q55 1.08 1.11 0.032 102.93%
q56 4.05 4.07 0.017 100.42%
q57 10.94 11.34 0.398 103.64%
q58 2.64 2.50 -0.141 94.68%
q59 10.82 11.00 0.179 101.66%
q60 4.14 4.17 0.024 100.58%
q61 4.17 4.18 0.007 100.16%
q62 4.59 4.61 0.022 100.48%
q63 2.42 2.36 -0.062 97.44%
q64 59.74 64.88 5.134 108.59%
q65 17.20 18.25 1.051 106.11%
q66 4.11 4.08 -0.034 99.18%
q67 410.92 392.57 -18.348 95.53%
q68 3.70 3.62 -0.082 97.77%
q69 5.58 5.08 -0.498 91.08%
q70 11.88 11.83 -0.049 99.59%
q71 2.68 2.47 -0.212 92.08%
q72 215.16 218.07 2.909 101.35%
q73 2.24 2.21 -0.031 98.63%
q74 23.90 23.84 -0.058 99.76%
q75 26.47 26.98 0.512 101.94%
q76 11.94 11.72 -0.223 98.13%
q77 2.41 2.33 -0.081 96.63%
q78 49.66 50.33 0.670 101.35%
q79 3.90 4.02 0.115 102.95%
q80 12.78 12.44 -0.339 97.34%
q81 5.08 4.88 -0.195 96.17%
q82 7.21 7.08 -0.131 98.19%
q83 1.68 1.64 -0.040 97.61%
q84 2.84 2.76 -0.076 97.33%
q85 8.07 7.92 -0.146 98.19%
q86 4.01 4.17 0.158 103.93%
q87 14.61 14.16 -0.449 96.93%
q88 22.02 21.67 -0.350 98.41%
q89 3.78 3.70 -0.078 97.94%
q90 3.20 3.17 -0.031 99.02%
q91 2.55 2.50 -0.052 97.97%
q92 1.37 1.36 -0.015 98.92%
q93 39.97 39.80 -0.174 99.56%
q94 24.67 24.61 -0.060 99.76%
q9 89.98 88.93 -1.046 98.84%
q5 3.18 2.86 -0.317 90.00%
q96 17.86 17.64 -0.220 98.77%
q97 1.94 1.85 -0.095 95.11%
q98 10.89 10.11 -0.776 92.88%
q99 10.89 10.11 -0.776 92.88%
total 2228.71 2190.11 -38.600 98.27%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_08_25_2024_time.csv log/native_master_08_24_2024_a575395a60_time.csv difference percentage
q1 40.89 40.07 -0.815 98.01%
q2 30.28 30.09 -0.196 99.35%
q3 53.78 53.28 -0.497 99.08%
q4 42.76 43.54 0.788 101.84%
q5 103.48 105.20 1.722 101.66%
q6 12.15 12.68 0.533 104.39%
q7 119.11 114.61 -4.500 96.22%
q8 115.93 115.07 -0.861 99.26%
q9 169.87 169.34 -0.531 99.69%
q10 65.36 65.30 -0.066 99.90%
q11 27.09 27.88 0.790 102.92%
q12 29.91 28.48 -1.428 95.23%
q13 51.30 51.26 -0.040 99.92%
q14 27.61 25.24 -2.367 91.43%
q15 55.28 55.93 0.657 101.19%
q16 18.91 17.80 -1.110 94.13%
q17 133.84 131.59 -2.253 98.32%
q18 197.60 201.19 3.587 101.82%
q19 27.69 26.05 -1.640 94.08%
q20 43.29 41.12 -2.176 94.97%
q21 384.30 384.37 0.075 100.02%
q22 17.74 15.25 -2.487 85.98%
total 1768.18 1755.36 -12.815 99.28%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CORE works for Gluten Core VELOX
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants