Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [+] #270 array_sort functions working with lambdas #275

Merged

Conversation

eruizalo
Copy link
Collaborator

@eruizalo eruizalo commented Aug 29, 2022

Description

  • [feat] ⬆️ array sort function with a lambda of two params. Now we provide an api not available until spark 3.4 (only available through sql expressions).
  • [feat] ⬆️ array sort function with a lambda of one param. I have to thank @alfonsorr for this one, as it is his own idea, implementation. This functions is not available in any version of spark (yet).
  • [fix] 😳💣 !BREAKING_CHANGE: ⚠️ filterWIndex from now on will be only available for spark 3.0+ (we have being testing this function only for spark 3.0+ and the filter function for spark 2.4 does not allow a lambda with 2 variables, so it would fail)
  • [fix] 😳 exists array function have being tested only for spark 3.0+, but it is available for all spark versions. Now it will be tested for all versions
  • [test] ♻️ createLambda functions have being copied/pasted from spark code in order to test some lambda functions available in doric but not available in older versions of spark api

Related Issue and dependencies

How Has This Been Tested?

  • This pull request contains appropriate tests?:
    • YES

@eruizalo eruizalo requested a review from a team as a code owner August 29, 2022 17:24
@github-actions github-actions bot added spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3 labels Aug 29, 2022
@eruizalo eruizalo marked this pull request as draft August 29, 2022 17:27
@eruizalo eruizalo removed the spark_2.4 PR changes to spark 2.4 label Aug 30, 2022
@codecov
Copy link

codecov bot commented Aug 30, 2022

Codecov Report

Merging #275 (837704c) into main (3363221) will increase coverage by 0.04%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #275      +/-   ##
==========================================
+ Coverage   97.30%   97.34%   +0.04%     
==========================================
  Files          58       58              
  Lines        1113     1128      +15     
  Branches       19       19              
==========================================
+ Hits         1083     1098      +15     
  Misses         30       30              
Flag Coverage Δ
spark-2.4.x 94.85% <ø> (+1.42%) ⬆️
spark-3.0.x 96.64% <100.00%> (+0.05%) ⬆️
spark-3.1.x 97.45% <100.00%> (+0.04%) ⬆️
spark-3.2.x 97.45% <100.00%> (+0.04%) ⬆️
spark-3.3.x 97.45% <100.00%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ore/src/main/scala/doric/syntax/ArrayColumns.scala 100.00% <ø> (ø)
....1_3.2_3.3/scala/doric/syntax/ArrayColumns3x.scala 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3363221...837704c. Read the comment docs.

@github-actions github-actions bot added the spark_2.4 PR changes to spark 2.4 label Aug 30, 2022
@eruizalo eruizalo force-pushed the feat/270-array_sort-functions_with_lambdas branch from 59b416f to 9601156 Compare August 31, 2022 10:05
@eruizalo eruizalo marked this pull request as ready for review August 31, 2022 10:48
@eruizalo eruizalo added enhancement New feature or request breaking 💣 Breaking changes API labels Aug 31, 2022
@eruizalo eruizalo enabled auto-merge (squash) August 31, 2022 11:39
@eruizalo eruizalo added the bug 🐛 Something isn't working label Aug 31, 2022
@eruizalo eruizalo merged commit 86620fa into hablapps:main Sep 7, 2022
@eruizalo eruizalo deleted the feat/270-array_sort-functions_with_lambdas branch September 7, 2022 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API breaking 💣 Breaking changes bug 🐛 Something isn't working enhancement New feature or request spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature request]: ⬆️ 🆒 🆙 array_sort function with lambda
2 participants