Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [+] #337 array_sort of structs & equals null safe functions #338

Merged
merged 2 commits into from
Apr 20, 2023

Conversation

eruizalo
Copy link
Collaborator

Description

This PR contains two new functions for doric:

  • Equals null safe: <=>
  • Sort arrays for structs: colArray[Row]("value").sortBy(CName("name"), CNameOrd("age", Desc)) is available instead of createng ad-hoc sort functions or using ad-hoc expressions like:
f.expr("array_sort(value, (l, r) -> case " +
  // name ASC
  "when l.name < r.name then -1 " +
  "when l.name > r.name then 1 " +
  "else ( case" +
  // age DESC
  "  when l.age > r.age then -1 " +
  "  when l.age < r.age then 1 " +
  "  else 0 end " +
  ") end)"
)

Related Issue and dependencies

How Has This Been Tested?

  • This pull request contains appropriate tests?:
    • YES

@eruizalo eruizalo requested a review from a team as a code owner March 13, 2023 11:19
@github-actions github-actions bot added spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3 labels Mar 13, 2023
@eruizalo eruizalo added documentation Improvements or additions to documentation enhancement New feature or request API labels Mar 13, 2023
@github-actions
Copy link

github-actions bot commented Mar 13, 2023

:octocat: This is an auto-generated comment created by:

  • Date : 2023-04-19 17:11:17 +0000 (UTC)
  • Workflow : PR comment
  • Job name : create_test_summary_report
  • Run : 4746039744
  • Commit : 9fdd6c3 feat: [+] null first/last order options
Actor Triggering actor Sender
eruizalo
eruizalo
eruizalo
eruizalo
eruizalo
eruizalo
Triggered by:

Test summary report 📊

Spark version testing
2.4.1 589 passed, 2 skipped
2.4.2 589 passed, 2 skipped
2.4.3 589 passed, 2 skipped
2.4.4 589 passed, 2 skipped
2.4.5 589 passed, 2 skipped
2.4.6 590 passed, 2 skipped
2.4.7 590 passed, 2 skipped
2.4 590 passed, 2 skipped
3.0.0 631 passed, 2 skipped
3.0.1 631 passed, 2 skipped
3.0.2 631 passed, 2 skipped
3.0 631 passed, 2 skipped
3.1.0 659 passed, 2 skipped
3.1.1 659 passed, 2 skipped
3.1.2 659 passed, 2 skipped
3.1 659 passed, 2 skipped
3.2.0 663 passed, 2 skipped
3.2.1 663 passed, 2 skipped
3.2.2 663 passed, 2 skipped
3.2 663 passed, 2 skipped
3.3.0 663 passed, 2 skipped
3.3.1 663 passed, 2 skipped
3.3 663 passed, 2 skipped

@codecov
Copy link

codecov bot commented Mar 14, 2023

Codecov Report

Merging #338 (9fdd6c3) into main (55f96b9) will increase coverage by 0.06%.
The diff coverage is 100.00%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #338      +/-   ##
==========================================
+ Coverage   97.42%   97.49%   +0.06%     
==========================================
  Files          60       61       +1     
  Lines        1164     1194      +30     
  Branches       19       21       +2     
==========================================
+ Hits         1134     1164      +30     
  Misses         30       30              
Flag Coverage Δ
spark-2.4.x 94.42% <25.00%> (-0.27%) ⬇️
spark-3.0.x 96.58% <100.00%> (+0.10%) ⬆️
spark-3.1.x 97.36% <100.00%> (+0.07%) ⬆️
spark-3.2.x 97.59% <100.00%> (+0.06%) ⬆️
spark-3.3.x 97.59% <100.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
core/src/main/scala/doric/CNameOrd.scala 100.00% <100.00%> (ø)
...re/src/main/scala/doric/syntax/CommonColumns.scala 100.00% <100.00%> (ø)
....1_3.2_3.3/scala/doric/syntax/ArrayColumns3x.scala 100.00% <100.00%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 55f96b9...9fdd6c3. Read the comment docs.

@eruizalo eruizalo self-assigned this Mar 14, 2023
@github-actions github-actions bot removed the documentation Improvements or additions to documentation label Mar 15, 2023
@eruizalo eruizalo force-pushed the feat/337-array_sort_structs branch 2 times, most recently from 71357bb to 10b9190 Compare March 16, 2023 08:35
@eruizalo eruizalo force-pushed the feat/337-array_sort_structs branch from 10b9190 to 9fdd6c3 Compare April 19, 2023 15:14
@eruizalo eruizalo merged commit 704185c into hablapps:main Apr 20, 2023
@eruizalo eruizalo deleted the feat/337-array_sort_structs branch April 20, 2023 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API enhancement New feature or request spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

[Feature request]: Custom array_sort function for structs
2 participants