Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [+] MatchBuilder column fun & AND/OR aggregate functions #336

Merged
merged 5 commits into from
Apr 20, 2023

Conversation

eruizalo
Copy link
Collaborator

@eruizalo eruizalo commented Mar 3, 2023

Description

  • AND / OR aggregate functions were added (equivalent to min and max respectively, but seems more clear this way)
  • WhenBuilder for columns, very similar to WhenBuilder, intended to do a sequence of comparisons with the column itself. It looks like this:
colInt("c1").matches[String]
    .caseW(_ > lit(10), "MAX".lit) // using a "compare" function
    .caseW(lit(8), "Eight".lit).   // this is direct comparison between both columns (the very same as ==)
    .otherwiseNull

Related Issue and dependencies

  • Resolves N/A
  • Depends on N/A

How Has This Been Tested?

  • This pull request contains appropriate tests?:
    • YES

@eruizalo eruizalo requested a review from a team as a code owner March 3, 2023 11:01
@eruizalo eruizalo self-assigned this Mar 3, 2023
@github-actions github-actions bot added spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3 labels Mar 3, 2023
@github-actions
Copy link

github-actions bot commented Mar 3, 2023

:octocat: This is an auto-generated comment created by:

  • Date : 2023-04-20 10:07:36 +0000 (UTC)
  • Workflow : PR comment
  • Job name : create_test_summary_report
  • Run : 4753054228
  • Commit : e6e110d Apply suggestions from code review
Actor Triggering actor Sender
eruizalo
eruizalo
eruizalo
eruizalo
eruizalo
eruizalo
Triggered by:

Test summary report 📊

Spark version testing
2.4.1
2.4.2
2.4.3
2.4.4
2.4.5
2.4.6
2.4.7
2.4
3.0.0
3.0.1
3.0.2
3.0
3.1.0
3.1.1
3.1.2
3.1
3.2.0
3.2.1
3.2.2
3.2
3.3.0
3.3.1
3.3

@eruizalo eruizalo force-pushed the feat/and_or_agg_fun_and_matchBuilder branch 2 times, most recently from bf8d4fe to 3c9ec96 Compare March 3, 2023 11:16
@codecov
Copy link

codecov bot commented Mar 3, 2023

Codecov Report

Merging #336 (36d798c) into main (704185c) will increase coverage by 0.02%.
The diff coverage is 100.00%.

❗ Current head 36d798c differs from pull request most recent head e6e110d. Consider uploading reports for the commit e6e110d to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #336      +/-   ##
==========================================
+ Coverage   97.49%   97.51%   +0.02%     
==========================================
  Files          61       61              
  Lines        1194     1203       +9     
  Branches       15       20       +5     
==========================================
+ Hits         1164     1173       +9     
  Misses         30       30              
Flag Coverage Δ
spark-2.4.x 94.47% <100.00%> (+0.05%) ⬆️
spark-3.0.x 96.60% <100.00%> (+0.03%) ⬆️
spark-3.1.x 97.38% <100.00%> (+0.02%) ⬆️
spark-3.2.x 97.61% <100.00%> (+0.02%) ⬆️
spark-3.3.x 97.61% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...c/main/scala/doric/syntax/AggregationColumns.scala 100.00% <100.00%> (ø)
...rc/main/scala/doric/syntax/ControlStructures.scala 100.00% <100.00%> (ø)
core/src/main/scala/doric/syntax/WhenBuilder.scala 100.00% <100.00%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 704185c...e6e110d. Read the comment docs.

@eruizalo eruizalo force-pushed the feat/and_or_agg_fun_and_matchBuilder branch from 3c9ec96 to 61c8b4f Compare March 3, 2023 11:29
@eruizalo eruizalo added enhancement New feature or request API labels Mar 6, 2023
@eruizalo eruizalo force-pushed the feat/and_or_agg_fun_and_matchBuilder branch from 61c8b4f to 23cfad0 Compare March 8, 2023 08:23
alfonsorr
alfonsorr previously approved these changes Apr 19, 2023
core/src/main/scala/doric/syntax/WhenBuilder.scala Outdated Show resolved Hide resolved
alfonsorr
alfonsorr previously approved these changes Apr 20, 2023
@eruizalo eruizalo enabled auto-merge (squash) April 20, 2023 10:04
@eruizalo eruizalo merged commit e892661 into hablapps:main Apr 20, 2023
@eruizalo eruizalo deleted the feat/and_or_agg_fun_and_matchBuilder branch April 20, 2023 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API enhancement New feature or request spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants