Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-33847][SQL][FOLLOWUP] Remove the CaseWhen should consider deterministic #30960

Closed
wants to merge 3 commits into from
Closed

Conversation

wangyum
Copy link
Member

@wangyum wangyum commented Dec 29, 2020

What changes were proposed in this pull request?

This pr fix remove the CaseWhen if elseValue is empty and other outputs are null because of we should consider deterministic.

Why are the changes needed?

Fix bug.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

@github-actions github-actions bot added the SQL label Dec 29, 2020
@wangyum
Copy link
Member Author

wangyum commented Dec 29, 2020

cc @cloud-fan

@cloud-fan
Copy link
Contributor

cloud-fan commented Dec 29, 2020

We need to fix https://github.com/apache/spark/pull/30852/files#diff-9723970916a34149cbb2e3a2853614e523a8a00d5a126926c54f23b4b758e3baR96 as well.

To simplify the code, I'd suggest we also turn unspecified else value into false in the rule ReplaceNullWithFalseInPredicate. It's idempotent and should not have a real perf impact even if there is no following optimizations.

@SparkQA
Copy link

SparkQA commented Dec 29, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38072/

@SparkQA
Copy link

SparkQA commented Dec 29, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38072/

@SparkQA
Copy link

SparkQA commented Dec 29, 2020

Test build #133478 has finished for PR 30960 at commit 4c21e43.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 29, 2020

Test build #133483 has finished for PR 30960 at commit 41a8fed.

  • This patch fails SparkR unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

This is definitely not related to Spark R. Mering to master, thanks!

@cloud-fan cloud-fan closed this in c425024 Dec 29, 2020
@wangyum wangyum deleted the SPARK-33847-2 branch December 29, 2020 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants