Skip to content

Commit

Permalink
[SPARK-49628][SQL] ConstantFolding should copy stateful expression be…
Browse files Browse the repository at this point in the history
…fore evaluating

### What changes were proposed in this pull request?

It's possible that a logical plan instance is being shared by multiple DFs and these DFs are executed in parallel. Spark always copy stateful expressions before evaluating them, but one place is missed: `ConstantFolding` can also execute expressions. This PR fixes it.

### Why are the changes needed?

avoid concurrency issues.

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

Not able to write a test for it, but this concurrency issue is quite obvious

### Was this patch authored or co-authored using generative AI tooling?

no

Closes #48104 from cloud-fan/constant.

Authored-by: Wenchen Fan <wenchen@databricks.com>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit 319e7cc)
Signed-off-by: Kent Yao <yao@apache.org>
  • Loading branch information
cloud-fan authored and yaooqinn committed Sep 13, 2024
1 parent 560efed commit e693e18
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ object ConstantFolding extends Rule[LogicalPlan] {
// Fold expressions that are foldable.
case e if e.foldable =>
try {
Literal.create(e.eval(EmptyRow), e.dataType)
Literal.create(e.freshCopyIfContainsStatefulExpression().eval(EmptyRow), e.dataType)
} catch {
case NonFatal(_) if isConditionalBranch =>
// When doing constant folding inside conditional expressions, we should not fail
Expand Down

0 comments on commit e693e18

Please sign in to comment.