-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-47692][SQL] Fix default StringType meaning in implicit casting #45819
Conversation
@dbatomic @stefankandic @uros-db @stevomitric @nikolamand-db Could you take a look at this PR? It introduces a new flag for collations and makes sure most of the functions that return StringType without StringType in inputs return default collation. |
How do we want to handle cases like string split (work in progress, used as illustration) in general? The regex parameter accepts only |
If the collation is not important to the function execution we can create a separate pattern for it where we will exclude parameters that do not need casting. For example, substring is done in that way. We only cast if it is DefaultST or ExplicitST to ImplicitST, as we need the return type to have implicit collation. |
# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala # sql/core/src/test/scala/org/apache/spark/sql/CollationSuite.scala
# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
val implicitTypes = expr.map(_.dataType) | ||
val implicitTypes = expr.filter { | ||
case Literal(_, _: StringType) => false | ||
case Cast(child, _: StringType, _, _) => child.dataType.isInstanceOf[StringType] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's also check cast.getTagValue(Cast.USER_SPECIFIED_CAST).isEmpty
to make sure it's not a user-specified cast.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh wait, if users use scala/python DataFrame to cast a column to a StringType with specific collation, should it be explicit collation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same for SQL: CAST(col AS STRING WITH ...)
# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala
thanks, merging to master! |
### What changes were proposed in this pull request? Addition of priority flag to StringType. ### Why are the changes needed? In order to follow casting rules for collations, we need to know whether StringType is considered default, implicit or explicit. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? Implicit tests in CollationSuite. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45819 from mihailom-db/SPARK-47692. Authored-by: Mihailo Milosevic <mihailo.milosevic@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
Addition of priority flag to StringType.
Why are the changes needed?
In order to follow casting rules for collations, we need to know whether StringType is considered default, implicit or explicit.
Does this PR introduce any user-facing change?
Yes.
How was this patch tested?
Implicit tests in CollationSuite.
Was this patch authored or co-authored using generative AI tooling?
No.