-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-44866][SQL] Add SnowflakeDialect
to handle BOOLEAN type correctly
#42545
Conversation
In snowflake, the BIT data type does not exist. A Boolean is a Boolean. That's what this is about.
Could you file a ticket? https://issues.apache.org/jira/browse/SPARK |
Juste created the ticket https://issues.apache.org/jira/browse/SPARK-44866 |
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
Outdated
Show resolved
Hide resolved
sql/core/src/main/scala/org/apache/spark/sql/jdbc/SnowflakeDialect.scala
Outdated
Show resolved
Hide resolved
SnowflakeDialect
to handle BOOLEAN type correctly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for making a PR, @hayssams .
Since this PR adds a new dialect, I need to ask if you are using this code without any other issues.
Hi @DongJoon
Yes we are using this PR without any other issue.
On the other hand before submitting this PR, I looked at the default JDBC
mapping and did not find any other incompatibility
…On Fri 8 Sep 2023 at 22:51 Dongjoon Hyun ***@***.***> wrote:
***@***.**** commented on this pull request.
Thank you for making a PR, @hayssams <https://github.com/hayssams> .
Since this PR adds a new dialect, I need to ask if you are using this code
without any other issues.
—
Reply to this email directly, view it on GitHub
<#42545 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAS4ACE34FTZ3OUURND777DXZOAOXANCNFSM6AAAAAA3USZKMY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Thank you for the confirmation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM for Apache Spark 4.0.0.
Welcome to the Apache Spark community and congratulations for your first commit! |
What changes were proposed in this pull request?
In Snowflake a BOOLEAN data type exist but not the BIT data type.
This PR adds
SnowflakeDialect
to override the default JdbcDialect and redefine the default mapping behaviour for the boolean type. Currently, it's mapped toBIT(1)
type.spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
Line 149 in a663c0b
Why are the changes needed?
The BIT type does not exist in Snowflake. This cause the Spark Job to fail on table creation.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Unit test and directly on Snowflake