-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-8573] [SPARK-8568] [SQL] [PYSPARK] raise Exception if column is used in booelan expression #6961
Conversation
cc @rxin |
@@ -396,6 +396,11 @@ def over(self, window): | |||
jc = self._jc.over(window._jspec) | |||
return Column(jc) | |||
|
|||
def __nonzero__(self): | |||
raise ValueError("Can't convert column into bool: please use '&' for 'and', '|' for 'or', " |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we mention not as well?
And "Can't" -> "Cannot"
lgtm otherwise |
@rxin updated. |
@@ -396,6 +396,11 @@ def over(self, window): | |||
jc = self._jc.over(window._jspec) | |||
return Column(jc) | |||
|
|||
def __nonzero__(self): | |||
raise ValueError("Cannot convert column into bool: please use '&' for 'and', '|' for 'or', " | |||
"'~'for 'not', when using Column in a boolean expression.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missed a space after ~
actually let's do this
Cannot convert column into bool: please use '&' for 'and', '|' for 'or', and '~' for 'not' when building DataFrame boolean expressions.
Test build #35590 has finished for PR 6961 at commit
|
Test build #954 has finished for PR 6961 at commit
|
Test build #35592 has finished for PR 6961 at commit
|
Test build #35597 has finished for PR 6961 at commit
|
…s used in booelan expression It's a common mistake that user will put Column in a boolean expression (together with `and` , `or`), which does not work as expected, we should raise a exception in that case, and suggest user to use `&`, `|` instead. Author: Davies Liu <davies@databricks.com> Closes #6961 from davies/column_bool and squashes the following commits: 9f19beb [Davies Liu] update message af74bd6 [Davies Liu] fix tests 07dff84 [Davies Liu] address comments, fix tests f70c08e [Davies Liu] raise Exception if column is used in booelan expression (cherry picked from commit 7fb5ae5) Signed-off-by: Davies Liu <davies@databricks.com>
Test build #35604 has finished for PR 6961 at commit
|
Merged build finished. Test FAILed. |
…s used in booelan expression It's a common mistake that user will put Column in a boolean expression (together with `and` , `or`), which does not work as expected, we should raise a exception in that case, and suggest user to use `&`, `|` instead. Author: Davies Liu <davies@databricks.com> Closes apache#6961 from davies/column_bool and squashes the following commits: 9f19beb [Davies Liu] update message af74bd6 [Davies Liu] fix tests 07dff84 [Davies Liu] address comments, fix tests f70c08e [Davies Liu] raise Exception if column is used in booelan expression (cherry picked from commit 7fb5ae5) Signed-off-by: Davies Liu <davies@databricks.com>
It's a common mistake that user will put Column in a boolean expression (together with
and
,or
), which does not work as expected, we should raise a exception in that case, and suggest user to use&
,|
instead.