-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-35069][SQL] TRANSFORM forbidden DISTICNT
and ALL
, also make the error clear
#32149
Conversation
FYI @cloud-fan |
Kubernetes integration test starting |
Kubernetes integration test status failure |
|
||
|
||
-- !query | ||
SELECT TRANSFORM(distinct b, a, c) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah the parser treats it as distinct as b
.
Can we improve the parser to recognize distinct
correctly and give a clear error message?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW why do we support TRANSFORM(a as b)
? is it from Hive? Normal function parameters can't have alias.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW why do we support
TRANSFORM(a as b)
? is it from Hive? Normal function parameters can't have alias.
Hive not support this.
may be we should not use
namedExpressionSeq
: namedExpression (',' namedExpression)*
;
(SELECT kind=TRANSFORM '(' namedExpressionSeq ')'
| kind=MAP namedExpressionSeq
| kind=REDUCE namedExpressionSeq)
maybe
expressionSeq
: expression(',' expression)
;
(SELECT kind=TRANSFORM '(' expressionSeq ')'
| kind=MAP expressionSeq
| kind=REDUCE eExpressionSeq)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea, let's fix it as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea, let's fix it as well.
Should I fix both issue in this pr or in separated PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's have 2 PRs. forbid distinct and forbid alias are two things.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's have 2 PRs. forbid distinct and forbid alias are two things.
I have make a new ticket for this one. and changed the title.
How about current
Test build #137289 has finished for PR 32149 at commit
|
distinct
DISTICNT
and ALL
, also make the error clear
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala
Outdated
Show resolved
Hide resolved
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala
Outdated
Show resolved
Hide resolved
DISTICNT
and ALL
, also make the error clearDISTICNT
and ALL
, also make the error clear
Kubernetes integration test starting |
Kubernetes integration test status failure |
Test build #137340 has finished for PR 32149 at commit
|
thanks, merging to master! |
What changes were proposed in this pull request?
According to #29087 (comment), add UT in
transform.sql
It seems that distinct is not recognized as a reserved word here
Hive's error
Why are the changes needed?
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added Ut