Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sqlparser: Improve interval parsing #13165

Merged
merged 3 commits into from
May 29, 2023
Merged

sqlparser: Improve interval parsing #13165

merged 3 commits into from
May 29, 2023

Conversation

dbussink
Copy link
Contributor

Parse only strict valid units for intervals. We should not allow any arbitrary string value here. We already had the internal enumerable available, but we were not using yet.

Related Issue(s)

In support of #9647 to then make it easier to add support for functions using intervals.

Checklist

  • "Backport to:" labels have been added if this change should be back-ported
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on the CI
  • Documentation was added or is not required

@vitess-bot vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels May 26, 2023
@vitess-bot
Copy link
Contributor

vitess-bot bot commented May 26, 2023 •
edited by GuptaManan100
Loading

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
  • If a test is added or modified, there should be a documentation on top of the test to explain what the expected behavior is what the test does.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

@dbussink dbussink requested a review from vmg May 26, 2023 13:34
@dbussink dbussink removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says NeedsIssue A linked issue is missing for this Pull Request labels May 26, 2023
@github-actions github-actions bot added this to the v17.0.0 milestone May 26, 2023
@dbussink dbussink requested a review from frouioui as a code owner May 26, 2023 14:42
@dbussink dbussink force-pushed the improve-interval-parsing branch from 9b475bb to c57ede6 Compare May 26, 2023 15:40
@dbussink
Copy link
Contributor Author

@systay Looks like there's an upgrade / downgrade failing test unrelated to the changes here:

=== RUN   TestOrderBy
    orderby_test.go:70: 
        	Error Trace:	/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/orderby/cmp.go:195
        	            				/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/orderby/cmp.go:65
        	            				/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/orderby/orderby_test.go:70
        	Error:      	Received unexpected error:
        	            	VT13001: [BUG] in scatter query: complex ORDER BY expression: reverse(id2) (errno 1815) (sqlstate HY000) during query: select /*vt+ PLANNER=Gen4 */ id1, id2 from t4 order by reverse(id2) desc
        	Test:       	TestOrderBy
        	Messages:   	[Vitess Error] for query: select /*vt+ PLANNER=Gen4 */ id1, id2 from t4 order by reverse(id2) desc
--- FAIL: TestOrderBy (0.04s)
FAIL

dbussink added 3 commits May 29, 2023 13:50
@dbussink
sqlparser: Improve interval parsing
fe6fca5 
Parse only strict valid units for intervals. We should not allow any
arbitrary string value here. We already had the internal enumerable
available, but we were not using yet.

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
@dbussink
sqlparser: Explicit parsing for date functions with intervals
0d9384d 
In order to better be able to implement these in the evalengine, it's
easier to parse them explicitly in the parser. This ensures we have the
interval type etc. directly available instead of having to infer it
from a separate expression.

Also makes parsing these functions more strict and better match MySQL.

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
@dbussink
sqlparser: Remove INTERVAL as general expression
9c2a16f 
INTERVAL is not a possible general expression, but only used in specific
cases. With the previous change, we already have it all handled for date
functions. The only remaining cases are for + and - where this is
handled.

Those cases are a bit special. You can have an interval on both sides of
a +, but for a - it can only be done on the right hand side.

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
@dbussink dbussink force-pushed the improve-interval-parsing branch from e5bd5d6 to 9c2a16f Compare May 29, 2023 11:51
GuptaManan100
GuptaManan100 approved these changes May 29, 2023
View reviewed changes
Copy link
Member

@GuptaManan100 GuptaManan100 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@GuptaManan100
Copy link
Member

The upgrade downgrade failure is unrelated and being fixed by #13183

systay
systay reviewed May 29, 2023
View reviewed changes
go/test/endtoend/vtgate/queries/misc/misc_test.go
@@ -174,8 +174,8 @@ func TestIntervalWithMathFunctions(t *testing.T) {
// Set the time zone explicitly to UTC, otherwise the output of FROM_UNIXTIME is going to be dependent
// on the time zone of the system.
mcmp.Exec("SET time_zone = '+00:00'")
mcmp.AssertMatches("select '2020-01-01' + interval month(DATE_SUB(FROM_UNIXTIME(1234), interval 1 month))-1 month", `[[CHAR("2020-12-01")]]`)
mcmp.AssertMatches("select DATE_ADD(MIN(FROM_UNIXTIME(1673444922)),interval -DAYOFWEEK(MIN(FROM_UNIXTIME(1673444922)))+1 DAY)", `[[DATETIME("2023-01-08 13:48:42")]]`)
mcmp.AssertMatches("select '2020-01-01' + interval month(date_sub(FROM_UNIXTIME(1234), interval 1 month))-1 month", `[[CHAR("2020-12-01")]]`)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: why change this? the capitalized form is also valid, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@systay It's because we normalize explicit function names here to lowercase for explicitly known parser implementations and I didn't want to diverge from that here. Similarly here, for example month also was emitted as lowercase.

It's also how MySQL itself normalizes it if you use it for example in a default in create table if you then issue a show create table statement. We try to follow that as the canonical form then.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here you are changing the query we are sending down to vtgate/mysql. why did that need to change? we should follow the canonical form for the printing of ASTs, but we have to be able to receive the queries in any valid form, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah, it would also still be valid. I was pretty aggressive though with the find / replace to lowercase it which is why this one was caught as well 😄.

systay
systay approved these changes May 29, 2023
View reviewed changes
Copy link
Collaborator

@systay systay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dbussink dbussink merged commit 2ba1edc into vitessio:main May 29, 2023
@dbussink dbussink deleted the improve-interval-parsing branch May 29, 2023 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GuptaManan100 GuptaManan100 GuptaManan100 approved these changes

@systay systay systay approved these changes

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal is a code owner

@vmg vmg Awaiting requested review from vmg

@frouioui frouioui Awaiting requested review from frouioui frouioui is a code owner

Assignees
No one assigned
Labels
Component: Query Serving Type: Internal Cleanup
Projects
None yet
Milestone
v17.0.0
Development

Successfully merging this pull request may close these issues.
3 participants
@dbussink @GuptaManan100 @systay