-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sqlutil: remove Query and QueryEx from InternalExecutor interface #60428
Conversation
@andreimatei or @ajwerner, could one of you please take a look at the first commit? @miretskiy, could you please take a look at the second commit? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 16 of 16 files at r1, 8 of 8 files at r2.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @andreimatei and @miretskiy)
pkg/jobs/adopt.go, line 283 at r1 (raw file):
Just rambles for myself:
Quoted 6 lines of code…
// Note that we have to buffer all rows first - before processing each // job - because we have to make sure that the query executes without an // error (otherwise, the system.jobs table might diverge from the jobs // registry). // TODO(yuzefovich): use QueryBufferedEx method once it is added to // sqlutil.InternalExecutor interface.
I'm not so sure about this. We could just make it two loops but I guess at that point, it's the same complexity in terms of memory.
Also, we could add a LIMIT
here if we wanted without violating correctness.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @andreimatei and @miretskiy)
pkg/jobs/adopt.go, line 283 at r1 (raw file):
Previously, ajwerner wrote…
Just rambles for myself:
// Note that we have to buffer all rows first - before processing each // job - because we have to make sure that the query executes without an // error (otherwise, the system.jobs table might diverge from the jobs // registry). // TODO(yuzefovich): use QueryBufferedEx method once it is added to // sqlutil.InternalExecutor interface.
I'm not so sure about this. We could just make it two loops but I guess at that point, it's the same complexity in terms of memory.
Also, we could add a
LIMIT
here if we wanted without violating correctness.
I tried avoiding the buffering of rows here, but the CI was failing (I think something about auto stats collection, but don't remember for sure) which I bisected to this spot. The comment is my best explanation for why the buffering is needed.
All usages of `sqlutil.InternalExecutor.Query` and `sqlutil.InternalExecutor.QueryEx` have been refactored: most of the changes are taking advantage of the iterator API, in a couple of places we now use `Exec` and `ExecEx` methods since in those places we only need the number of affected rows. In a few places where we still need to buffer all rows, the caller currently performs the buffering. In the follow-up commit `QueryBuffered` and `QueryBufferedEx` will be added into the interface (essentially renaming `Query` and `QueryEx` to make it explicit that the buffering occurs), and the custom logic will be removed. Release note: None
I removed the second commit and will merge it in a separate PR. TFTR! bors r+ |
This PR was included in a batch that was canceled, it will be automatically retried |
Build failed (retrying...): |
Build succeeded: |
All usages of
sqlutil.InternalExecutor.Query
andsqlutil.InternalExecutor.QueryEx
have been refactored: most ofthe changes are taking advantage of the iterator API, in a couple
of places we now use
Exec
andExecEx
methods since in thoseplaces we only need the number of affected rows.
In a few places where we still need to buffer all rows, the caller
currently performs the buffering. In the follow-up commit
QueryBuffered
andQueryBufferedEx
will be added into the interface(essentially renaming
Query
andQueryEx
to make it explicit that thebuffering occurs), and the custom logic will be removed.
Addresses: #48595.
Release note: None