Avoid WHERE IN for comment migration query
#28500
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Rewrite `UpdateCommentsMigrationsByType` to not use `WHERE IN` as that's a performance diaster for MariaDB, it now use batching to query the the relevant comment IDs via JOINs (which is not possible in a UPDATE query for SQLite) and then update them in a seperate query. - Resolves https://codeberg.org/forgejo/forgejo/issues/1856
GiteaBot
added
the
lgtm/need 2
pull-request-size
bot
added
the
size/M
lunny
reviewed
Dec 18, 2023
| "original_author_id": 0, | ||
| }) | ||
| return err | ||
| const batchSize = 50 |
There was a problem hiding this comment.
Maybe we can first interate all the special repositories since they will not be expected too much.
|
Instead od in would a join be more efective? |
delvh
approved these changes
Dec 19, 2023
| } | ||
|
|
||
| for _, commentID := range commentIDs { | ||
| if _, err := db.GetEngine(ctx).Table(&Comment{}).ID(commentID).Update(map[string]any{ |
There was a problem hiding this comment.
Woohoo.
This sounds like such an improvement.
Instead of one long query we now open one million db connections 😒.
SQLite, please fix your weird non-SQL conform shenanigans.
There was a problem hiding this comment.
per #28500 (comment), I think we can reduce the requests times.
There was a problem hiding this comment.
I don't see how that reduces the requests.
GiteaBot
added
lgtm/need 1
6543
pushed a commit
that referenced
this pull request
Dec 21, 2023
Replace #28500 --------- Co-authored-by: Giteabot <teabot@gitea.io>
GiteaBot
added a commit
to GiteaBot/gitea
that referenced
this pull request
Dec 21, 2023
Replace go-gitea#28500 --------- Co-authored-by: Giteabot <teabot@gitea.io>
techknowlogick
pushed a commit
to techknowlogick/gitea
that referenced
this pull request
Dec 23, 2023
Replace go-gitea#28500 --------- Co-authored-by: Giteabot <teabot@gitea.io>
fuxiaohei
pushed a commit
to fuxiaohei/gitea
that referenced
this pull request
Jan 17, 2024
Replace go-gitea#28500 --------- Co-authored-by: Giteabot <teabot@gitea.io>
silverwind
pushed a commit
to silverwind/gitea
that referenced
this pull request
Feb 20, 2024
Replace go-gitea#28500 --------- Co-authored-by: Giteabot <teabot@gitea.io>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
UpdateCommentsMigrationsByTypeto not useWHERE INas that's a performance diaster for MariaDB, it now use batching to query the the relevant comment IDs via JOINs (which is not possible in a UPDATE query for SQLite) and then update them in a seperate query.