Pull requests stuck in "Merge conflict checking is in progress. Try again in few moments" #22578
Labels
Comments
|
Also some PR's are still stuck in Looks like it doesn't retrigger the check. Is there a way to trigger the scan manually? Current situation is pretty ugly because we cannot properly update/merge our branches via Gitea. |
have you seen the |
|
Unfortunately not. This is the block that is always printed: I also found out that the problem does not happen initially on new pull requests. Only as soon as they get updated. |
|
If i click on a affected pull request, the following block is the only thing that is printed: |
|
I noticed also, the amount of given approvals differs in the /pulls view to the approvals inside the /pulls/:id view. And this PR is in status ''Merge conflict checking is in progress. Try again in few moments." See: |
|
Found the following in the logs, may cause this issue? |
|
Looks like its a duplicate of #17204. |
|
Closing the PR and reopening it, seems to fix the issue after i reverted git Version to 2.29.1, the version we have been on before we updated to 1.18.2 |
|
@Tauf I suspect that there is something wrong with that repository on the filesystem. Have you ran git gc, git fsck and git repack on that repo? It would also be helpful to check if there is a process stuck in conflict checking. On the admin pages there is a process stacktrace available off a link on the management pages. You can also get it from |
|
@zeripath Can you assist me with the git commands you mentioned? I'm not that familiar with git. I found stacktraces but there is no link to download them, can you tell me where exactly? Also this is still an issue: |
|
After some time it worked with new pull requests, now we have the same situation again. The pull requests are stuck in the same state. May i ask somebody to assist me with this? |
|
@zeripath like this? |
|
More info from gitea admin pages: Muted logs to warn, that what i see all day: |
|
@tautf you're not providing us with anything that can help us to get to the bottom of the problem. STACKTRACESI've asked for the stacktraces. Could you please give me the stacktraces? These can be obtained extremely easily: GIT COMMITYou say you see: Have you been able to see if there is a git process already running in the repository? Are there hanging processes in your process manager? Somewhat annoyingly you've cut off the first line of that message so I cannot determine which line this relates to. Which piece of code emits that? LOGSSetting your logs to WARN is not going to help us to debug your problem. It would be also be helpful to adjust your logging: Your logs above in #22578 (comment) are not related in any way to the PR checking process so are unhelpful. The logs in #22578 (comment) are also unhelpful. Without telling me the line that emits this: #22578 (comment) it is also unhelpful. I need the logs from when the PR starts to be checked. So right from the push that sets off the checker, for a good while afterwards until you're sure that the PR has been checked and failed to be checked. APP.INIIt would also be helpful to see your app.ini. I need to know whether you have some weird queue set up that is breaking the PR checker. SIZEI need to know around how many PRs you have on your system - it may be that you've just got so many that PR checker takes too long. |
|
as an aside I've just looked at the code for |
zeripath
added a commit
to zeripath/gitea
that referenced
this issue
Jan 26, 2023
The code for checking if a commit has caused a change in a PR is extremely inefficient and affects the head repository instead of using a temporary repository. This PR therefore makes several significant improvements: * A temporary repo like that used in merging. * The diff code is then significant improved to use a three-way diff instead of comparing diffs (possibly binary) line-by-line - in memory... Ref go-gitea#22578 Signed-off-by: Andrew Thornton <art27@cantab.net>
|
@zeripath thanks for your response. Sorry that i was providing unhelpful information. Requested info: Stacktraces: `./gitea.exe manager processes --stacktraces`Logs
|
|
Heya! Thanks for the much more extensive logs. StacktracesLooking at the stacktraces - your request to the stacktrace started at 2023-01-27 08:12:24 and at 2023-01-27 08:12:23 PR[653] = PR number #998 in Repo[24] started being checked. So that doesn't provide evidence of hanging PR checks. You'd need to run stacktraces again in a bit to see if the same PR was hanging somewhere. When did you run stacktraces - it appears early as compared to your logs? Stacktraces is instantaneous - it tells me what is happening at the point of running it - it's helpful to get the stacktrace when you think the problem is occurring - although it is also helpful to get a baseline. Could you tell me the ID and/or Index number of the/a PR that is blocked? Is #998 in Repo[24]/PR[653] the blocked PR? App.iniI see: This was related to a previous bug which should have been fixed. I don't think this could be causing problems but it's not really necessary anymore. LogsI need to know which PR is blocked. There is a lot of PR checking going on there. (I do think our logging here is a little poor as despite my attempts to improve things we still don't have enough context within the trace logging to clearly identify immediately which PR each trace is going to and there doesn't appear to be clear tracing as to when PRs finish being checked.) For example, working backwards from your logs:
Now the PRs that stop after merge-base I think that's because the pr.HeadCommitID is equal to the MergeBase and they're probably ancestors. However, looking at your logs I see you've tried to view heat-coat/app#1079 head kd-chore-fixed-css-and-html-for-order-item-creation but I don't see any checking for that PR. I can't tell from your logs if that's just due to timing What I do see is that there are 24 PRs in the patch checking queue but your logs stop at 12 PRs - so maybe you're just checking too early? But I see that in the monitor screenshot you sent above there were none waiting. OK thanks for logs but I need more. It probably is worth running: and then send us the logs and the stacktraces. If We also need to improve our logging considerably too. I will look at that. |
zeripath
added a commit
that referenced
this issue
Jan 28, 2023
The code for checking if a commit has caused a change in a PR is extremely inefficient and affects the head repository instead of using a temporary repository. This PR therefore makes several significant improvements: * A temporary repo like that used in merging. * The diff code is then significant improved to use a three-way diff instead of comparing diffs (possibly binary) line-by-line - in memory... Ref #22578 Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath
added a commit
to zeripath/gitea
that referenced
this issue
Jan 28, 2023
Backport go-gitea#22611 The code for checking if a commit has caused a change in a PR is extremely inefficient and affects the head repository instead of using a temporary repository. This PR therefore makes several significant improvements: * A temporary repo like that used in merging. * The diff code is then significant improved to use a three-way diff instead of comparing diffs (possibly binary) line-by-line - in memory... Ref go-gitea#22578 Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath
added a commit
that referenced
this issue
Jan 28, 2023
Backport #22611 The code for checking if a commit has caused a change in a PR is extremely inefficient and affects the head repository instead of using a temporary repository. This PR therefore makes several significant improvements: * A temporary repo like that used in merging. * The diff code is then significant improved to use a three-way diff instead of comparing diffs (possibly binary) line-by-line - in memory... Ref #22578 Signed-off-by: Andrew Thornton <art27@cantab.net>
techknowlogick
added a commit
that referenced
this issue
Feb 3, 2023
Our trace logging is far from perfect and is difficult to follow. This PR: * Add trace logging for process manager add and remove. * Fixes an errant read file for git refs in getMergeCommit * Brings in the pullrequest `String` and `ColorFormat` methods introduced in #22568 * Adds a lot more logging in to testPR etc. Ref #22578 --------- Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
zeripath
added a commit
to zeripath/gitea
that referenced
this issue
Feb 7, 2023
Backport go-gitea#22633 Our trace logging is far from perfect and is difficult to follow. This PR: * Add trace logging for process manager add and remove. * Fixes an errant read file for git refs in getMergeCommit * Brings in the pullrequest `String` and `ColorFormat` methods introduced in go-gitea#22568 * Adds a lot more logging in to testPR etc. Ref go-gitea#22578 --------- Signed-off-by: Andrew Thornton <art27@cantab.net>
lunny
pushed a commit
that referenced
this issue
Feb 13, 2023
Backport #22633 Our trace logging is far from perfect and is difficult to follow. This PR: * Add trace logging for process manager add and remove. * Fixes an errant read file for git refs in getMergeCommit * Brings in the pullrequest `String` and `ColorFormat` methods introduced in #22568 * Adds a lot more logging in to testPR etc. Ref #22578 --------- Signed-off-by: Andrew Thornton <art27@cantab.net>
|
Hi! We encountered the same issue on our Gitea instance. We had a 500 error when opening a When I looked at the log, I saw this error: So I added some dashes to commands: dontnod@80e504c Then I was still left with an error that I managed to track to this code: https://github.com/go-gitea/gitea/blob/main/modules/git/repo_compare.go#L42 I can rebase these commits (or only the However, the fact the function somewhat ignores the fetch error and goes along maybe should be addressed as well? |
|
Also running into this issue (Windows Server), with no further information: Not having this issue on the go-git version, apparently. |
My personal opinion (does not represent the opinion of the Gitea or maintainers):
I think it's good enough (sorry that I missed your message about "open a PR if this approach is acceptable"), so I closed my duplicate one.
I have some questions about it:I guess there is a concurrency problem
|
IIRC, this is in the 'true' repository (didn't push my investigation into if a tmp repository would be better suited) but it's a tmp remote that is added then removed just after. So I guess it could impact performances for git commands that would run while this remote is present but should only impact a subset of commands that use the commit graph?
The lock file seem to be unique for commit-graph operations |
Agree, that's also an unclear part in my mind. According to "the commit-graph file format is faster to parse than decompressing commit files and parsing them to find their parents and root trees.", so I guess the following "merge-base" and "log" commands may benefit from it, but I do not know how much benefit it brings, maybe it differs between big and small repositories. So I have no idea TBH .... Personally I would agree with that adding "--no-write-commit-graph" is fine for most cases.
From another side, I think maybe Gitea has a design flaw (bug) in its conflict checking code. If there are concurrent git operations, then the "conflict checking" should retry instead of getting stuck if the competition for the lock fails. If the lock competition could be handled correctly, then no need to worry about the commit-graph lock? |
zeripath
pushed a commit
that referenced
this issue
Mar 26, 2023
Reference: #22578 (comment) Credits to @tdesveaux , thank you very much for catching the problem. If you'd like to open a PR, feel free to replace this one. Git reports fatal errors for ambiguous arguments: ``` fatal: ambiguous argument 'refs/a...refs/b': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' ``` So the `--` separator is necessary in some cases.
GiteaBot
pushed a commit
to GiteaBot/gitea
that referenced
this issue
Mar 26, 2023
…23606) Reference: go-gitea#22578 (comment) Credits to @tdesveaux , thank you very much for catching the problem. If you'd like to open a PR, feel free to replace this one. Git reports fatal errors for ambiguous arguments: ``` fatal: ambiguous argument 'refs/a...refs/b': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' ``` So the `--` separator is necessary in some cases.
6543
pushed a commit
that referenced
this issue
Apr 4, 2023
…23720) Backport #23606 by @wxiaoguang Reference: #22578 (comment) Credits to @tdesveaux , thank you very much for catching the problem. If you'd like to open a PR, feel free to replace this one. Git reports fatal errors for ambiguous arguments: ``` fatal: ambiguous argument 'refs/a...refs/b': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' ``` So the `--` separator is necessary in some cases. Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
|
We ran into the same The Perhaps a Git crash or kill after a timeout could cause this too. |
|
We can have a cron job to cleanup the |
wxiaoguang
added a commit
to wxiaoguang/gitea
that referenced
this issue
Jul 20, 2023
|
@zeripath Observation:
Problem:
Here are the stack logs |
|
Each merge merged into the main branch， all open status pull request will perform a merge conflict checking，If you have a very old pull request, Please mark it as WIP, reduce the number of conflict queues and accelerate the speed of conflict checking |
This was referenced Jun 13, 2024
@lunny is there anything regarding this already implemented in the latest versions? Or a other suggested Workaround? |
|
As a workaround, just find all |
|
@lunny Would you like me to try to implement this cron job solution ? If it is ok with you guys I would love to |
Maybe we can have two improvements. One is trying to delete the lock file once we got |
|
Makes sense. I'll work on that :) I think by next week I should have both PRs done. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
After the update to 1.18.2 all our pull requests are stuck with above error message. I did restart the server multiple times already, we cannot get out of this view.
This was not happening with 1.17.4 and 1.18.0.
I upgraded from 1.18.0 to 1.18.2 and skipped 1.18.1, could that cause the issue?
Problem got worse with 1.18.0 that it became slower but it at least resolved after 10-15 minutes. Now we are stuck for multiple hours.
I am using the gogit variant but it looks like the windows side git is used or am i wrong?
Not sure if the logs are related to it but thats the only errors i could find. I have disabled my anti-virus but the issue still persists.
Gitea Version
1.18.2
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
Screenshots
No response
Git Version
2.39.1
Operating System
Windows Server 2019
How are you running Gitea?
Windows Service
Database
PostgreSQL
The text was updated successfully, but these errors were encountered: