Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync branches to DB immediately when handle git hook calling #29493

Merged
merged 12 commits into from
Mar 6, 2024
Merged

Sync branches to DB immediately when handle git hook calling #29493

merged 12 commits into from
Mar 6, 2024

Conversation

wolfogre
Copy link
Member

@wolfogre wolfogre commented Feb 29, 2024

Unlike other async processing in the queue, we should sync branches to the DB immediately when handling git hook calling. If it fails, users can see the error message in the output of the git command.

It can avoid potential inconsistency issues, and help #29494.

@wolfogre wolfogre added the type/enhancement An improvement of existing functionality label Feb 29, 2024
@wolfogre wolfogre added this to the 1.22.0 milestone Feb 29, 2024
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Feb 29, 2024
@pull-request-size pull-request-size bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 29, 2024
@lunny
Copy link
Member

lunny commented Feb 29, 2024

The first sync for a repository with many branches will wait for a long time.

wolfogre and others added 2 commits March 1, 2024 09:57
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
@wolfogre
Copy link
Member Author

wolfogre commented Mar 1, 2024

The first sync for a repository with many branches will wait for a long time.

It makes sense. What about keeping the old way to sync branches but adding additional pre-sync for the default branch only?

It's not ideal, but I don't have a better idea.


Update:

Maybe it's not a problem, because if the user synced many branches, they have already waited for a long time for the git operations. After that, a little more time to wait for db operations may not matter.


Update:

I tested both the main branch and this branch by creating 1000 branches and pushing them to the remote at once.

Steps
  1. Created 1000 branches
for i in $(seq 1 1000); do
  git checkout -b branch-$i main
  echo "This is branch $i" > branch-$i.txt
  git add --all
  git commit -m "Commit for branch $i"
done
  1. Delete and create an empty repo on Gitea.
  2. Push
time git push origin --all

Result:

main this PR delta
1 137.98s 178.18s
2 135.78s 180.32s
3 136.19s 188.29s
avg 136.65s 182.26s +33.38%

So, this PR will increase the time to push by about 30%. It's hard to say if the result is good or bad. I think this is a matter of trade-offs.

I am surprised it took so long, and I tried to push the repo to gitea.com, it also took 108.76s.


Update:

After 49f224a , the performance has been improved.

The new result:

main this PR delta
1 137.98s 139.79s
2 135.78s 138.22s
3 136.19s 136.41s
avg 136.65s 138.14s +1.09%

In this way, the performance impact is very limited.

@pull-request-size pull-request-size bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 1, 2024
services/repository/branch.go Outdated Show resolved Hide resolved
@lunny
Copy link
Member

lunny commented Mar 1, 2024

Looks like the default branch of the repository hasn't been set correctly for the first push

@wolfogre
Copy link
Member Author

wolfogre commented Mar 4, 2024

Looks like the default branch of the repository hasn't been set correctly for the first push

I cannot reproduce; I believe it's unrelated.

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Mar 4, 2024
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Mar 6, 2024
@wolfogre wolfogre added the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Mar 6, 2024
@wolfogre wolfogre merged commit 5bdf805 into go-gitea:main Mar 6, 2024
26 checks passed
@GiteaBot GiteaBot removed the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Mar 6, 2024
silverwind pushed a commit that referenced this pull request Mar 6, 2024
Detect broken git hooks by checking if the commit id of branches in DB
is the same with the git repo.

It can help #29338 #28277 and maybe more issues.

Users could complain about actions, webhooks, and activities not
working, but they were not aware that it is caused by broken git hooks
unless they could see a warning.

<img width="1348" alt="image"
src="https://github.com/go-gitea/gitea/assets/9418365/2b92a46d-7f1d-4115-bef4-9f970bd695da">


It should be merged after #29493. Otherwise, users could see a ephemeral
warning after committing and opening the repo home page immediately.

And it also waits for #29495, since the doc link (the anchor part) will
be updated.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
silverwind added a commit to silverwind/gitea that referenced this pull request Mar 6, 2024
* origin/main: (29 commits)
  Tweak actions color and borders (go-gitea#29640)
  Add download URL for executable files (go-gitea#28260)
  Move all login and account creation page labels to be above inputs (go-gitea#29432)
  Avoid issue info panic (go-gitea#29625)
  Cache repository default branch commit status to reduce query on commit status table (go-gitea#29444)
  Avoid unexpected panic in graceful manager (go-gitea#29629)
  Add a link for the recently pushed branch notification (go-gitea#29627)
  Fix wrong header of org project view page (go-gitea#29626)
  Detect broken git hooks (go-gitea#29494)
  Sync branches to DB immediately when handle git hook calling (go-gitea#29493)
  Fix wrong line number in code search result (go-gitea#29260)
  Make wiki default branch name changable (go-gitea#29603)
  A small refactor for agit implementation (go-gitea#29614)
  Update Twitter Logo (go-gitea#29621)
  Fix 500 error when adding PR comment (go-gitea#29622)
  Run editorconfig-checker on `locale_en-US.ini` (go-gitea#29608)
  bump protobuf module (go-gitea#29617)
  Add ac claim for old docker/build-push-action@v3 / current buildx gha cache (go-gitea#29584)
  Skip email domain check when admins edit user emails (go-gitea#29609)
  Improve natural sort (go-gitea#29611)
  ...
zjjhot added a commit to zjjhot/gitea that referenced this pull request Mar 7, 2024
* giteaofficial/main:
  Use strict protocol check when redirect (go-gitea#29642)
  Update various logos and unify their filenames (go-gitea#29637)
  Tweak actions color and borders (go-gitea#29640)
  Add download URL for executable files (go-gitea#28260)
  Move all login and account creation page labels to be above inputs (go-gitea#29432)
  Avoid issue info panic (go-gitea#29625)
  Cache repository default branch commit status to reduce query on commit status table (go-gitea#29444)
  Avoid unexpected panic in graceful manager (go-gitea#29629)
  Add a link for the recently pushed branch notification (go-gitea#29627)
  Fix wrong header of org project view page (go-gitea#29626)
  Detect broken git hooks (go-gitea#29494)
  Sync branches to DB immediately when handle git hook calling (go-gitea#29493)
  Fix wrong line number in code search result (go-gitea#29260)
  Make wiki default branch name changable (go-gitea#29603)
  A small refactor for agit implementation (go-gitea#29614)
  Update Twitter Logo (go-gitea#29621)
  Fix 500 error when adding PR comment (go-gitea#29622)
@wolfogre wolfogre mentioned this pull request Mar 11, 2024
lunny pushed a commit that referenced this pull request Mar 11, 2024
Follow #29493.

Sync branches to DB first, then trigger push events.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/enhancement An improvement of existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants