Sync issues, PRs, labels, milestones, comments from GitHub mirrors

[harryzcy]

This PR adds support for one-way syncing of topics, milestones, labels, issues, pull requests, comments, reviews, releases, and reactions from pull mirror (GitHub) to Gitea.
Closes #18369.

Downloader interface is extended with GetNewIssues, GetNewPullRequests, since new issues and pull requests should be appended/updated to the existing repository.

Uploader interface is extended with UpdateTopics, UpdateMilestones, UpdateLabels, PatchReleases, PatchComments, PatchIssues, and PatchPullRequests. All Update* should replace existing data and Patch* should append/update but should not delete existing data.

The above methods will be implement here for GitHub, or possibly also GitLab, in this pull request.

In order to make the updates work, we would need to uniquely identify a Topic, Milestone, Label, Release, Comments, Issue and PullRequest. The following list shows how they are identified:

• Topic - Name (it's just a string, unrelated to any other things)
• Milestone - OriginalID*
• Label - OriginalID*
• Release - OriginalID*
• Comments - OriginalID*
• Issue - IssueID and Index
• PullRequest - IssueID and Index
• Reviews - OriginalID*
• Reactions - IssueID, CommentID and Type

OriginalID*: additional column added to database, used to track ID from the source

Screenshot of toggles to set which migration Items should be synced (the only change is now when mirror option is checked, the migration items are still available for supported platforms):

[lunny]

You need to implement UpdateIssues, UpdateReactions, UpdateComments and etc. functions. You need to find the unique external id to update them.

[codecov-commenter]

Codecov Report

Merging #20311 (dd20d7d) into main (f521e88) will decrease coverage by 0.43%.
The diff coverage is 20.20%.

❗ Current head dd20d7d differs from pull request most recent head 8fb9956. Consider uploading reports for the commit 8fb9956 to get more accurate results

@@            Coverage Diff             @@
##             main   #20311      +/-   ##
==========================================
- Coverage   47.14%   46.71%   -0.43%     
==========================================
  Files        1149      994     -155     
  Lines      151446   137597   -13849     
==========================================
- Hits        71397    64285    -7112     
+ Misses      71611    65414    -6197     
+ Partials     8438     7898     -540     

Impacted Files	Coverage Δ
models/repo/mirror.go	66.07% <ø> (ø)
modules/migration/null_downloader.go	23.40% <0.00%> (-8.03%)	⬇️
modules/migration/null_uploader.go	0.00% <0.00%> (ø)
modules/notification/notification.go	93.02% <ø> (+10.01%)	⬆️
routers/api/v1/repo/migrate.go	53.43% <ø> (+2.24%)	⬆️
routers/web/repo/migrate.go	40.72% <ø> (+1.91%)	⬆️
services/migrations/dump.go	39.83% <ø> (+0.55%)	⬆️
services/migrations/migrate.go	30.01% <0.00%> (-18.85%)	⬇️
models/migrate.go	17.34% <4.26%> (-28.64%)	⬇️
models/issues/review.go	48.31% <15.00%> (-4.42%)	⬇️
... and 7 more

... and 1146 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[rcarmo]

Does this include mirroring release packages as well? Right now Gitea only mirrors the tarballs, and any artifacts (like .appimages) added to GitHub releases are lost.

[harryzcy]

Does this include mirroring release packages as well? Right now Gitea only mirrors the tarballs, and any artifacts (like .appimages) added to GitHub releases are lost.

Yes, included in this PR

[siddarthkay]

This PR looks cool, I'll try a build from this branch locally to see if this fixes our needs for having a mirror set up for Github repositories that also syncs, issues, comments on issues and PRs.
Thanks @harryzcy
Looking forward to this PR getting vetted and merged in the future.

[harryzcy]

@siddarthkay Thank you for helping testing this out.

[siddarthkay]

hmm @harryzcy : I used your branch url as the source of my docker image and now I see that the options to select Issues, PRs while setting up a mirror are disabled.

Is it intended to be that way?

Here is my docker-compose.yml Incase it helps to replicate the issue.

version: "3"

networks:
  gitea:
    external: false

services:
  server:
    build:
      context: https://github.com/harryzcy/gitea.git#sync-issue-pr-and-more
      dockerfile: Dockerfile
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - GITEA__database__DB_TYPE=mysql
      - GITEA__database__HOST=db:3306
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=gitea
      - GITEA__database__PASSWD=gitea
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    image: mysql:8
    restart: always
    environment:
      - MYSQL_ROOT_PASSWORD=gitea
      - MYSQL_USER=gitea
      - MYSQL_PASSWORD=gitea
      - MYSQL_DATABASE=gitea
    networks:
      - gitea
    volumes:
      - ./mysql:/var/lib/mysql

[ajvpot]

Thanks so much for your work on this PR! I've set it up to attempt to see if it'll work for us.

I imported a private repository using a Personal Access Token. The initial sync works fine but I'm encountering this error on subsequent syncs:

gitea-1        | 2024/05/24 23:02:23 ...eb/routing/logger.go:102:func1() [I] router: completed GET /ORGNAME/REPONAME/settings for ...:0, 200 OK in 15.1ms @ setting/setting.go:95(setting.Settings)
db-1           | 2024-05-24 21:02:24.317 UTC [35] ERROR:  duplicate key value violates unique constraint "issue_index_pkey"
db-1           | 2024-05-24 21:02:24.317 UTC [35] DETAIL:  Key (group_id)=(1452) already exists.
db-1           | 2024-05-24 21:02:24.317 UTC [35] STATEMENT:  INSERT INTO issue_index (group_id, max_index) VALUES ($1, $2)
db-1           | 2024-05-24 21:02:24.317 UTC [35] ERROR:  current transaction is aborted, commands ignored until end of transaction block
db-1           | 2024-05-24 21:02:24.317 UTC [35] STATEMENT:  SELECT max_index FROM issue_index WHERE group_id=$1
gitea-1        | 2024/05/24 23:02:24 ...irror/mirror_pull.go:465:runSyncMisc() [E] runSyncMisc [repo: <Repository 1452:ORGNAME/REPONAME>]: failed to run SyncRepository: pq: current transaction is aborted, commands ignored until end of transaction block