Add support for migrating from Gitlab #9084

aqtrans · 2019-11-20T04:26:27Z

This adds a new migration interface to migrations from Gitlab, either self-hosted or hosted.

Another fix for all migrations is included, in eaf52a4.
While working on this, I noticed due to OriginalURL being set to CloneAddr in CreateMigrateTask that all migrated repos, Github as well, included the credentials in the database, as well as any "Migrated from" links in comments and such.

I didn't want to change the UI, so the self-hosted Gitlab detection is based on a '#gitlab' URL fragment added to the end of the Clone Address. Not the best solution, but it does the trick for now and doesn't interfere with the migration process.

The Gitlab calls are all powered by github.com/xanzy/go-gitlab, which has been vendored.

All objects in the migrations API are supported.
There are some odd hacks related to comments and PRs, due to PRs and Issues sharing IDs in Gitea.

Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr

CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it.

Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials.

RepoID is grabbed in NewGitlabDownloader

Properly set milestone deadline time. Consistently use Gitlab username for 'Name'.

- Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments

techknowlogick

Thank you for your contribution!! This will be so helpful for many users.

I haven't done a thorough review of this PR, but I found a few nits that would need to be resolved.

modules/migrations/gitlab.go

lunny · 2019-11-20T05:51:08Z

I think we have to add a selection on UI for self-hosted git services.

lunny · 2019-11-20T05:55:47Z

For make the PR easier to merge I think maybe you could just support gitlab.com on this PR so that you could not change the UI and move self-host part for another PR.

aqtrans · 2019-11-20T16:12:12Z

For make the PR easier to merge I think maybe you could just support gitlab.com on this PR so that you could not change the UI and move self-host part for another PR.

Ah yeah that makes sense. I'll get on that.
For a UI, I was thinking maybe a "Migrating from [enabled services]?" dropdown could be added. But I haven't dug into what changes that'd require on the backend.

aqtrans · 2019-11-20T16:37:35Z

Alright. Hopefully I did that right. Self-hosted support has been ripped out of this PR, and a new PR has been opened to re-add self-hosted support.

mrsdizzie · 2019-11-20T17:02:57Z

The fix in eaf52a4 should go it its own small bugfix PR imo and should get back ported (where as this PR can't be)

CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place

aqtrans · 2019-11-20T17:36:29Z

The fix in eaf52a4 should go it its own small bugfix PR imo and should get back ported (where as this PR can't be)

Good call. I thought about that but wasn't sure how to go about doing that haha...
Now that I'm getting the hang of git, I've gone ahead and done so. PR#9097

6543 · 2020-04-08T18:15:29Z

@aqtrans since you Updated the branch I can send new patches to your branch ... thanks

I'll look into why CI failes

6543 · 2020-04-10T00:56:29Z

@aqtrans by the way if you would give me write permissions to your repo I could push directly to this pull ...

you have to use go v1.14 when executing make vendor : aqtrans#3

6543 · 2020-04-10T01:16:58Z

Status: just tested localy: It migrates: Labels, Milestones, Branches (git rev in generel), repo-topics

and then It hangs in a loop

make vendor with go1.14

aqtrans · 2020-04-11T01:47:29Z

Thanks a lot for the help! I've merged your latest PR and granted you access to my branch.
I'll try and run a migration myself and see what's up.

6543 · 2020-04-16T15:56:01Z

Found the issues - not related to this pull: #11096

Will wait untill it is merged - after it I think this pull is ready :)

6543

🎉

lafriks · 2020-04-17T18:11:03Z

@lunny nees your approval

6543 · 2020-04-19T14:00:20Z

Ready to merge 🚀

6543 · 2020-04-19T15:11:33Z

ping lgtm

6543 · 2020-04-19T16:20:28Z

It finaly went in 🎉 - thanks to all!
it will help many to switch to gitea :)

aqtrans · 2020-04-19T16:38:15Z

Fantastic! Thanks a ton to y'all for all the help.

* First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try go-gitea#2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>

aqtrans added 14 commits November 18, 2019 17:54

First stab at a Gitlab migrations interface.

0a8bd8a

Modify JS to show migration for Gitlab

f58c065

Properly strip out #gitlab tag from repo name

b10ed23

Working Gitlab migrations!

823d181

Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr

Try #2 at trying to hide credentials.

eaf52a4

CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it.

Add go-gitlab dependency

4efa640

Vendor go-gitlab

d651d11

Use gitlab.BasicAuthClient

9b38e86

Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials.

Replaced repoPath with repoID globally.

ee4cefa

RepoID is grabbed in NewGitlabDownloader

Logging touchup

f61f9cd

Properly set private repo status.

1a47b38

Properly set milestone deadline time. Consistently use Gitlab username for 'Name'.

Add go-gitlab vendor cache

88b8918

Fix PR migrations:

22a6080

- Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments

Ensure merged PRs are closed and set with the proper time

9f316b3

techknowlogick reviewed Nov 20, 2019

View reviewed changes

modules/migrations/gitlab.go Outdated Show resolved Hide resolved

modules/migrations/gitlab.go Outdated Show resolved Hide resolved

modules/migrations/gitlab.go Outdated Show resolved Hide resolved

modules/migrations/gitlab.go Outdated Show resolved Hide resolved

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Nov 20, 2019

Remove copyright and some commented code

c993d84

lunny added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Nov 20, 2019

lunny added this to the 1.12.0 milestone Nov 20, 2019

Rip out '#gitlab' based self-hosted Gitlab support

0aa2da8

aqtrans mentioned this pull request Nov 20, 2019

Add self-hosted Gitlab migration support #9096

Closed

6543 mentioned this pull request Nov 20, 2019

Migration Summary #8689

Open

24 tasks

Hide given credentials for migrated repos.

356b5ac

CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place

aqtrans mentioned this pull request Nov 20, 2019

Hide given credentials for migrated repos. #9097

Merged

make vendor with go1.14

90b8f42

Merge pull request #3 from 6543-forks/gitlab_migrate-fix

7fec22e

make vendor with go1.14

6543 mentioned this pull request Apr 16, 2020

[BugFix] remove nil inserts in models #11096

Merged

Merge branch 'master' into gitlab-migrate

29937a9

6543 approved these changes Apr 17, 2020

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Apr 17, 2020

lafriks approved these changes Apr 17, 2020

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Apr 17, 2020

Merge branch 'master' into gitlab-migrate

4dd6e89

lunny approved these changes Apr 19, 2020

View reviewed changes

Merge branch 'master' into gitlab-migrate

a651248

Merge branch 'master' into gitlab-migrate

f5988e8

lunny merged commit 5c092eb into go-gitea:master Apr 19, 2020

6543 mentioned this pull request Apr 19, 2020

[Feature] add migration support for Reviews to gitlab migration #11137

Closed

6543 mentioned this pull request May 20, 2020

Please implement migration features Gitea -> Gitea as available for Github -> Gitea #8380

Closed

7 tasks

go-gitea locked and limited conversation to collaborators Nov 24, 2020

delvh removed the type/changelog Adds the changelog for a new Gitea version label Oct 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for migrating from Gitlab #9084

Add support for migrating from Gitlab #9084

aqtrans commented Nov 20, 2019

techknowlogick left a comment

lunny commented Nov 20, 2019

lunny commented Nov 20, 2019 •

edited

Loading

aqtrans commented Nov 20, 2019

aqtrans commented Nov 20, 2019 •

edited

Loading

mrsdizzie commented Nov 20, 2019

aqtrans commented Nov 20, 2019

6543 commented Apr 8, 2020

6543 commented Apr 10, 2020

6543 commented Apr 10, 2020 •

edited

Loading

aqtrans commented Apr 11, 2020

6543 commented Apr 16, 2020

6543 left a comment

lafriks commented Apr 17, 2020

6543 commented Apr 19, 2020

6543 commented Apr 19, 2020

6543 commented Apr 19, 2020

aqtrans commented Apr 19, 2020

Add support for migrating from Gitlab #9084

Add support for migrating from Gitlab #9084

Conversation

aqtrans commented Nov 20, 2019

techknowlogick left a comment

Choose a reason for hiding this comment

lunny commented Nov 20, 2019

lunny commented Nov 20, 2019 • edited Loading

aqtrans commented Nov 20, 2019

aqtrans commented Nov 20, 2019 • edited Loading

mrsdizzie commented Nov 20, 2019

aqtrans commented Nov 20, 2019

6543 commented Apr 8, 2020

6543 commented Apr 10, 2020

6543 commented Apr 10, 2020 • edited Loading

aqtrans commented Apr 11, 2020

6543 commented Apr 16, 2020

6543 left a comment

Choose a reason for hiding this comment

lafriks commented Apr 17, 2020

6543 commented Apr 19, 2020

6543 commented Apr 19, 2020

6543 commented Apr 19, 2020

aqtrans commented Apr 19, 2020

lunny commented Nov 20, 2019 •

edited

Loading

aqtrans commented Nov 20, 2019 •

edited

Loading

6543 commented Apr 10, 2020 •

edited

Loading