Migration stuck for some repos

[Codeberg-org]

• Gitea version (or commit ref): 1.11.0+dev-155-gdce22efbe
• Git version: whatever runs on try.gitea.io
• Operating system: whatever runs on try.gitea.io
• Database (use [x]): whatever runs on try.gitea.io
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL) : https://try.gitea.io/pcsv/Pix-Art-Messenger
  • No
  • Not relevant
• Log gist:

Description

Trying to migrate https://github.com/kriztan/Pix-Art-Messenger with issues, wiki, pull requests and releases to Gitea, the migration is stuck, and the repo inaccessible.

No Settings tab either, so it is impossible to cancel the migration or delete the broken repo.

Screenshots

[ashimokawa]

@lunny
This is clearly a kind/bug.

The migration fails silently and the animation is running forever.
Not even the first issue (which is a PR with deleted branches) is migrated properly.

[lunny]

I think we can add a cancel button here.

[ashimokawa]

@lunny
Sure, but that does not fix the underlying problem. The migrations fails somehow, always, reproducible with the repo mentioned above https://github.com/kriztan/Pix-Art-Messenger

[6543]

it works on gitea.com ( 1.11.0+dev-110-gc4bc5abda)
and 1.9.5 (with a timeout message but it works)

and https://try.gitea.io/abcKing/s

[6543]

maby some space limitations based on user? or something like that?

[6543]

@lunny what happen wen a user run out of granted space when he has a migration job?

[lunny]

@6543 The repository should be deleted if any error found on the migrating. And a notice will be displayed on admin panel notices.

[lunny]

As an enhancements for the migration process, I will send another PRs to split the process as serval parts.

[ashimokawa]

@6543
No it does not work when migration with issues and pr, please read the original issue description:

with issues, wiki, pull requests and releases to Gitea, the migration is stuck, and the repo inaccessible.

EDIT: This also has nothing to do with space restrictions, happens on all gitea instances, always. I think above repository is perfect for testing and fixing the problem.

Maybe it was not wise to create a 2-in-1 issue, actually these should be two:

• FULL migration of https://github.com/kriztan/Pix-Art-Messenger ALWAYS fails
• can't cancel failed migration

[lafriks]

Imho problem is that if Gitea is restarted while migration is running it will be left unfinished

[ashimokawa]

@lafriks
@lunny

No, it happens always on any instance, not only when restarting.

We did not restart try.gitea.io (we can't) and it happens on our test instance also. You can reproduce it easily by migrating https://github.com/kriztan/Pix-Art-Messenger

[lunny]

@ashimokawa I will try that repository.

[mrsdizzie]

I started to debug this a bit and ran out of time but if it is helpful for you @lunny what is happening is that at some point inserting a batch of releases from this repo is throwing the error:

UNIQUE constraint failed: release.repo_id, release.tag_name

And then it goes into rollback but never comes out. It seems to get here:

gitea/vendor/xorm.io/xorm/session_delete.go

Line 211 in 45f2192

res, err := session.exec(realSQL, condArgs...)

And then the it hits this code and seems to never really do anything after that:

gitea/vendor/xorm.io/xorm/session_raw.go

Lines 177 to 179 in 45f2192

	if !session.isAutoCommit {
	return session.tx.ExecContext(session.ctx, sqlStr, args...)
	}

That runs some code in the specific driver for the database, but I'm able to reproduce on sqlite3 and assume try.gitea.io and codeberg are using something else so I wonder if it just isn't returning an error where it should at some point. Not familiar with xorm at all though :-/

I tried with lafriks patch too but seems unrelated in this case (though helpful in general)

[ashimokawa]

@mrsdizzie
Yep, codeberg.org runs mariadb and not sqlite.

I assume you tried the repo I mentioned with full migration?

The only thing a saw in logs is that a rollback failed (timout? stuck?), I really do not remember because I was also in a hurry.

[mrsdizzie]

I assume you tried the repo I mentioned with full migration?

Yes, it is specifically the releases that are causing the initial migration of this particular repo to fail and then there is a separate (and much worse) problem of the rollback not finishing properly after the migration fails.

[lunny]

@mrsdizzie Thanks for your work!
So

UNIQUE constraint failed: release.repo_id, release.tag_name

means there are duplicated releases on that repository??? I will take a look at that code.

[ashimokawa]

I can confirm that migrating the repository without releases works.

[techknowlogick]

It seems unusual that there would be a duplicate release, as I'm not sure that git allows same tag to be used twice. Regardless of that assumption, I am able to replicate same error as above, but able to have successful migration when release is unchecked (I tested on sqlite as that's what I am using on my dev instance, but will try with postgres later, as that's whats on my prod instance).

[mrsdizzie]

It could be a bug in the migrating that perhaps tries to insert the same release twice as well.

The 100% certain bug is that the rollback isn't finishing but I didn't know enough about xorm etc to really figure out why that is happening here.

[lunny]

#9319 shoud fix another bug but not this one. But @ashimokawa , you can still try it.

[mrsdizzie]

The specific example from this issue should be fixd, see:

https://try.gitea.io/mrsdizzie/Pix-Art-Messenger

But the original issue exposed a lower level bug in rollback that is still present (though not triggered anymore through this issue) and should probably stay open until that is fixed.

Plus there are the other aspects like not being able to cancel a migration that is stuck for any reason which should be possible

[lunny]

@mrsdizzie it could be deleted from admin panel currently. :)

[ashimokawa]

@lunny
I can confirm that the specific case I opened this issue for is fixed. So normally I would close this one.

[ashimokawa]

@lunny @mrsdizzie

Stuck repos can be deleted by the user by just manually navigating to his settings, even in that stuck state.

https://try.gitea.io/mrsdizzie/Pix-Art-Messenger/settings (example, I tried this with a real stuck repo)

Don't know if that is good or not.

[lunny]

@ashimokawa :)

[mrsdizzie]

There is no link to that settings page if it is stuck migrating (though visiting the URL directly works). So it is good but could be better.

I think the general bug still exists that it shouldn't be possible for it to get stuck at all (it should work or fail), but now this example will not trigger it so perhaps harder to reproduce. @lunny maybe if you have time to look at the xorm code from above, if not then I'm not sure : ) Glad that this example helped fix the bug causing it anyway so thanks!

[saurasambit]

I faced the issue while migrating https://github.com/jitsi/jitsi-meet.
See https://try.gitea.io/dashadsh/jitsi-meet for the reproduced problem.

[dkebler]

Just had this happen to me after migrating many repos to my backup site. How does one get rid of this aborted migration? Not via the webui as there is no access to "settings" since the migration was not completed. I restarted my instance and web server but it must be in the db itself. How can on repair the git.db?

Of Note. I see that I had 7000+ pages of system notices all one date all about not being able to access the main git from which the migrations come. This backup install has been acting funny too. Slow page loads, popup notices above loosing data if moving to another page which I wasn't. I'm guessing this is a corrupted data base maybe from loading up on all those system notifications. 7000 pages is excessive no :-). Luckily I have daily backups and a backup from when I upgraded in Jan. Will try those if I can solve this otherwise.

Of real note is that my main gitea db is only 2mb
but the backup gitea (of the same repos as migrations) is 47mb! Maybe 7000 pages i.e. 45mb of system notifications? Anyway something has gone horribly wrong with the db of my backup gitea install. It's look more and more like the logging system gone amok. Today's gitea.log is 179mb! with unending entries like so
2020/03/16 00:00:11 ...eue/queue_wrapped.go:69:setInternal() [W] [Attempt: 706] Failed to create queue: level for task cfg: {/mnt/data/git/data/queues/task 1000 20 1 6 1s 5m0s 5 task-level} error: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000020]

[dkebler]

I was able to delete all the notifications via the UI and under site administration /admin/repos I was able to delete the offending migration.
I then attempted to make anew the migration and it was the same issue :-(

I have been using 1.1 now going to try upgrading to 1.96.

[dkebler]

I upgraded both installs to 1.9.6. I deleted and recreated the offending repo on the main gitea install and pushed it again from local. Then tried again to make a migration on my backup gitea from main gitea install and YEA that worked.

So only 2 issues for me now.

1. that my git.db on my backup install is still 47mb. Apparently sqlite has no way to compact after deleting all those pages of notifications? https://www.recoveryandmanagement.com/repair-sqlite-database-manually/

2. Why did gitea create 7000 pages worth of notifications/logs. Was that 'bug' fixed going from 1.1 to 1.9.6?

[lunny]

That admin notices could be deleted after your problem resolved.

[guillep2k]

@dkebler Sincerely, 1.1 was so long ago... so many things have changed since then: refactors, rewriting, updating libraries... I'd even suggest you upgrade to 1.11.3 while you're at it; a lot of good features and bugfixes were added since 1.9.6 already.

[qdzhaov]

I found the bug like this.

• Gitea version (or commit ref):1.11.4
• Git version:2.17.1
• Operating system:linux
• Database : SQLite
  When I Migration a Repositories from gogs(,need a user name a password ,but I don't give,so
  the task is in dead loop.(This bug is also in gogs.)

[guillep2k]

@qdzhaov What version of Gogs are you migrating from? (I take it's not an upgrade but a site-to-site migration)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

[theAkito]

I've upgraded from 6af13db to 15d3aa7, because I had this issue on the former version. Now the issue is gone and mirroring works.

---

I encountered that issue with another repository. The mirroring worked after restarting Gitea.

[1byte2bytes]

I encounter the issue of migration getting stuck while trying to mirror https://github.com/apache/spamassassin to my Gitea instance, using Gitea 1.14.8 on Debian 10. Restarting the Gitea server did not fix the importing for me.

[jolheiser]

using Gitea 1.14.8 on Debian 10

Currently Gitea is version 1.12.4 heading towards 1.13.

[6543]

@1byte2bytes found a bug related to DefaultBranch != "master" - it could have hit you ...

(#12843)

[6543]

ok since Pix-Art-Messenger can be migrated (tested) I think it's time to close this issue

@1byte2bytes if gitea v1.12.5 has ben released and it still does not work, feel free to create an issue :)
and for cancel suggestions: (#12845)