Gitlab importer worker crashes

[Codeberg-org]

• Gitea version (or commit ref): 1.12.5
• Git version: 2.20.1
• Operating system: Linux/Debian stable
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

We see some dozen worker crashes seemingly related to the gitlab importer in our logs. Related issue: https://codeberg.org/Codeberg/Community/issues/304

Logs

2020/10/09 11:52:43 modules/task/task.go:51:handle() [E] Run task failed: Handler crashed with error: /build/go/src/runtime/panic.go:212 (0x44d709)
        /build/go/src/runtime/signal_unix.go:695 (0x44d558)
        /build/gitea/src/code.gitea.io/gitea/modules/migrations/gitlab.go:126 (0x19089c5)
        /build/gitea/src/code.gitea.io/gitea/modules/migrations/base/downloader.go:58 (0x1872e16)
        /build/gitea/src/code.gitea.io/gitea/modules/migrations/migrate.go:73 (0x190c76d)
        /build/gitea/src/code.gitea.io/gitea/modules/task/migrate.go:99 (0x192042f)
        /build/gitea/src/code.gitea.io/gitea/modules/task/task.go:28 (0x1920c97)
        /build/gitea/src/code.gitea.io/gitea/modules/task/task.go:50 (0x1920f51)
        /build/gitea/src/code.gitea.io/gitea/modules/queue/workerpool.go:383 (0x168de51)
        /build/gitea/src/code.gitea.io/gitea/modules/queue/workerpool.go:238 (0x16901d2)
        /build/go/src/runtime/asm_amd64.s:1373 (0x469720)

[zeripath]

This is very odd...

The line marked responsible for the panic is:

gitea/modules/migrations/gitlab.go

Line 126 in aa1d9ef

g.ctx = ctx

Which can only normally panic if g is nil.

gitea/modules/migrations/base/downloader.go

Line 58 in aa1d9ef

d.Downloader.SetContext(ctx)

Which would imply d.Downloader was nil here

gitea/modules/migrations/migrate.go

Line 73 in aa1d9ef

downloader.SetContext(ctx)

Which would imply that downloader.Downloader was nil here.

gitea/modules/migrations/migrate.go

Line 70 in aa1d9ef

downloader = base.NewRetryDownloader(downloader, setting.Migrations.MaxAttempts, setting.Migrations.RetryBackoff)

Meaning downloader was nil here.

But that doesn't make sense because:

gitea/modules/migrations/migrate.go

Line 52 in aa1d9ef

if downloader == nil {

Should stop that...

However if we look again at the panic stack we see:

build/go/src/runtime/signal_unix.go:695

Which makes me suspicious that we're not seeing an npe here but rather makes me suspicious that it's something to do with the context and its channel being assigned to another struct.

I wonder if you're hitting some notify limit in regards to channels.There may be a hard limit of how many channels can be assigned to structs.

If there is we're in for a world of trouble as we'll have to consider migrating off channels on to atomic flags and some kind of polling but also consider if you've hit a go bug.

(There is an issue in that we've not got the recover result logged here which is strange because I was sure it should be...)

[zeripath]

What version of go are you building with?

[zeripath]

(yup there's an issue with our recoverer here meaning we're not logging the underlying problem here.

gitea/modules/task/migrate.go

Line 43 in 9066d09

fmt.Fprintf(&buf, "Handler crashed with error: %v", log.Stack(2))

Needs to log e as a %v argument just before the stack.)

[Codeberg-org]

What version of go are you building with?

go1.14.9

[6543]

I have found 2 bugs (only in v1.12.x):

• lable color with short-code do not work (e.g. #234) must be normalized to #223344
• git clone from private repos pass token wrong way :
  GitHub Like it that way:
  

  gitea/modules/migrations/gitea.go

  Line 101 in a0a77c9

  u.User = url.UserPassword(opts.AuthUsername, opts.AuthPassword)

GitLab like token to be passed via:

 u.User = url.UserPassword("", opts.AuthUsername) 

there is no option to pass a token as token-option to the migrator in v1.12