x/tools/gopls: regtest flakes due to hanging go commands #54461

findleyr · 2022-08-15T18:50:27Z

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

greplogs --dashboard -md -l -e 'runCmdContext' --since=2022-07-14

We recently started waiting for all go command invocations when shutting down gopls regtests. It appears that sometimes we kill the go command and still don't get a result from cmd.Wait(). For example, here:
https://build.golang.org/log/00046e0b005c7660d676a3a415561950048f756a

In that failure, the test runner looks otherwise healthy (other tests ran fast), and yet the goroutine stack clearly shows a go command hanging for 9 minutes here:
https://cs.opensource.google/go/x/tools/+/master:internal/gocommand/invoke.go;l=260;drc=f38573358cbedf46d64c5759ef41b72afcf0c5c0

@bcmills do you happen to have any idea of what might cause this?

The text was updated successfully, but these errors were encountered:

findleyr · 2022-08-15T19:05:19Z

Two observations:

We're not checking the error from Kill. If killing failed, I don't think we can expect Wait to return.
Discussion at https://groups.google.com/g/golang-nuts/c/pUaEjqd5wo8

adonovan · 2022-08-15T19:26:41Z

The only reason the kill system call can fail (at least in this situation) is when the child process has already exited, so failure of kill is unlikely to be the culprit. More likely kill terminated the go process itself, but not the tree of processes rooted at it. If one of them (a test?) retains an open file descriptor to the stdout pipe created by os/exec then the cmd.Run operation will hang indefinitely.

To dig further, we could add logic to run during the failure (on linux) that does ps -e f and ls -l /proc/*/fd/* to show the process tree and their open files.

bcmills · 2022-08-16T15:05:57Z

Please do check the error from Kill — that would at least give a useful data point for diagnostics. (This may be closely related to ~~#53400~~ #50187.)

bcmills · 2022-09-14T14:25:22Z

greplogs -l -e 'panic: test timed out(?:.*\n)*goroutine \d+ \[.*, \d+ minutes\]:\n.*runCmdContext' --since=2022-08-15 --details

(66 matching logs)

findleyr · 2022-09-14T14:43:14Z

Interesting, I was debugging this in https://go.dev/cl/424075. On windows, our call to Process.Kill() is failing with "invalid argument":
https://storage.googleapis.com/go-build-log/e2f60200/windows-386-2008_71ad7007.log

A bit of googling suggests that this is because we can't kill subprocesses on windows.

@bcmills any advice for how to properly kill the go command on windows?

findleyr · 2022-09-14T15:08:28Z

After reading the source a bit more: this is EINVAL, which appears to mean that the Process.wait() has exited and the handle released, so this is a race, although it is surprising that we hit it so reliably.

gopherbot · 2022-09-14T15:17:54Z

Change https://go.dev/cl/424075 mentions this issue: internal/gocommand: add instrumentation for hanging go commands

bcmills · 2022-09-14T16:44:06Z

any advice for how to properly kill the go command on windows?

Can't be done without creating a whole extra process group, unfortunately. (Probably we should add a side-channel — perhaps an open file descriptor or a pidfile? — to request clean shutdown on Windows.)

When a go command hangs during gopls regression tests, print out additional information about processes and file descriptors. For golang/go#54461 Change-Id: I92aa4665e9056d15a274c154fce2783bed79718e Reviewed-on: https://go-review.googlesource.com/c/tools/+/424075 gopls-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Alan Donovan <adonovan@google.com> Run-TryBot: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2022-09-15T14:47:42Z

Change https://go.dev/cl/431075 mentions this issue: internal/gocommand: tweak debugging for hanging go commands

Add a TODO and wait for a shorter period of time following Kill, per post-submit advice from bcmills on CL 424075. For golang/go#54461 Change-Id: Ia0e388c0119660844dad32629ebca4f122fded12 Reviewed-on: https://go-review.googlesource.com/c/tools/+/431075 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Robert Findley <rfindley@google.com> gopls-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Bryan Mills <bcmills@google.com>

findleyr · 2022-09-16T17:24:36Z

A hit!
https://build.golang.org/log/4eea0e5bfe425fcd097a49fcb92fd015fa88f383

adonovan · 2022-09-16T17:32:38Z

Nice. Well, that test process seems very much alive, falsifying my hypothesis.

findleyr · 2022-09-16T17:35:17Z

13032 12910 (compile)
 9802 13032 go list -modfile=/tmp/workdir/tmp/go.8f5d4cc245da9790bda529e64ae3e7

Looks like the hanging go command is in the middle of a compile. Wish we had the full subprocesss command line -- I'll look into that.

Not sure how to interpret the fstat output.

adonovan · 2022-09-16T17:36:24Z

That's a dead cmd/compile process: there's no command because argv has been destroyed along with the rest of the address space. Perhaps the go list parent simply hasn't called waitpid yet, so the process table entry has to be retained. I suspect the problem is in go list.

findleyr · 2022-09-16T17:38:43Z

Aha, thanks (excuse my ps noobness).

Note that we instrumented this panic in two places: once before Kill() and once after. This is before, so I don't think it falsifies your hypothesis, if I'm understanding correctly.

bcmills · 2022-09-16T18:06:58Z

That one is netbsd, which could plausibly be explained as #50138. (I'd rather we wait for one of these on a platform without a known issue for Wait before we do too much digging.)

findleyr · 2022-09-17T12:18:46Z

greplogs --dashboard -md -l -e 'HANGING GO' --since=2022-09-15

2022-09-17T02:56:51-4d18923-cc1b20e/netbsd-amd64-9_0
2022-09-16T17:59:31-a61f20e-a84f46a/netbsd-amd64-9_0
2022-09-16T16:33:11-a61f20e-8451529/netbsd-386-9_0
2022-09-16T14:49:13-a61f20e-b35b4a3/netbsd-amd64-9_0

Still only netbsd. Posting the greplogs query here for future reference.

bcmills · 2022-09-21T20:04:36Z

Ooh, nice! https://go.dev/issue/55323#issuecomment-1254107802 has a darwin/arm64 hang. 😃

gopherbot · 2024-12-23T19:27:00Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2024-12-23 17:52 x_tools-gotip-openbsd-amd64 tools@b75baa00 go@b9955f0a x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.239377596s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-1232823115/TestRegenerateCgo/default/work/... builtin
		pid:50272

goroutine 5700 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc00ab98830?, 0xc00ab986cc?, 0x1c89920?}, 0xc0078c4600)
	/home/swarming/.swarming/w/ir/x/w/targetrepo277034074/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x14be7e0, 0xc00e2c88a0}, 0xc0078c4600)
	/home/swarming/.swarming/w/ir/x/w/targetrepo277034074/internal/gocommand/invoke.go:391 +0x4ab
...
golang.org/x/tools/go/packages.goListDriver(0xc009932548, 0xc00992da20, {0x0, 0x0}, {0xc00992d960, 0x2, 0x0?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo277034074/go/packages/golist.go:210 +0x7ff
golang.org/x/tools/go/packages.defaultDriver.func1(0x0?, {0xc00992d960?, 0x0?, 0x48b7e0?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo277034074/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo277034074/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5714
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x93

— watchflakes

gopherbot · 2024-12-23T20:27:05Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2024-12-23 19:36 x_tools-go1.24-openbsd-amd64 tools@851152fe release-branch.go1.24@16afa6a7 x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.11811373s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-1697677723/TestRegenerateCgo/default/work/... builtin
		pid:6032

goroutine 5762 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc0095f0830?, 0xc0095f06cc?, 0x1c88900?}, 0xc008e74480)
	/home/swarming/.swarming/w/ir/x/w/targetrepo519391724/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x14bd638, 0xc00e572bd0}, 0xc008e74480)
	/home/swarming/.swarming/w/ir/x/w/targetrepo519391724/internal/gocommand/invoke.go:391 +0x4ab
...
golang.org/x/tools/go/packages.goListDriver(0xc0065b3a48, 0xc00ab601e0, {0x0, 0x0}, {0xc00ab60120, 0x2, 0xffffffffffffffff?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo519391724/go/packages/golist.go:210 +0x7ff
golang.org/x/tools/go/packages.defaultDriver.func1(0x8?, {0xc00ab60120?, 0x0?, 0xc00ab600e0?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo519391724/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo519391724/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5746
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x93

— watchflakes

gopherbot · 2024-12-27T17:52:40Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2024-12-27 16:23 x_tools-gotip-openbsd-amd64 tools@412b5e97 go@d7c3e93c x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.18342928s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-2680724534/TestRegenerateCgo/default/work/... builtin
		pid:46196

goroutine 5632 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc00b4ae830?, 0xc00b4ae6cc?, 0x1c94ac0?}, 0xc0093fe600)
	/home/swarming/.swarming/w/ir/x/w/targetrepo4094380473/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x14c53e0, 0xc00e3bc7b0}, 0xc0093fe600)
	/home/swarming/.swarming/w/ir/x/w/targetrepo4094380473/internal/gocommand/invoke.go:391 +0x4ab
...
golang.org/x/tools/go/packages.goListDriver(0xc00b48c548, 0xc00b49ce60, {0x0, 0x0}, {0xc00b49cd80, 0x2, 0xffffffffffffffff?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo4094380473/go/packages/golist.go:210 +0x7ff
golang.org/x/tools/go/packages.defaultDriver.func1(0x8?, {0xc00b49cd80?, 0x0?, 0xc00b49cd40?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo4094380473/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo4094380473/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5820
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x93

— watchflakes

gopherbot · 2025-01-02T17:03:08Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-02 15:44 x_tools-go1.24-openbsd-amd64 tools@bf516ba1 release-branch.go1.24@16afa6a7 x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.301186584s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-4246494676/TestRegenerateCgo/default/work/... builtin
		pid:17555

goroutine 5809 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc008d1c830?, 0xc008d1c6cc?, 0x1c9ab60?}, 0xc008f60180)
	/home/swarming/.swarming/w/ir/x/w/targetrepo3534879136/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x14c8f98, 0xc00e2b00f0}, 0xc008f60180)
	/home/swarming/.swarming/w/ir/x/w/targetrepo3534879136/internal/gocommand/invoke.go:391 +0x4ab
...
golang.org/x/tools/go/packages.goListDriver(0xc00a8b8548, 0xc00b046f00, {0x0, 0x0}, {0xc00b046e40, 0x2, 0xffffffffffffffff?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo3534879136/go/packages/golist.go:210 +0x7ff
golang.org/x/tools/go/packages.defaultDriver.func1(0x8?, {0xc00b046e40?, 0x0?, 0xc00b046e00?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo3534879136/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo3534879136/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5806
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x93

— watchflakes

gopherbot · 2025-01-07T00:19:35Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-06 17:23 x_tools-go1.24-openbsd-amd64 tools@1743d1a7 release-branch.go1.24@16afa6a7 x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.082129642s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-3176950313/TestRegenerateCgo/default/work/... builtin
		pid:33504

goroutine 5560 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc00910a830?, 0xc00910a6cc?, 0x1c99c60?}, 0xc008338c00)
	/home/swarming/.swarming/w/ir/x/w/targetrepo3793421859/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x14c7ca8, 0xc00dfdf2f0}, 0xc008338c00)
	/home/swarming/.swarming/w/ir/x/w/targetrepo3793421859/internal/gocommand/invoke.go:391 +0x4ab
...
golang.org/x/tools/go/packages.goListDriver(0xc0072bbb28, 0xc00b288a00, {0x0, 0x0}, {0xc00b288940, 0x2, 0xffffffffffffffff?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo3793421859/go/packages/golist.go:210 +0x7ff
golang.org/x/tools/go/packages.defaultDriver.func1(0x8?, {0xc00b288940?, 0x0?, 0xc00b288900?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo3793421859/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo3793421859/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5570
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x93

— watchflakes

gopherbot · 2025-01-07T19:19:27Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-06 23:35 x_tools-go1.22-windows-amd64-race tools@ee69ea24 release-branch.go1.22@8f3f22ee x/tools/gopls/internal/telemetry.TestTelemetry/default#01 [ABORT] (log)

=== RUN   TestTelemetry/default#01
panic: detected hanging go command (golang/go#54461); waited 1m0.0030241s
		command:C:\b\s\w\ir\x\w\gopath\pkg\mod\golang.org\toolchain@v0.0.1-go1.23.4.windows-amd64\bin\go.exe list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- ./ builtin
		pid:6424

goroutine 184 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc00666c320?, 0xc00666c1ac?, 0x1420f5900?}, 0xc0067a6180)
	C:/b/s/w/ir/x/w/targetrepo2481765731/internal/gocommand/invoke.go:458 +0x14d
golang.org/x/tools/internal/gocommand.runCmdContext({0x1418e5468, 0xc006784510}, 0xc0067a6180)
	C:/b/s/w/ir/x/w/targetrepo2481765731/internal/gocommand/invoke.go:391 +0x6eb
...
golang.org/x/tools/go/packages.goListDriver(0xc000268548, 0xc00660d220, {0x0, 0x0}, {0xc00660d160, 0x2, 0xc0065a3e88?})
	C:/b/s/w/ir/x/w/targetrepo2481765731/go/packages/golist.go:210 +0xcff
golang.org/x/tools/go/packages.defaultDriver.func1(0xc000268548, {0xc00660d160, 0x2, 0x2})
	C:/b/s/w/ir/x/w/targetrepo2481765731/go/packages/packages.go:343 +0x85
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	C:/b/s/w/ir/x/w/targetrepo2481765731/go/packages/packages.go:390 +0xa6
golang.org/x/sync/errgroup.(*Group).Go.func1()
	C:/b/s/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x92
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 217
	C:/b/s/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x125

— watchflakes

gopherbot · 2025-01-07T22:19:04Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-07 20:06 x_tools-gotip-openbsd-amd64 tools@fc2161a7 go@b2aa18b9 x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.046311785s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-1538265364/TestRegenerateCgo/default/work/... builtin
		pid:34171

goroutine 5593 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc0084ca830?, 0xc0084ca6cc?, 0x1ca3c80?}, 0xc00986cd80)
	/home/swarming/.swarming/w/ir/x/w/targetrepo3087435509/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x14caa90, 0xc00e49a1b0}, 0xc00986cd80)
	/home/swarming/.swarming/w/ir/x/w/targetrepo3087435509/internal/gocommand/invoke.go:391 +0x4ab
...
golang.org/x/tools/go/packages.goListDriver(0xc00b5920e8, 0xc00b565e20, {0x0, 0x0}, {0xc00b565d40, 0x2, 0xffffffffffffffff?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo3087435509/go/packages/golist.go:210 +0x7ff
golang.org/x/tools/go/packages.defaultDriver.func1(0x8?, {0xc00b565d40?, 0x0?, 0xc00b565d00?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo3087435509/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo3087435509/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5590
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x93

— watchflakes

gopherbot · 2025-01-09T22:23:22Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-09 17:24 x_tools-gotip-openbsd-amd64 tools@6efe0f4b go@c7c4420a x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.036601772s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-2273633919/TestRegenerateCgo/default/work/... builtin
		pid:63081

goroutine 5607 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc0088de830?, 0xc0088de6cc?, 0x1ccae80?}, 0xc00ae0e480)
	/home/swarming/.swarming/w/ir/x/w/targetrepo1636935490/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x14e5df0, 0xc00e3be690}, 0xc00ae0e480)
	/home/swarming/.swarming/w/ir/x/w/targetrepo1636935490/internal/gocommand/invoke.go:391 +0x4ab
...
golang.org/x/tools/go/packages.goListDriver(0xc0092e4a88, 0xc00b332780, {0x0, 0x0}, {0xc00b3326c0, 0x2, 0xffffffffffffffff?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1636935490/go/packages/golist.go:210 +0x7ff
golang.org/x/tools/go/packages.defaultDriver.func1(0x8?, {0xc00b3326c0?, 0x0?, 0xc00b332680?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1636935490/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo1636935490/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5735
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x93

— watchflakes

When a go command hangs during gopls regression tests, print out additional information about processes and file descriptors. For golang/go#54461 Change-Id: I92aa4665e9056d15a274c154fce2783bed79718e Reviewed-on: https://go-review.googlesource.com/c/tools/+/424075 gopls-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Alan Donovan <adonovan@google.com> Run-TryBot: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

Add a TODO and wait for a shorter period of time following Kill, per post-submit advice from bcmills on CL 424075. For golang/go#54461 Change-Id: Ia0e388c0119660844dad32629ebca4f122fded12 Reviewed-on: https://go-review.googlesource.com/c/tools/+/431075 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Robert Findley <rfindley@google.com> gopls-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Bryan Mills <bcmills@google.com>

We're almost certain that the go process shown by ps is not the one that we're waiting for in runCmdContext, but only by printing the pid can we be sure. Updates golang/go#54461 Change-Id: I1ce9580de6ee6bc4557d76c2a6b05f5a3e2eb6c2 Reviewed-on: https://go-review.googlesource.com/c/tools/+/434637 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Alan Donovan <adonovan@google.com> gopls-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com>

As suggested by bcmills, this change increases the time between SIGINT and SIGKILL to 5s (was 1s), and also suppresses the process dump if SIGKILL returned "already done". Updates golang/go#57999 Updates golang/go#54461 Change-Id: Ie0e55a69d3bbfb4224e5f4ea272c7c2f3210e342 Reviewed-on: https://go-review.googlesource.com/c/tools/+/483215 Run-TryBot: Alan Donovan <adonovan@google.com> gopls-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

If ctx is done, that means (by definition) that the caller is no longer interested in the output of the command. Since 'go list' output can be quite long, we shouldn't waste time copying it if we no longer care about it. If the command doesn't shut down on its own in response to the initial Interrupt signal, or if it triggers a bug in the kernel leading to a deadlock, or I/O is otherwise extremely slow (perhaps due to swapping), the savings from this earlier shutdown could be significant. Even if not, the impact (or lack thereof) of this change could tell us more about what's going on. For golang/go#54461. Change-Id: I0df067cb77496dacd29497f0995da6fc1a4a37f1 Reviewed-on: https://go-review.googlesource.com/c/tools/+/484741 gopls-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Alan Donovan <adonovan@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> Commit-Queue: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

For golang/go#54461 Change-Id: I2de5a10673345342e30e50cb39359c10e8eb7319 Reviewed-on: https://go-review.googlesource.com/c/tools/+/589956 Auto-Submit: Robert Findley <rfindley@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Alan Donovan <adonovan@google.com>

gopherbot · 2025-01-13T21:45:11Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-13 20:35 x_tools-go1.23-openbsd-amd64 tools@8a5a6d75 release-branch.go1.23@1dde0b48 x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.121420804s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-1906642907/TestRegenerateCgo/default/work/... builtin
		pid:13915

goroutine 5332 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc0003e48b0?, 0xc0003e474c?, 0x1be9d20?}, 0xc008754780)
	/home/swarming/.swarming/w/ir/x/w/targetrepo2326867135/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x1469678, 0xc00a9320c0}, 0xc008754780)
	/home/swarming/.swarming/w/ir/x/w/targetrepo2326867135/internal/gocommand/invoke.go:391 +0x4cb
...
golang.org/x/tools/go/packages.goListDriver(0xc007d1a708, 0xc00991b5c0, {0x0, 0x0}, {0xc00991b500, 0x2, 0xc00bceb320?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo2326867135/go/packages/golist.go:210 +0x81f
golang.org/x/tools/go/packages.defaultDriver.func1(0x14696b0?, {0xc00991b500?, 0xc009b0bf60?, 0x40a177?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo2326867135/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo2326867135/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5493
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96

— watchflakes

gopherbot · 2025-01-22T16:50:18Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-22 14:05 x_tools-go1.23-windows-amd64-race tools@38d06313 release-branch.go1.23@ab44565b x/tools/gopls/internal/test/integration/codelens.TestUpgradeCodelens_Workspace/Upgrade_transitive_dependencies/default [ABORT] (log)

=== RUN   TestUpgradeCodelens_Workspace/Upgrade_transitive_dependencies/default
panic: detected hanging go command (golang/go#54461); waited 1m0.0038579s
		command:C:\b\s\w\ir\x\w\goroot\bin\go.exe list -m -e -json ...
		pid:500

goroutine 574 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc006ed0a68?, 0xc006ed08f4?, 0x142133960?}, 0xc006ff4480)
	C:/b/s/w/ir/x/w/targetrepo2976050623/internal/gocommand/invoke.go:458 +0x14d
golang.org/x/tools/internal/gocommand.runCmdContext({0x14191e938, 0x14217d600}, 0xc006ff4480)
	C:/b/s/w/ir/x/w/targetrepo2976050623/internal/gocommand/invoke.go:391 +0x6eb
...
golang.org/x/tools/internal/imports.newModuleResolver(0xc006dd9200, 0xc006da78a8)
	C:/b/s/w/ir/x/w/targetrepo2976050623/internal/imports/mod.go:158 +0x138f
golang.org/x/tools/internal/imports.(*ProcessEnv).GetResolver(0xc006dd9200)
	C:/b/s/w/ir/x/w/targetrepo2976050623/internal/imports/fix.go:1036 +0x47f
golang.org/x/tools/gopls/internal/cache.(*importsState).refreshProcessEnv(0xc0001e79f0)
	C:/b/s/w/ir/x/w/targetrepo2976050623/gopls/internal/cache/imports.go:266 +0xfa
golang.org/x/tools/gopls/internal/cache.(*refreshTimer).schedule.func1()
	C:/b/s/w/ir/x/w/targetrepo2976050623/gopls/internal/cache/imports.go:71 +0x7b
created by time.goFunc
	C:/b/s/w/ir/x/w/goroot/src/time/sleep.go:215 +0x45

2025-01-22 14:05 x_tools-go1.23-windows-amd64-race tools@38d06313 release-branch.go1.23@ab44565b x/tools/gopls/internal/test/integration/completion.TestFuzzFunc/default [ABORT] (log)

=== RUN   TestFuzzFunc/default
panic: detected hanging go command (golang/go#54461); waited 1m0.0037689s
		command:C:\b\s\w\ir\x\w\goroot\bin\go.exe list -mod=readonly -m -f {{.Path}}
	{{.Dir}}
	{{.GoMod}}
	{{.GoVersion}}
	{{range context.ReleaseTags}}{{if eq . "go1.14"}}{{.}}{{end}}{{end}}
	
		pid:9232

...
golang.org/x/tools/internal/imports.newModuleResolver(0xc006afd440, 0xc006956fc0)
	C:/b/s/w/ir/x/w/targetrepo2976050623/internal/imports/mod.go:123 +0x74c
golang.org/x/tools/internal/imports.(*ProcessEnv).GetResolver(0xc006afd440)
	C:/b/s/w/ir/x/w/targetrepo2976050623/internal/imports/fix.go:1036 +0x47f
golang.org/x/tools/gopls/internal/cache.(*importsState).refreshProcessEnv(0xc006af6730)
	C:/b/s/w/ir/x/w/targetrepo2976050623/gopls/internal/cache/imports.go:266 +0xfa
golang.org/x/tools/gopls/internal/cache.(*refreshTimer).schedule.func1()
	C:/b/s/w/ir/x/w/targetrepo2976050623/gopls/internal/cache/imports.go:71 +0x7b
created by time.goFunc
	C:/b/s/w/ir/x/w/goroot/src/time/sleep.go:215 +0x45

2025-01-22 15:58 x_tools-gotip-linux-ppc64le_power10 tools@9f4a509f go@f6d17c54 x/tools/gopls/internal/test/integration/completion.TestDefinition/default [ABORT] (log)

=== RUN   TestDefinition/default
DETECTED A HANGING GO COMMAND

The gopls test runner has detected a hanging go command. In order to debug
this, the output of ps and lsof/fstat is printed below.

See golang/go#54461 for more details.

ps axo ppid,pid,command:
-------------------------
...
panic: running lsof: exec: "lsof": executable file not found in $PATH

goroutine 25768 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc01197e6f8?, 0xc01197e594?, 0x189c2c0?}, 0xc00d8e8180)
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:455 +0x390
golang.org/x/tools/internal/gocommand.runCmdContext({0x1106d30, 0xc011055260}, 0xc00d8e8180)
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:391 +0x3b8
golang.org/x/tools/internal/gocommand.(*Invocation).run(0xc01197eae8, {0x1106d30, 0xc011055260}, {0x10fc4c0, 0xc011055380}, {0x10fc4c0, 0xc0110553b0})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:285 +0xc3c
golang.org/x/tools/internal/gocommand.(*Invocation).runWithFriendlyError(0xc01197eaa0?, {0x1106d30, 0xc011055260}, {0x10fc4c0?, 0xc011055380?}, {0x10fc4c0, 0xc0110553b0})
...
golang.org/x/tools/go/packages.goListDriver(0xc008333188, 0xc0119842e0, {0x0, 0x0}, {0xc011984220, 0x2, 0x0?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/go/packages/golist.go:210 +0x770
golang.org/x/tools/go/packages.defaultDriver.func1(0x8?, {0xc011984220?, 0x0?, 0xc00f57b410?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/go/packages/packages.go:343 +0x6c
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/go/packages/packages.go:390 +0x6c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x60
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 25648
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0xb4

2025-01-22 15:58 x_tools-gotip-linux-ppc64le_power10 tools@9f4a509f go@f6d17c54 x/tools/gopls/internal/test/integration/diagnostics.TestNoMod/without_workspace_module/default [ABORT] (log)

=== RUN   TestNoMod/without_workspace_module/default
DETECTED A HANGING GO COMMAND

The gopls test runner has detected a hanging go command. In order to debug
this, the output of ps and lsof/fstat is printed below.

See golang/go#54461 for more details.

ps axo ppid,pid,command:
-------------------------
...
panic: running lsof: exec: "lsof": executable file not found in $PATH

goroutine 25349 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc0065b26f8?, 0xc0065b2594?, 0x188c600?}, 0xc00e27cd80)
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:455 +0x390
golang.org/x/tools/internal/gocommand.runCmdContext({0x1101358, 0xc00f31ab70}, 0xc00e27cd80)
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:391 +0x3b8
golang.org/x/tools/internal/gocommand.(*Invocation).run(0xc0065b2ae8, {0x1101358, 0xc00f31ab70}, {0x10f6b80, 0xc00f31ac90}, {0x10f6b80, 0xc00f31acc0})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:285 +0xc3c
golang.org/x/tools/internal/gocommand.(*Invocation).runWithFriendlyError(0xc0065b2aa0?, {0x1101358, 0xc00f31ab70}, {0x10f6b80?, 0xc00f31ac90?}, {0x10f6b80, 0xc00f31acc0})
...
golang.org/x/tools/go/packages.goListDriver(0xc0092280e8, 0xc00ef91260, {0x0, 0x0}, {0xc00ef911a0, 0x2, 0xc00f030980?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/go/packages/golist.go:210 +0x770
golang.org/x/tools/go/packages.defaultDriver.func1(0x0?, {0xc00ef911a0?, 0x1101390?, 0xc00f015a90?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/go/packages/packages.go:343 +0x6c
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/go/packages/packages.go:390 +0x6c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x60
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 25347
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0xb4

2025-01-22 15:58 x_tools-gotip-linux-ppc64le_power10 tools@9f4a509f go@f6d17c54 x/tools/gopls/internal/test/integration/misc.TestGoToStdlibDefinition_Issue37045/default [ABORT] (log)

=== RUN   TestGoToStdlibDefinition_Issue37045/default
DETECTED A HANGING GO COMMAND

The gopls test runner has detected a hanging go command. In order to debug
this, the output of ps and lsof/fstat is printed below.

See golang/go#54461 for more details.

ps axo ppid,pid,command:
-------------------------
...
panic: running lsof: exec: "lsof": executable file not found in $PATH

goroutine 13472 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc0000471c0?, 0xc00004705c?, 0x194d160?}, 0xc0152e8000)
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:455 +0x390
golang.org/x/tools/internal/gocommand.runCmdContext({0x11824e0, 0xc0152e25a0}, 0xc0152e8000)
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:391 +0x3b8
golang.org/x/tools/internal/gocommand.(*Invocation).run(0xc0000475b0, {0x11824e0, 0xc0152e25a0}, {0x1177c20, 0xc0152e26c0}, {0x1177c20, 0xc0152e26f0})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/gocommand/invoke.go:285 +0xc3c
golang.org/x/tools/internal/gocommand.(*Invocation).runWithFriendlyError(0xc000047568?, {0x11824e0, 0xc0152e25a0}, {0x1177c20?, 0xc0152e26c0?}, {0x1177c20, 0xc0152e26f0})
...
golang.org/x/tools/gopls/internal/cache.(*Snapshot).ModTidy.func1({0x1182518?, 0xc014c96af0?}, {0xe3c940?, 0xc0148f70e0?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/gopls/internal/cache/mod_tidy.go:80 +0x70
golang.org/x/tools/internal/memoize.(*Promise).run.func2.1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/memoize/memoize.go:187 +0xa4
runtime/trace.WithRegion({0x1182518?, 0xc014c96af0?}, {0xc00a6a4bb8, 0x13}, 0xc0108e4f70)
	/home/swarming/.swarming/w/ir/x/w/goroot/src/runtime/trace/annotation.go:141 +0xf8
golang.org/x/tools/internal/memoize.(*Promise).run.func2()
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/memoize/memoize.go:180 +0x100
created by golang.org/x/tools/internal/memoize.(*Promise).run in goroutine 13471
	/home/swarming/.swarming/w/ir/x/w/targetrepo1931866900/internal/memoize/memoize.go:179 +0x208

— watchflakes

gopherbot · 2025-01-22T18:50:23Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-22 17:02 x_tools-go1.22-openbsd-amd64 tools@e4adc385 release-branch.go1.22@c3c6a500 x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
panic: detected hanging go command (golang/go#54461); waited 1m0.057228601s
		command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-15424345/TestRegenerateCgo/default/work/... builtin
		pid:49387

goroutine 4609 [running]:
golang.org/x/tools/internal/gocommand.HandleHangingGoCommand({0xc0066548b0?, 0xc00665474c?, 0x1c02ce0?}, 0xc00665a000)
	/home/swarming/.swarming/w/ir/x/w/targetrepo3039677807/internal/gocommand/invoke.go:458 +0xb9
golang.org/x/tools/internal/gocommand.runCmdContext({0x1472f58, 0xc00bef3bf0}, 0xc00665a000)
	/home/swarming/.swarming/w/ir/x/w/targetrepo3039677807/internal/gocommand/invoke.go:391 +0x4cb
...
golang.org/x/tools/go/packages.goListDriver(0xc007a900e8, 0xc00ce91280, {0x0, 0x0}, {0xc00ce911c0, 0x2, 0xc007c65050?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo3039677807/go/packages/golist.go:210 +0x81f
golang.org/x/tools/go/packages.defaultDriver.func1(0x0?, {0xc00ce911c0?, 0x0?, 0xffffffffffffffff?})
	/home/swarming/.swarming/w/ir/x/w/targetrepo3039677807/go/packages/packages.go:343 +0x36
golang.org/x/tools/go/packages.callDriverOnChunks.func1()
	/home/swarming/.swarming/w/ir/x/w/targetrepo3039677807/go/packages/packages.go:390 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 5227
	/home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96

— watchflakes

gopherbot · 2025-01-23T14:37:01Z

Change https://go.dev/cl/643915 mentions this issue: internal/gocommand: send SIGQUIT to hanging go commands on posix

Existing debug information has not proven useful for understanding hanging go commands. On posix systems we can send SIGQUIT, and peek at stderr, to try to see what the go command is doing. Tested locally by setting the timeout to 10ms. Updates golang/go#54461 Change-Id: I1bc38d6da82b0dffb55081364b0af8f20a3afcfc Reviewed-on: https://go-review.googlesource.com/c/tools/+/643915 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Alan Donovan <adonovan@google.com> Auto-Submit: Robert Findley <rfindley@google.com>

gopherbot · 2025-01-23T22:04:24Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-23 17:02 x_tools-go1.24-openbsd-amd64 tools@7a015ab4 release-branch.go1.24@8a4c24f9 x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
2025/01/23 09:22:27 detected hanging go command (golang/go#54461); waited 1m0.120057005s
	command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-4157861698/TestRegenerateCgo/default/work/... builtin
	pid:67416

— watchflakes

gopherbot · 2025-01-24T18:11:49Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-24 16:52 x_tools-go1.22-openbsd-amd64 tools@3e68f53e release-branch.go1.22@c3c6a500 x/tools/gopls/internal/test/integration/codelens.TestRegenerateCgo/default [ABORT] (log)

=== RUN   TestRegenerateCgo/default
2025/01/24 09:14:04 detected hanging go command (golang/go#54461); waited 1m0.156171759s
	command:/home/swarming/.swarming/w/ir/x/w/goroot/bin/go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /home/swarming/.swarming/w/ir/x/t/gopls-test-961543484/TestRegenerateCgo/default/work/... builtin
	pid:6276

— watchflakes

gopherbot · 2025-01-24T18:32:02Z

Change https://go.dev/cl/644016 mentions this issue: internal/gocommand: add openbsd to the set of GOOS to debug

We encountered a hanging go command on openbsd, which unfortunately did not have logic to SIGQUIT. Add openbsd to the special set of GOOS in handleHandingGoCommand. Updates golang/go#54461 Change-Id: I36e32559f23a3ace28a1088a1f910642eb0074ec Reviewed-on: https://go-review.googlesource.com/c/tools/+/644016 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Alan Donovan <adonovan@google.com> Auto-Submit: Robert Findley <rfindley@google.com>

gopherbot · 2025-01-24T22:11:42Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos != "netbsd" && (pkg ~ `^golang.org/x/tools/gopls` || (repo == "tools" && pkg == "")) && 
        (`detected hanging go command` || log ~ `detected hanging go command` || `DETECTED A HANGING GO COMMAND` || log ~ `DETECTED A HANGING GO COMMAND`)

2025-01-24 18:50 x_tools-go1.23-windows-amd64-race tools@1c9f0026 release-branch.go1.23@ab44565b x/tools/gopls/internal/test/integration/codelens.TestUpgradeCodelens_Workspace/Upgrade_direct_dependencies/default [ABORT] (log)

=== RUN   TestUpgradeCodelens_Workspace/Upgrade_direct_dependencies/default
2025/01/24 11:06:07 detected hanging go command (golang/go#54461); waited 1m0.0031226s
	command:C:\b\s\w\ir\x\w\goroot\bin\go.exe list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- C:\b\s\w\ir\x\t\gopls-test-1897178920\TestUpgradeCodelens_Workspace\Upgrade_direct_dependencies\default\work\a\... C:\b\s\w\ir\x\t\gopls-test-1897178920\TestUpgradeCodelens_Workspace\Upgrade_direct_dependencies\default\work\b\... builtin
	pid:2836

2025-01-24 21:09 x_tools-go1.22-windows-amd64-race tools@114ac827 release-branch.go1.22@c3c6a500 x/tools/gopls/internal/test/integration/completion.TestFuzzFunc/default [ABORT] (log)

=== RUN   TestFuzzFunc/default
2025/01/24 13:24:23 detected hanging go command (golang/go#54461); waited 1m0.0032698s
	command:C:\b\s\w\ir\x\w\gopath\pkg\mod\golang.org\toolchain@v0.0.1-go1.23.4.windows-amd64\bin\go.exe list -mod=readonly -m -f {{.Path}}
{{.Dir}}
{{.GoMod}}
{{.GoVersion}}
{{range context.ReleaseTags}}{{if eq . "go1.14"}}{{.}}{{end}}{{end}}

	pid:2944

2025-01-24 21:09 x_tools-go1.22-windows-amd64-race tools@114ac827 release-branch.go1.22@c3c6a500 x/tools/gopls/internal/test/integration/misc.TestCodeActionsInGeneratedFiles/default [ABORT] (log)

=== RUN   TestCodeActionsInGeneratedFiles/default
2025/01/24 13:23:59 detected hanging go command (golang/go#54461); waited 1m0.0035154s
	command:C:\b\s\w\ir\x\w\gopath\pkg\mod\golang.org\toolchain@v0.0.1-go1.23.4.windows-amd64\bin\go.exe list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- C:\b\s\w\ir\x\t\gopls-test-3058880644\TestCodeActionsInGeneratedFiles\default\work\... builtin
	pid:3964

— watchflakes

findleyr · 2025-01-27T19:22:18Z

Here's a SIGQUIT dump from a hanging Go routine. FYI @matloob @samthanawalla this goroutine was hanging for a minute when invoked from one of our test repos (which should be very small).

https://logs.chromium.org/logs/golang/buildbucket/cr-buildbucket/8724574916972056209/+/u/step/21/log/2

gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels Aug 15, 2022

gopherbot added this to the Unreleased milestone Aug 15, 2022

findleyr modified the milestones: Unreleased, gopls/later Aug 15, 2022

findleyr self-assigned this Aug 16, 2022

bcmills mentioned this issue Sep 8, 2022

x/tools: timeout in TestManuallyCreatingXTest #54934

Closed

bcmills mentioned this issue Sep 14, 2022

build dashboard triage log #52653

Closed

bcmills mentioned this issue Sep 21, 2022

x/tools/gopls/internal/regtest/diagnostics: hanging go command failures #55323

Closed

bcmills added this to Test Flakes Sep 21, 2022

bcmills moved this to Active in Test Flakes Sep 21, 2022

findleyr mentioned this issue Jan 22, 2025

x/tools/gopls/internal/test/integration/inlayhints: TestEnablingInlayHints/enable_const/default failures #71387

Closed

findleyr mentioned this issue Jan 22, 2025

x/tools/gopls/internal/cmd: TestSemtok failures #71386

Closed

adonovan marked this as a duplicate of #71390 Jan 22, 2025

adonovan mentioned this issue Jan 22, 2025

x/tools/go/packages: TestAdHocOverlays/GO111MODULE=auto failures #71390

Closed

findleyr mentioned this issue Jan 23, 2025

x/tools/gopls/internal/cmd: TestStats failures #71393

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/gopls: regtest flakes due to hanging go commands #54461

x/tools/gopls: regtest flakes due to hanging go commands #54461

findleyr commented Aug 15, 2022 •

edited by cherrymui

Loading

findleyr commented Aug 15, 2022

adonovan commented Aug 15, 2022 •

edited

Loading

bcmills commented Aug 16, 2022 •

edited

Loading

bcmills commented Sep 14, 2022

findleyr commented Sep 14, 2022

findleyr commented Sep 14, 2022

gopherbot commented Sep 14, 2022

bcmills commented Sep 14, 2022

gopherbot commented Sep 15, 2022

findleyr commented Sep 16, 2022

adonovan commented Sep 16, 2022

findleyr commented Sep 16, 2022

adonovan commented Sep 16, 2022

findleyr commented Sep 16, 2022

bcmills commented Sep 16, 2022

findleyr commented Sep 17, 2022

bcmills commented Sep 21, 2022

gopherbot commented Dec 23, 2024

gopherbot commented Dec 23, 2024

gopherbot commented Dec 27, 2024

gopherbot commented Jan 2, 2025

gopherbot commented Jan 7, 2025

gopherbot commented Jan 7, 2025

gopherbot commented Jan 7, 2025

gopherbot commented Jan 9, 2025

gopherbot commented Jan 13, 2025

gopherbot commented Jan 22, 2025

gopherbot commented Jan 22, 2025

gopherbot commented Jan 23, 2025

gopherbot commented Jan 23, 2025

gopherbot commented Jan 24, 2025

gopherbot commented Jan 24, 2025

gopherbot commented Jan 24, 2025

findleyr commented Jan 27, 2025

x/tools/gopls: regtest flakes due to hanging go commands #54461

x/tools/gopls: regtest flakes due to hanging go commands #54461

Comments

findleyr commented Aug 15, 2022 • edited by cherrymui Loading

findleyr commented Aug 15, 2022

adonovan commented Aug 15, 2022 • edited Loading

bcmills commented Aug 16, 2022 • edited Loading

bcmills commented Sep 14, 2022

findleyr commented Sep 14, 2022

findleyr commented Sep 14, 2022

gopherbot commented Sep 14, 2022

bcmills commented Sep 14, 2022

gopherbot commented Sep 15, 2022

findleyr commented Sep 16, 2022

adonovan commented Sep 16, 2022

findleyr commented Sep 16, 2022

adonovan commented Sep 16, 2022

findleyr commented Sep 16, 2022

bcmills commented Sep 16, 2022

findleyr commented Sep 17, 2022

bcmills commented Sep 21, 2022

gopherbot commented Dec 23, 2024

gopherbot commented Dec 23, 2024

gopherbot commented Dec 27, 2024

gopherbot commented Jan 2, 2025

gopherbot commented Jan 7, 2025

gopherbot commented Jan 7, 2025

gopherbot commented Jan 7, 2025

gopherbot commented Jan 9, 2025

gopherbot commented Jan 13, 2025

gopherbot commented Jan 22, 2025

gopherbot commented Jan 22, 2025

gopherbot commented Jan 23, 2025

gopherbot commented Jan 23, 2025

gopherbot commented Jan 24, 2025

gopherbot commented Jan 24, 2025

gopherbot commented Jan 24, 2025

findleyr commented Jan 27, 2025

findleyr commented Aug 15, 2022 •

edited by cherrymui

Loading

adonovan commented Aug 15, 2022 •

edited

Loading

bcmills commented Aug 16, 2022 •

edited

Loading