ko v0.11.1 can deadlock/hang

[dprotaso]

System Details

OS: macOS Monterey Version 12.3 (21E230)
Arch: amd64 (Intel i9)
Go Version: go1.17.8 darwin/amd64 & go version go1.18 darwin/amd64
Bash: GNU bash, version 5.1.16(1)-release (x86_64-apple-darwin20.6.0)
KO_DOCKER_REPO set to a gcr repo - but I've reproduced this using kind.local as well (see kind setup)
Terminal: iTerm2 Build 3.4.15

Repro Steps

$ go install github.com/google/ko@v0.11.2
$ git clone https://github.com/knative/serving
$ cd serving
 # default is zsh run bash 5.1
$ bash

# Run the following a few times until it frozen
$ ko resolve -R -f config/core/ > /dev/null

In another terminal run ps -l to see the fork succeed by the exec fail

 UID   PID  PPID        F CPU PRI NI       SZ    RSS WCHAN     S             ADDR TTY           TIME CMD
  502  4408   917     4006   0  31  0 35734780 117648 -      S+                  0 ttys004    0:00.86 ko resolve -R -f config/core/
  502  4651  4408        6   0  31  0 34924148    372 -      R+                  0 ttys004    0:08.97 ko resolve -R -f config/core/

Workaround

Using Apple's Terminal.app I couldn't reproduce the issue (~10 runs). Comparing the difference between the shells I noticed iTerm had some extra env vars.

Setting __CF_USER_TEXT_ENCODING of those did cause ko in Apple's Terminal.app to fail sporadically

export __CF_USER_TEXT_ENCODING=0x1F6:0x0:0x52

After unset'ing that env var in iTerm I didn't encounter a hang

Example run with backtraces
https://gist.github.com/dprotaso/c2725de85ce0e3226709071a29f97bc3

[imjasonh]

Seems related to the removal of watch mode in #585 -- there was a lot of fiddly concurrency involved in there before, and removing it may not have cleaned everything up correctly.

[dprotaso]

So this isn't specific to that code in pkg/commands/resolver.go

I'm seeing this when the docker credential helper process (docker-credential-gcloud) isn't running but ko is still waiting for it to return

[dprotaso]

  Goroutine 58 - User: /usr/local/Cellar/go/1.17.6/libexec/src/runtime/sys_darwin.go:77 syscall.rawSyscall (0x1062327)
	 0  0x0000000001063580 in runtime.systemstack_switch
	     at /usr/local/Cellar/go/1.17.6/libexec/src/runtime/asm_amd64.s:350
	 1  0x0000000001053a86 in runtime.libcCall
	     at /usr/local/Cellar/go/1.17.6/libexec/src/runtime/sys_libc.go:49
	 2  0x0000000001062327 in syscall.rawSyscall
	     at /usr/local/Cellar/go/1.17.6/libexec/src/runtime/sys_darwin.go:77
	 3  0x0000000001068046 in syscall.rawSyscall
	     at :0
	 4  0x0000000001071f2d in syscall.forkAndExecInChild
	     at /usr/local/Cellar/go/1.17.6/libexec/src/syscall/exec_libc2.go:80
	 5  0x0000000001072cc5 in syscall.forkExec
	     at /usr/local/Cellar/go/1.17.6/libexec/src/syscall/exec_unix.go:210
	 6  0x00000000010d1ff2 in syscall.StartProcess
	     at /usr/local/Cellar/go/1.17.6/libexec/src/syscall/exec_unix.go:256
	 7  0x00000000010d1ff2 in os.startProcess
	     at /usr/local/Cellar/go/1.17.6/libexec/src/os/exec_posix.go:55
	 8  0x00000000010d19fa in os.StartProcess
	     at /usr/local/Cellar/go/1.17.6/libexec/src/os/exec.go:109
	 9  0x00000000010fe44a in os/exec.(*Cmd).Start
	     at /usr/local/Cellar/go/1.17.6/libexec/src/os/exec/exec.go:422
	10  0x00000000010fddfe in os/exec.(*Cmd).Run
	     at /usr/local/Cellar/go/1.17.6/libexec/src/os/exec/exec.go:338

I think there's something broken on my system that's preventing the credential helper/go build from starting properly and it hangs when ko is calling os/exec.Cmd.Output()

[dprotaso]

Bumping from go1.17.6 to go1.17.8 and rebuilding ko seems to have worked

I wonder if this is related to golang/go#50246

[dprotaso]

Made an go issue here: golang/go#52086

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Keep fresh with the 'lifecycle/frozen' label.

[dprotaso]

/lifecycle frozen

[dprotaso]

This is fixed upstream - we just need to wait for a release and rebuild ko
golang/go#52086 (comment)

[dprotaso]

This is done