Bug: Controller occasionally crashes with index out of range

[alyssaruth]

We've been running this in our cluster for a while now via the helm chart, and I just noticed today that the controller had a bunch of restarts. Did a kubectl logs --previous to get the error output, which I've included below.

It's ~benign impact-wise for our use-case - the fact we're running it in k8s means a new container replaces the old one and things continue on merrily. But I thought I'd report it anyway so you can be aware!

I don't have any insight particularly into what might be happening - perhaps some unlucky timing with nodes being scaled down / disappearing whilst the controller is doing its thing? That's just a guess, though 🤷

I1211 22:25:10.600773       1 image_manager.go:446] Job executor-cache-kvh2s created (pull:- sonocent/jenkins-awscli:0.0.3 --> gke-ci-cluster-monitoring-2000352b-psns, runtime: containerd://1.4.8)
I1211 22:25:10.799295       1 image_manager.go:446] Job executor-cache-zkf9g created (pull:- sonocent/node-with-zip:14.17.3 --> gke-ci-cluster-monitoring-2000352b-psns, runtime: containerd://1.4.8)
E1211 22:25:10.874879       1 runtime.go:78] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range [0] with length 0)
goroutine 131 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x16ef4c0, 0xc000339020)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/runtime/runtime.go:48 +0x86
panic(0x16ef4c0, 0xc000339020)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/senthilrch/kube-fledged/pkg/images.(*ImageManager).handlePodStatusChange(0xc000588000, 0xc000a0ac00)
	/go/src/github.com/senthilrch/kube-fledged/pkg/images/image_manager.go:186 +0xbf3
github.com/senthilrch/kube-fledged/pkg/images.NewImageManager.func1(0x17a7800, 0xc000495400, 0x17a7800, 0xc000a0ac00)
	/go/src/github.com/senthilrch/kube-fledged/pkg/images/image_manager.go:157 +0x199
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/go/pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/controller.go:238
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/shared_informer.go:775 +0x1c5
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000586f60)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00052af60, 0x197f700, 0xc0000e4030, 0x15b9101, 0xc0004ce000)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000586f60, 0x3b9aca00, 0x0, 0x1, 0xc0004ce000)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0000de100)
	/go/pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0000a6300, 0xc0004be000)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:71 +0x65
panic: runtime error: index out of range [0] with length 0 [recovered]
	panic: runtime error: index out of range [0] with length 0

goroutine 131 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/runtime/runtime.go:55 +0x109
panic(0x16ef4c0, 0xc000339020)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/senthilrch/kube-fledged/pkg/images.(*ImageManager).handlePodStatusChange(0xc000588000, 0xc000a0ac00)
	/go/src/github.com/senthilrch/kube-fledged/pkg/images/image_manager.go:186 +0xbf3
github.com/senthilrch/kube-fledged/pkg/images.NewImageManager.func1(0x17a7800, 0xc000495400, 0x17a7800, 0xc000a0ac00)
	/go/src/github.com/senthilrch/kube-fledged/pkg/images/image_manager.go:157 +0x199
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/go/pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/controller.go:238
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/shared_informer.go:775 +0x1c5
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000586f60)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000bcbf60, 0x197f700, 0xc0000e4030, 0x15b9101, 0xc0004ce000)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000586f60, 0x3b9aca00, 0x0, 0x1, 0xc0004ce000)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0000de100)
	/go/pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0000a6300, 0xc0004be000)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/wait/wait.go:71 +0x65

[senthilrch]

Hi @alexburlton-sonocent : Thanks for reporting this issue. It's a little bug in the Image manager code while fetching the container status of the image puller/deleter Pod. I'll have this fixed in the upcoming v0.9.0 release.