olm-operator crashes with "fatal error: concurrent map iteration and map write"

[ctron]

The pod olm-operators crashes with the following error message:

fatal error: concurrent map iteration and map write

goroutine 242 [running]:
runtime.throw(0x150fa40, 0x26)
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:616 +0x81 fp=0xc421e1f438 sp=0xc421e1f418 pc=0x429da1
runtime.mapiternext(0xc421e1f5d0)
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/hashmap.go:747 +0x55c fp=0xc421e1f4c8 sp=0xc421e1f438 pc=0x4087cc
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.(*ObjectMeta).DeepCopyInto(0xc42168c820, 0xc422af5820)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/zz_generated.deepcopy.go:653 +0x782 fp=0xc421e1f640 sp=0xc421e1f4c8 pc=0x833342
github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1.(*ClusterServiceVersion).DeepCopyInto(0xc42168c800, 0xc422af5800)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1/zz_generated.deepcopy.go:321 +0x9c fp=0xc421e1f668 sp=0xc421e1f640 pc=0x103651c
github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1.(*ClusterServiceVersion).DeepCopy(0xc42168c800, 0x2)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1/zz_generated.deepcopy.go:333 +0x4f fp=0xc421e1f690 sp=0xc421e1f668 pc=0x103660f
github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm.(*Operator).csvSet(0xc42034eea0, 0xc4239407b0, 0x24, 0x0, 0x0, 0x0)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm/operator.go:1125 +0x1f1 fp=0xc421e1f738 sp=0xc421e1f690 pc=0x11a6b01
github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm.(*Operator).annotateCSVs(0xc42034eea0, 0xc4200ef080, 0xc420e907f0, 0x1, 0x1, 0xc423dac9a0, 0x0, 0x0)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup.go:124 +0xa0 fp=0xc421e1f910 sp=0xc421e1f738 pc=0x11abb90
github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm.(*Operator).syncOperatorGroups(0xc42034eea0, 0x14bdec0, 0xc4200ef080, 0xc4200ef080, 0x1)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup.go:81 +0x7db fp=0xc421e1fdf0 sp=0xc421e1f910 pc=0x11ab1fb
github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm.(*Operator).(github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm.syncOperatorGroups)-fm(0x14bdec0, 0xc4200ef080, 0x32, 0x14bdec0)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm/operator.go:302 +0x3e fp=0xc421e1fe28 sp=0xc421e1fdf0 pc=0x11ba14e
github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*Operator).sync(0xc42040bdc0, 0xc42056c7e0, 0xc423f129c0, 0x32, 0xc420e90780, 0x0)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:215 +0x1a4 fp=0xc421e1fed8 sp=0xc421e1fe28 pc=0x1189334
github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*Operator).processNextWorkItem(0xc42040bdc0, 0xc42056c7e0, 0x465a464f614a3100)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:183 +0xfa fp=0xc421e1ffa8 sp=0xc421e1fed8 pc=0x1188e3a
github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*Operator).worker(0xc42040bdc0, 0xc42056c7e0)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:169 +0x35 fp=0xc421e1ffd0 sp=0xc421e1ffa8 pc=0x1188d25
runtime.goexit()
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc421e1ffd8 sp=0xc421e1ffd0 pc=0x456621
created by github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*Operator).Run.func1
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:152 +0x9fc

Full stack trace: crash-stack.txt

[ecordell]

Thanks for the report - at first glance this appears to be an instance of kubernetes/kubernetes#65092

Can you verify which version of OLM and kubernetes or openshift you're on?

[ctron]

Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+a8c5f5b", GitCommit:"a8c5f5b", GitTreeState:"clean", BuildDate:"2019-08-21T12:56:45Z", GoVersion:"go1.11.6", Compiler:"gc", Platform:"linux/amd64"}

I am not sure how get the OLM version:

- name: RELEASE_VERSION
  value: 4.1.13

quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3be6fb1543a9e90659ac7d878127153f4261f416a5ea6b885116201c50454ca

[ecordell]

I believe this issue is fixed in master. Please re-open if you've seen it since.