Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

package server panic when deploying operator source from operator-marketplace #728

Closed
astefanutti opened this issue Feb 26, 2019 · 4 comments
Closed

package server panic when deploying operator source from operator-marketplace #728

astefanutti opened this issue Feb 26, 2019 · 4 comments

Comments

@astefanutti
Copy link

  • Deploying OLM version 0.8.1
  • Deploying the operator-marketplace
  • Deploying the community operator source: deploy/examples/community.operatorsource.cr.yaml
  • The package server pods to panic:
$ kubectl -n olm logs packageserver-54b889f8dd-lslhz
time="2019-02-26T17:02:48Z" level=info msg="Using in-cluster kube client config"
time="2019-02-26T17:02:48Z" level=info msg="package-server configured to watch namespaces []"
time="2019-02-26T17:02:48Z" level=info msg="Using in-cluster kube client config"
time="2019-02-26T17:02:48Z" level=info msg="connection established. cluster-version: v1.13.3"
time="2019-02-26T17:02:48Z" level=info msg="operator ready"
time="2019-02-26T17:02:48Z" level=info msg="starting informers..."
time="2019-02-26T17:02:48Z" level=info msg="waiting for caches to sync..."
I0226 17:02:48.295895       1 reflector.go:202] Starting reflector *v1alpha1.CatalogSource (5m0s) from github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:112
I0226 17:02:48.295926       1 reflector.go:240] Listing and watching *v1alpha1.CatalogSource from github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:112
time="2019-02-26T17:02:48Z" level=info msg="in for loop getting data from list stream..." action="List PackageManifests" namespace=olm
time="2019-02-26T17:02:48Z" level=info msg="in for loop getting data from list stream..." action="List PackageManifests" namespace=olm
time="2019-02-26T17:02:48Z" level=info msg="end of stream" action="List PackageManifests" namespace=olm
I0226 17:02:48.396432       1 shared_informer.go:123] caches populated
time="2019-02-26T17:02:48Z" level=info msg="starting workers..."
[restful] 2019/02/26 17:02:48 log.go:33: [restful/swagger] listing is available at https://:5443/swaggerapi
[restful] 2019/02/26 17:02:48 log.go:33: [restful/swagger] https://:5443/swaggerui/ is mapped to folder /swagger-ui/
I0226 17:02:48.468416       1 serve.go:116] Serving securely on [::]:5443
I0226 17:02:48.484925       1 wrap.go:42] GET /healthz: (13.6013ms) 200 [[kube-probe/1.13] 172.17.0.1:45880]
I0226 17:02:49.891354       1 wrap.go:42] GET /healthz: (286.6µs) 200 [[kube-probe/1.13] 172.17.0.1:45902]
I0226 17:02:51.010639       1 authorization.go:73] Forbidden: "/", Reason: ""
I0226 17:02:51.010811       1 wrap.go:42] GET /: (5.9803ms) 403 [[Go-http-client/2.0] 192.168.64.57:33350]
I0226 17:02:51.090694       1 authorization.go:73] Forbidden: "/", Reason: ""
I0226 17:02:51.091252       1 wrap.go:42] GET /: (921.9µs) 403 [[Go-http-client/2.0] 192.168.64.57:33350]
I0226 17:02:51.142628       1 wrap.go:42] GET /apis/packages.apps.redhat.com/v1alpha1?timeout=32s: (25.0889ms) 200 [[olm/v0.0.0 (linux/amd64) kubernetes/$Format] 192.168.64.57:33358]
I0226 17:02:51.285685       1 wrap.go:42] GET /apis/packages.apps.redhat.com/v1alpha1?timeout=32s: (381.7µs) 200 [[olm/v0.0.0 (linux/amd64) kubernetes/$Format] 192.168.64.57:33358]
time="2019-02-26T17:02:53Z" level=warning msg="could not publish event" action="CatalogSource Added" error="context deadline exceeded" event="&{0 olm 0xc42026b380}" name=olm-operators namespace=olm
E0226 17:02:53.334112       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1/catalogsource_types.go:46
/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/package-server/provider/registry.go:166
/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/package-server/provider/registry.go:85
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/controller.go:195
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/shared_informer.go:554
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/shared_informer.go:390
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13a93e8]

goroutine 86 [running]:
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x161a300, 0x26270a0)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1.(*RegistryServiceStatus).Address(0x0, 0xc4205df680, 0x1853ec6)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1/catalogsource_types.go:46 +0x38
github.com/operator-framework/operator-lifecycle-manager/pkg/package-server/provider.(*RegistryProvider).catalogSourceAdded(0xc42061e900, 0x181d4a0, 0xc42031cfb8)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/package-server/provider/registry.go:166 +0x2a1
github.com/operator-framework/operator-lifecycle-manager/pkg/package-server/provider.(*RegistryProvider).(github.com/operator-framework/operator-lifecycle-manager/pkg/package-server/provider.catalogSourceAdded)-fm(0x181d4a0, 0xc42031cfb8)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/pkg/package-server/provider/registry.go:85 +0x3e
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(0xc420634ea0, 0xc420634eb0, 0xc420634ec0, 0x181d4a0, 0xc42031cfb8)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/controller.go:195 +0x49
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0x0, 0x0, 0x0)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/shared_informer.go:554 +0x21a
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0xc420453df0, 0x429b19, 0xc42001e210)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203 +0x9c
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/shared_informer.go:548 +0x81
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420055768)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420453f68, 0xdf8475800, 0x0, 0x15e1801, 0xc4200a0360)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc420055768, 0xdf8475800, 0xc4200a0360)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc4204f8a00)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/shared_informer.go:546 +0x78
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache.run)-fm()
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/client-go/tools/cache/shared_informer.go:390 +0x2a
github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc420162530, 0xc420338000)
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

Env: Minikube v0.34.1 / Kubernetes 1.13.3, Mac OS X
@alecmerdler
Copy link
Member

Thanks for the issue! Please show the output of kubectl get -n marketplace catalogsources -o yaml.

@astefanutti astefanutti mentioned this issue Feb 26, 2019
Closed
@astefanutti
Copy link
Author

@alecmerdler, here is the catalog source:

$ kubectl get -n marketplace catalogsources -o yaml
apiVersion: v1
items:
- apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  metadata:
    creationTimestamp: "2019-02-26T18:10:18Z"
    generation: 1
    labels:
      olm-visibility: hidden
      openshift-marketplace: "true"
      opsrc-datastore: "true"
    name: community-operators
    namespace: marketplace
    ownerReferences:
    - apiVersion: marketplace.redhat.com/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: CatalogSourceConfig
      name: community-operators
      uid: c5bbc4d5-39f1-11e9-9db7-cee3c4953432
    resourceVersion: "4363"
    selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/marketplace/catalogsources/community-operators
    uid: c60ce515-39f1-11e9-9db7-cee3c4953432
  spec:
    address: 10.105.109.137:50051
    displayName: Community Operators
    icon:
      base64data: ""
      mediatype: ""
    publisher: Red Hat
    sourceType: grpc
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

@alecmerdler
Copy link
Member

Unfortunately, the quay.io/coreos/olm image is woefully out of date since we switched to OpenShift CI. This issue was fixed in #590. We're working on pushing a new upstream image. In the meantime, you can switch the deployment image to quay.io/openshift/origin-operator-lifecycle-manager (works on upstream Kubernetes).

@astefanutti
Copy link
Author

Ah ok. Thanks for the quick update and the workaround. Let me close this then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@astefanutti @alecmerdler