Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.15.0 install hang #1534

Closed
kfox1111 opened this issue May 20, 2020 · 19 comments · Fixed by #1542
Closed

0.15.0 install hang #1534

kfox1111 opened this issue May 20, 2020 · 19 comments · Fixed by #1542
Assignees

Comments

@kfox1111
Copy link

In a new minikube instance (1.10.1), I installed the crd's and that works. but applying the olm manifest is hanging:

[kfox@zathras ~]$ kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/$OLMVER/olm.yaml
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com configured

Tried to download it and apply it locally. same effect. Blew away minikube, started a fresh one, and same problem.

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

@kfox1111
Copy link
Author

Not sure why, but i was thinking the crd's shouldn't be in that file either. there is a separate crd file usually.

@kfox1111
Copy link
Author

if I cut out the crds for subscriptions and csv's out of the file, it loads ok.

@kfox1111
Copy link
Author

If I cut everything but subscripts crd out, it fails to load.

@fabstu
Copy link

fabstu commented May 21, 2020

Same problem here, did not finish after running a whole night. It is my first time trying out olm and I didn't realize the release was only 1-2 hours old at the time^^ Gonna use 0.14.1 for the time being.

Platform: Digitalocean/Kubernetes

customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com configured

@kfox1111
Copy link
Author

The manifest also refers to:

quay.io/operator-framework/olm@sha256:788915592f42882559a63c39e14ef211f64c93f31983ea67136533c7983f2896

which is inaccessible.

@kfox1111
Copy link
Author

Still not sure what's going on with the CRDs. I think it has to do with there being the string '---' being used in the crd definitions. Stripping them works:

minikube start
kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.0/crds.yaml
curl https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.0/olm.yaml -L -s > /tmp/olm.yaml
sed -i 's/--- //g' /tmp/olm.yaml
kubectl apply -f /tmp/olm.yaml

Oddly, if you apply a second time, it hangs:

kubectl apply -f /tmp/olm.yaml

@kfox1111
Copy link
Author

Ok. Narrowing it down I think...
The s/--- // thing is only semi-related.
if you simply do a:

kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.0/crds.yaml

Twice, it fails. Two of the crd's don't like being applied a second time. As its both in the crds.yaml and the olm.yaml (incorrect), it fails during olm.yaml unless there is some change.

I suspect some interaction with these crd's and the k8s serverside apply code in 1.18.

When loading the crd's into k8s, the apiserver throws this in the logs:

E0521 10:59:11.014017       1 customresource_handler.go:652] error building openapi models for subscriptions.operators.coreos.com: ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.env.items.<array>.properties.valueFrom.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.resources.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.resources.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.volumes.items.<array>.properties.downwardAPI.properties.items.items.<array>.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.volumes.items.<array>.properties.emptyDir.properties.sizeLimit has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.volumes.items.<array>.properties.projected.properties.sources.items.<array>.properties.downwardAPI.properties.items.items.<array>.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
I0521 10:59:11.015411       1 client.go:361] parsed scheme: "endpoint"
I0521 10:59:11.015443       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379  <nil> 0 <nil>}]
E0521 10:59:11.115451       1 customresource_handler.go:652] error building openapi models for clusterserviceversions.operators.coreos.com: ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.strategy.properties.rollingUpdate.properties.maxSurge has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.strategy.properties.rollingUpdate.properties.maxUnavailable has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.env.items.<array>.properties.valueFrom.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.lifecycle.properties.postStart.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.lifecycle.properties.postStart.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.lifecycle.properties.preStop.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.lifecycle.properties.preStop.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.resources.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.resources.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.startupProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.startupProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.env.items.<array>.properties.valueFrom.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.lifecycle.properties.postStart.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.lifecycle.properties.postStart.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.lifecycle.properties.preStop.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.lifecycle.properties.preStop.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.resources.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.resources.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.startupProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.startupProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.env.items.<array>.properties.valueFrom.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.lifecycle.properties.postStart.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.lifecycle.properties.postStart.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.lifecycle.properties.preStop.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.lifecycle.properties.preStop.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.resources.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.resources.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.startupProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.startupProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.overhead.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.volumes.items.<array>.properties.downwardAPI.properties.items.items.<array>.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.volumes.items.<array>.properties.emptyDir.properties.sizeLimit has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.volumes.items.<array>.properties.projected.properties.sources.items.<array>.properties.downwardAPI.properties.items.items.<array>.properties.resourceFieldRef.properties.divisor has invalid property: anyOf

@anik120
Copy link
Contributor

anik120 commented May 21, 2020

We are aware that there are issues with the 0.15.0 release, and are working on recalling the 0.15.0 release and cutting a new release 0.15.1. Will update this issue when 0.15.1 is ready.

Apologies for the confusion.
/assign

@estroz
Copy link
Member

estroz commented May 21, 2020

@kfox1111 @anik120 0.15.0 seems to install fine for me now.

@kfox1111
Copy link
Author

@anik120 +1. I'd be very curious what is going on with the CRD's. I've not seen a kubectl apply deadlock a client before.

@estroz With the workarounds above, I was able to start it now too.

@anik120
Copy link
Contributor

anik120 commented May 21, 2020

@kfox1111 the issue was that the 0.15.0 tag in quay was deleted. It's been restored now, however the 0.15.0 assets contains a commit beyond the 0.15.0 tag in quay. We're publishing a patch before we recall 0.15.0 so that the assets and the quay tag are in sync.

@kfox1111
Copy link
Author

That explains why it wouldn't start after the crds were removed from the olm.yaml file. Still unsure why the crd loading themselves ffail on reapply though. I'm wondering if 0.15.1 will have the same issue.

@anik120
Copy link
Contributor

anik120 commented May 21, 2020

@kfox1111 the failure you're seeing on reapplying the CRDs might be due to a kubectl bug in this library: evanphx/json-patch@v4.4.0...v4.6.0

Applying quickstart/olm.yaml was failing because the CRDs were being included in that file too.
Thank you for bringing it to our notices. We're making sure we don't run into this for 0.15.1.

@kfox1111
Copy link
Author

Ah. Thanks for the link and the details and for looking into it. :)

And thanks for all the hard work. OLM's really a great piece of software.

@benluddy
Copy link
Contributor

@kfox1111 I dug into the hanging kubectl apply, and as far as I can tell it's legitimately busy constructing a JSON patch the entire time due to the size of some of the CRD schemas. The log message you found is an open issue on the Kube API server (kubernetes/kubernetes#90902), but it's a red herring.

@benluddy
Copy link
Contributor

benluddy commented May 21, 2020

Explanation and fix went in here evanphx/json-patch#87, but kube seems to be using 4.2.0, which pre-dates that change.

@kfox1111
Copy link
Author

Ah. Very interesting. Thanks. :)

@CermakM
Copy link

CermakM commented May 22, 2020

Still hitting this with 0.15.1

@jianzhangbjz
Copy link
Contributor

/cc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants