Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to start controlplane in travis CI #628

Closed
magicsong opened this issue Mar 12, 2019 · 15 comments
Closed

Failed to start controlplane in travis CI #628

magicsong opened this issue Mar 12, 2019 · 15 comments

Comments

@magicsong
Copy link

Sometimes it throws errors like "timeout" in travis and sometimes it succeed. However, always good in my computer.
@Samze
Copy link
Contributor

Samze commented Mar 13, 2019 •
edited
Loading

We've been seeing this too:

      failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start

We found this was occurring when our CI machines were not powerful enough and we were running tests in parallel.

Irregardless this is not directly a kubebuilder issue but a controller-runtime envtest problem. Envtest starts the controlplane and this error is being thrown here. However, it also looks like you can change the default start/stop timeout by setting an environment variable.

@magicsong
Copy link
Author

@Samze thank you, you are right👍. After i set the timeout longer, it has not been failed

ferozjilla added a commit to rabbitmq/cluster-operator that referenced this issue Apr 5, 2019
@ferozjilla @mkuratczyk
Increase kubebuilder control plane timeout
aed4f5c 
- issue addressing CI failures kubernetes-sigs/kubebuilder#628
- we saw these faiures here https://pcf-rabbitmq.ci.cf-app.com/teams/main/pipelines/rmq-k8s/jobs/build-operator/builds/4

[#163365000]

Co-authored-by: Michal Kuratczyk <mkuratczyk@pivotal.io>
@merlintang
Copy link

@magicsong I run into this again even if I have set the timeout to be 600seconds, it fails in the travis ci. export KUBEBUILDER_CONTROLPLANE_START_TIMEOUT=600s

@merlintang
Copy link

@Samze what is the better time to set ?

@DirectXMan12
Copy link
Contributor

@merlintang you may have to check to see if your control plane is failing to start for some other reason. Perhaps you don't have the kube-apiserver or etcd binaries available?

@merlintang
Copy link

merlintang commented May 30, 2019 via email

@DirectXMan12
Copy link
Contributor

Right, so, your build environment is properly set up in Travis? What version of controller-runtime are you using? What version of the kubebuilder assets (the kube-apiserver and etcd)?

@merlintang
Copy link

merlintang commented May 30, 2019 via email

@DirectXMan12
Copy link
Contributor

Go modules aren't supported with v1.0.8. I doubt that's the issue, but I figured I'd point it out.

I think at some point there was a change to a flag on the API server. Think it was around 1.13 (we have v1.10 listed as the test version in CR v0.1.9).

@mengqiy PTAL

@mengqiy
Copy link
Member

mengqiy commented May 30, 2019

@merlintang It seems you are using the version that contains the fix related to the kube-apiserver.

I suspect it may be caused by an issue that is not directly related to kubebuilder.
kubernetes-sigs/controller-runtime#459 should help surface the issue from the testing control plane.

@merlintang
Copy link

merlintang commented May 31, 2019 via email

@mengqiy
Copy link
Member

mengqiy commented Jun 1, 2019

@merlintang You will need to pick up kubernetes-sigs/controller-runtime#459 in your controller-runtime dependency. How you do it depends on if you are using dep or go modules.
Then you can use normal approach to run test e.g. go test path/to/your/package. kube-apiserver and etcd logs will give you a idea if control plane is working properly.

@merlintang
Copy link

@mengqiy I have cherry pick to apply your pr, then, we can observe the error:
2019/06/03 15:23:19 failed to start the controlplane. retried 5 times
even if have change the KUBEBUILDER_CONTROLPLANE_START_TIMEOUT.

Can you point to me where I can look at the logs for further checking ?

@mengqiy
Copy link
Member

mengqiy commented Jun 4, 2019

@merlintang I'm not super sure what's going on on your side...

but this is not related my build environment. it fail in the travis.

If you can point me to your failing Travis log, it may help me understand what's going on.

@sri-vathsa
Copy link

sri-vathsa commented Apr 9, 2021 •
edited
Loading

I am also facing the same issue. Increasing the time out didn't solve the issue. I am running ubuntu on AWS graviton instance. I have the kube-apiserver, kubectl, etcd binaries in the same folder as kubebuilder. Can someone help me out?

@maxdrib maxdrib mentioned this issue May 17, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@DirectXMan12 @Samze @merlintang @mengqiy @magicsong @sri-vathsa