-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Restarted pods request new CSR instead of using existing cert in k8s secret #35
Comments
Did you approve an initial CSR for the webhook? This is a required step on install. The Makefile does this for you on the initial install, but if you installed via other means that may have gotten missed. |
Yes, I did approve the initial CSR, as well as all subsequent CSRs that are created. |
I should note that this is happening on two separate clusters that we manage. |
Can you post an approved and issued csr yaml? This should include the CSR and public cert (both b64 encoded). How many CSRs are you seeing generated in an hour? Are you approving and getting the certificate signed within 15minutes of the CSR request? |
I have a suspicion about what's causing this issue. When the webhook pod restarts while the cluster is under heavy load, the k8s API service may be temporarily degraded/unavailable due to the load. This results in the pod being unable to retrieve the stored secret, which starts the routine for creating a new CSR + storing the secret, overwriting the existing (valid) cert. I'll try to get some logs for the next time this happens. A workaround may be to configure the app for the "out of cluster" |
Just caught this happening again, and sure enough the logs seem to indicate that this is the scenario playing out:
|
hi @irlevesque, we're having what seems to be the exact same issue.
Out of curiosity, did you find a workaround for this yet? |
An example of out of band management with tlsKeyFile and tlsCertFile is here. #94 (comment) |
What happened:
Pods aren't annotated due to webhook service availability issue:
This appears to happen when the pod restarts, causing mass CSR requests:
What you expected to happen:
Re-use of existing cert stored in kubernetes secret.
How to reproduce it (as minimally and precisely as possible):
Environment:
us-east-1
aws eks describe-cluster --name <name> --query cluster.platformVersion
):NA
aws eks describe-cluster --name <name> --query cluster.version
):1.15.6
v0.1.0
The text was updated successfully, but these errors were encountered: