Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support apply and delete of cert-manager #320

Merged
merged 1 commit into from
Oct 27, 2022

Conversation

gthao313
Copy link
Member

@gthao313 gthao313 commented Oct 26, 2022

Issue number:
Closes: #319

Description of changes:
Support apply and delete of cert-manager

Testing done:
Integration test

service/nginx unchanged
statefulset.apps/web-test configured
deployment.apps/nginx-test unchanged
poddisruptionbudget.policy/pod-disruption-budget-test unchanged
namespace/cert-manager unchanged
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io unchanged
serviceaccount/cert-manager-cainjector unchanged
serviceaccount/cert-manager unchanged
serviceaccount/cert-manager-webhook unchanged
configmap/cert-manager-webhook configured
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews configured
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection unchanged
role.rbac.authorization.k8s.io/cert-manager:leaderelection unchanged
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving unchanged
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection unchanged
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection configured
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving configured
service/cert-manager unchanged
service/cert-manager-webhook unchanged
deployment.apps/cert-manager-cainjector unchanged
deployment.apps/cert-manager unchanged
deployment.apps/cert-manager-webhook unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
customresourcedefinition.apiextensions.k8s.io/bottlerocketshadows.brupop.bottlerocket.aws created
namespace/brupop-bottlerocket-aws created
clusterissuer.cert-manager.io/selfsigned-issuer created
issuer.cert-manager.io/my-ca-issuer created
certificate.cert-manager.io/brupop-apiserver-certificate created
serviceaccount/brupop-apiserver-service-account created
clusterrole.rbac.authorization.k8s.io/brupop-apiserver-role created
clusterrolebinding.rbac.authorization.k8s.io/brupop-apiserver-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/brupop-apiserver-auth-delegator-role-binding created
deployment.apps/brupop-apiserver created
service/brupop-apiserver created
serviceaccount/brupop-agent-service-account created
clusterrole.rbac.authorization.k8s.io/brupop-agent-role created
clusterrolebinding.rbac.authorization.k8s.io/brupop-agent-role-binding created
daemonset.apps/brupop-agent created
serviceaccount/brupop-controller-service-account created
clusterrole.rbac.authorization.k8s.io/brupop-controller-role created
clusterrolebinding.rbac.authorization.k8s.io/brupop-controller-role-binding created
priorityclass.scheduling.k8s.io/brupop-controller-high-priority created
deployment.apps/brupop-controller-deployment created
service/brupop-controller-server created
clusterrole.rbac.authorization.k8s.io "cert-manager-view" deleted
clusterrole.rbac.authorization.k8s.io "cert-manager-edit" deleted
clusterrole.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted
clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted
clusterrole.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-cainjector" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted
clusterrolebinding.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted
role.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted
role.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted
role.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted
rolebinding.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted
rolebinding.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted
rolebinding.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted
service "cert-manager" deleted
service "cert-manager-webhook" deleted
deployment.apps "cert-manager-cainjector" deleted
deployment.apps "cert-manager" deleted
deployment.apps "cert-manager-webhook" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted
validatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted
service "nginx" deleted
statefulset.apps "web-test" deleted
deployment.apps "nginx-test" deleted

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

integ/src/main.rs Outdated Show resolved Hide resolved
@gthao313 gthao313 force-pushed the integ-test-fix branch 2 times, most recently from 3c9a6dd to adb475e Compare October 26, 2022 18:23
process_cert_manager(Action::Apply, &kube_config_path)
.await
.context(error::RunBrupopSnafu)?;
sleep(WAIT_CERT_MANAGER_COMPLETE);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sleep(WAIT_CERT_MANAGER_COMPLETE);
sleep(Duration::from_secs(90));

Since this only gets called once and is specific to waiting for cert-manager to complete, could we just have the duration set here instead of declaring a constant?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should avoid using magical number, name the variable can be more explanatory。

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably doesn't matter but you never know when blocking an async thread could cause a problem. std::thread::sleep is blocking, but toklo::time::sleep is async.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but toklo::time::sleep is async.

Interesting!

process_cert_manager(Action::Apply, &kube_config_path)
.await
.context(error::RunBrupopSnafu)?;
sleep(WAIT_CERT_MANAGER_COMPLETE);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably doesn't matter but you never know when blocking an async thread could cause a problem. std::thread::sleep is blocking, but toklo::time::sleep is async.

let action_string: String = action.to_string();

// install cert-manager
let cert_manager_status = Command::new(KUBECTL_BINARY)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems weird to me to do things with kubectl instead of kube-rs, but I guess that's already the pattern.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems weird to me to do things with kubectl

This seems like a pretty reasonable punt: the alternative with kube-rs would be to

  • Deserialize the cert-manager yaml into rust objects that can be consumed by kube-rs
  • Create our own client via kube-rs
  • Apply those objects to the cluster via our kube-rs client
  • Wait / validate objects via kube-rs client

(essentially all the things kubectl does for us with just a static yaml manifest)

But I guess these integration tests sort of sit somewhere between a "script" and an actual program you'd compile/run. The script part of it assumes there's alot of stuff on your local host to actually accomplish the integration tests (access to the brupop testing accounts, kubectl, etc.)

Maybe this will be due for a refactor in the future but for now, I agree, I think this is ok.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's a reasonable ask. I remember we have that kind conversation before and I agreed to use kube-rs in future. I've created a new issue to tack this. Thanks.

@gthao313
Copy link
Member Author

Push above using tokio::time::sleep

@gthao313 gthao313 merged commit 19ff5fb into bottlerocket-os:develop Oct 27, 2022
@gthao313 gthao313 deleted the integ-test-fix branch October 27, 2022 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[integ-test] Support integration test to deploy cert-manager automaticlly
4 participants