[LFX-2024-Jun-Aug]Karmada Certificate Lifecycle Management

[RainbowMango]

Description:

The Karmada Certificate Lifecycle Management project is dedicated to systematically addressing the challenges faced by Karmada users related to certificate management, particularly those stemming from expiration issues that can disrupt services and compromise security. The project will achieve this through four core objectives:

• Certificate Visibility Enhancements: Develop a feature within the Karmada platform that enables users to effortlessly inspect and monitor the certificates of its components, presenting vital details such as type, issuer, and, most importantly, expiration dates. This functionality will incorporate early warning notifications to inform users well ahead of imminent certificate expirations.

• Manual Replacement Guidelines: Produce a thorough, step-by-step manual for replacing certificates in Karmada, catering to diverse deployment scenarios and encompassing best practices, potential obstacles, and troubleshooting advice. The document will be crafted for clarity and user-friendliness, integrating visual aids, screenshots, and sample command-line instructions to facilitate a seamless certificate replacement process.

• Configurable Certificate Validity during Deployment: Guarantee that Karmada's deployment tools, comprising the CLI, Helm charts, and Operator, afford users the option to define preferred certificate validity periods during installation or upgrades. This will empower users to align certificate lifecycles with their organization's security policies and maintenance schedules from the very beginning.

• Automated Certificate Rotation Capabilities: Design and integrate a mechanism for automated certificate rotation either within Karmada itself or by leveraging existing open-source projects dedicated to certificate management. This feature will dramatically reduce the operational overhead associated with maintaining a secure and up-to-date certificate landscape across the Karmada ecosystem.

Expected Outcome:

Participants in the project under the CNCF LFX program are expected to deliver the following artifacts and outcomes upon successful completion of the project:

• Certificate Visibility Tool/Feature:

  • Implement and integrate a user-friendly component within the Karmada platform or its associated tooling that allows users to inspect and monitor the certificates of Karmada components. This should include clear presentation of certificate details such as type, issuer, and expiration dates.
  • Develop and incorporate an alerting mechanism that proactively notifies users well in advance of impending certificate expirations, ensuring ample time for necessary actions.

• Manual Certificate Replacement Guide:

  • Create a comprehensive, step-by-step guide documenting the process of manually replacing certificates in Karmada for various deployment scenarios. The guide should cover best practices, potential challenges, and troubleshooting tips, accompanied by clear illustrations, screenshots, and command-line examples to facilitate user understanding and execution.

• Updated Installation Tools with Customizable Certificate Validity:

  • Modify Karmada's existing installation tools (CLI, Helm charts, and Operator) to enable users to specify desired certificate validity periods during deployment or upgrades. This customization should be clearly documented within the respective tool's user documentation or configuration instructions.

• Automated Certificate Rotation Solution Design or Integration:

  • Either design and implement a native solution within Karmada for automated certificate rotation or propose and integrate an appropriate open-source project that simplifies certificate maintenance for Karmada users. This solution should be thoroughly documented, including any necessary configuration steps and considerations for users adopting it.

In addition to these tangible deliverables, participants are expected to:

• Engage actively with the Karmada community throughout the project, soliciting feedback, addressing concerns, and promoting adoption of the developed solutions.
• Contribute to relevant Karmada documentation, updating it to reflect the new certificate management features and processes introduced by the project.
• Present their work at appropriate forums, such as Karmada community meetings, blog posts, or conference talks, to share their learnings and promote the project's outcomes.

By fulfilling these expectations, participants will contribute significantly to enhancing the overall certificate management experience for Karmada users, reducing the risk of service disruptions due to expired certificates and streamlining the process of maintaining a secure and compliant certificate infrastructure.

[AkhilJ321]

Hy @XiShanYongYe-Chang @RainbowMango
I looked into existing practices for certificate management in Kubernetes. However, Kubernetes itself does not have native support for certification rotation for all kinds of certificates. So Using an open source framework like cert-manager will be more suited for that. But Karmada is a multi-cluster environment and the framework needs to be installed on every cluster, so I am confused about how Karmada manages that thing.

The next thing is for Visibility enhancement. We have to integrate this feature into the existing karmada cli right?

[RainbowMango]

So Using an open source framework like cert-manager will be more suited for that.

+1, there might be an option to integrate cert-manager.

But Karmada is a multi-cluster environment and the framework needs to be installed on every cluster, so I am confused about how Karmada manages that thing.

This tasks isn't for managing certification for member clusters, it is for Karmada's components like karmada-apiserver, karmada-controller-managers.

The next thing is for Visibility enhancement. We have to integrate this feature into the existing karmada cli right?

That's the idea that we can build the capability into CLI tool(karmadactl).

[AkhilJ321]

Thanks for sharing the information

[AkhilJ321]

@RainbowMango @XiShanYongYe-Chang

1. I explored the documentation of karmada, I found this only :https://karmada.io/docs/next/installation/install-binary#generate-certificates. Here I understand the current certificates are installed and set up manually by OpenSSL.
   So this can be handled by Kubeadm for setting up and other operations.
   Correct me if I am wrong if we use cert-manager like this :
   
2. another query is how do we use tools like kubeadm in karamadactl cli codebase
3. Can you elaborate something about the various deployment scenarios mentioned in second task

[shivansh-bhatnagar18]

Hello @RainbowMango,
I am Shivansh Bhatnagar, a full-stack developer, and an open-source contributor. I have been contributing to Karmada Website for quite a while. I have applied to this LFX Mentorship Program recently where I have to work on Karmada Certificate Lifecycle Management. I had a small doubt, besides the questions mentioned in the LFX Platform, do I have to provide an approach to go with the project and a timeline for the same in my cover letter?
Thank you

[AkhilJ321]

hy @RainbowMango , @XiShanYongYe-Chang could you please address the above query?

[AdityaRaimec22]

@RainbowMango Can you please disclose the name of the person who get selected for this project?

[RainbowMango]

@AdityaRaimec22 will do that, probably by this week.

[JDTX0]

@RainbowMango I've opened up a PR that addresses some of these concerns around certificate lifecycle management, automatic renewals, and a better PKI hierarchy (e.g. not issuing everything off the single root)
karmada-io/karmada#5037

It took a bit of effort to get things working with the mounts and kubeconfigs, but my PR is in a place where it's ready for feedback and testing.

With the support for automatic renewal, I was able to drop the validity period of the certificates down to 30 days. Short lived certificates are better for security. I've already had Karmada deployed for over 30 days using my chart and the rotation worked flawlessly.

[AdityaRaimec22]

Hii @RainbowMango

Can I get to know the criteria for selection in LFX via your organisation?

I also applied for being a mentee in LFX via your org.

[pptfz]

Now karmada cannot be used normally due to the expiration of the certificate. The issue was raised in April. Can this problem be solved now? Is there a solution?

$ kubectl get pod
NAME                                               READY   STATUS             RESTARTS            AGE
etcd-0                                             1/1     Running            14370 (5m28s ago)   129d
karmada-apiserver-6dd844fdfd-xfltg                 0/1     CrashLoopBackOff   39401 (3m5s ago)    129d
karmada-controller-manager-7dbf7c6578-kmztg        0/1     CrashLoopBackOff   94216 (3m28s ago)   457d
karmada-kube-controller-manager-656cdc675f-cj6vw   0/1     CrashLoopBackOff   93946 (3m56s ago)   700d
karmada-scheduler-764fbdcd6d-6jhrd                 1/1     Running            1                   700d
karmada-webhook-6489787db4-wc9pp                   1/1     Running            0                   700d

karmada version

$ karmadactl version
karmadactl version: version.Info{GitVersion:"v0.5.0-2388-gccc39b2c", GitCommit:"ccc39b2cf54418face62c2e5fbdb7f697a6a5aa5", GitTreeState:"clean", BuildDate:"2022-08-18T05:56:51Z", GoVersion:"go1.18.5", Compiler:"gc", Platform:"linux/amd64"}

kubectl logs -f karmada-apiserver-6dd844fdfd-xfltg
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0806 04:28:36.351513       1 server.go:625] external host was not specified, using 10.80.4.234
I0806 04:28:36.351960       1 server.go:163] Version: v1.19.1
I0806 04:28:36.812361       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0806 04:28:36.812379       1 plugins.go:161] Loaded 9 validating admission controller(s) successfully in the following order: LimitRanger,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0806 04:28:36.814826       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0806 04:28:36.814853       1 plugins.go:161] Loaded 9 validating admission controller(s) successfully in the following order: LimitRanger,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0806 04:28:36.816937       1 client.go:360] parsed scheme: "endpoint"
I0806 04:28:36.816971       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}]
W0806 04:28:36.825743       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
I0806 04:28:37.811155       1 client.go:360] parsed scheme: "endpoint"
I0806 04:28:37.811198       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}]
W0806 04:28:37.820351       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:37.833260       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:38.828649       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:39.577379       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:40.185122       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:42.230713       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:42.733935       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:46.948481       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:47.096315       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:52.597815       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
W0806 04:28:54.494346       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match etcd-client.karmada-system.svc.cluster.local". Reconnecting...
Error: context deadline exceeded

[saowu]

Currently, we have been running Karamada for one year and our certificate is facing expiration. How can we renew the certificate. Does it support features like kubeadm certs renew all.

[RainbowMango]

Currently @yizhang-zen and @zhzhuang-zju are working on this. But I'm afraid it's too late for you to expect the new features, as new features will not applied to old releases.
But I think a guide for users on how to renew certificates would be more helpful for you, and that's exactly @yizhang-zen and @zhzhuang-zju working on right now.

By the way, are you present on the https://karmada.io/adopters/?

[pptfz]

@RainbowMango See the https://karmada.io/adopters/ I'm in vipkid

[RainbowMango]

Thanks @pptfz for the information, nice to meet you here.
Then, maybe we can get some help from @lfbear, I know he is an expert in this area.

[pptfz]

@RainbowMango karmada is the former operation and maintenance, integrated our own development of cmdb system, their level is very high, I now take over to do some operation and maintenance work, looking forward to the community to solve this problem

[zhzhuang-zju]

@pptfz May I ask how did you install Karmada and do you have any special configuration for your certificates?

[pptfz]

@zhzhuang-zju

By looking at the internal wiki, we found that the code used here for installation is the previous operation and maintenance operations
But I don't know what they have modified, or what kind of custom things, karmada is not available now, but it does not affect the current production environment, because of cost savings, we have changed the original production service to multiple cloud distribution strategy to only release to one cloud

https://github.com/lfbear/karmada/tree/vk_prod

$ helm list
NAME    NAMESPACE   REVISION    UPDATED STATUS  CHART   APP VERSION

$ kubectl get all
NAME                                                   READY   STATUS             RESTARTS            AGE
pod/etcd-0                                             1/1     Running            15271 (11m ago)     137d
pod/karmada-apiserver-6dd844fdfd-xfltg                 0/1     CrashLoopBackOff   41876 (4m44s ago)   137d
pod/karmada-controller-manager-7dbf7c6578-kmztg        0/1     Error              96499 (5m8s ago)    465d
pod/karmada-kube-controller-manager-656cdc675f-cj6vw   0/1     CrashLoopBackOff   96222 (36s ago)     709d
pod/karmada-scheduler-764fbdcd6d-6jhrd                 1/1     Running            1                   709d
pod/karmada-webhook-6489787db4-wc9pp                   1/1     Running            0                   709d

NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/etcd                ClusterIP      None            <none>        2379/TCP,2380/TCP   3y32d
service/etcd-client         ClusterIP      10.254.58.168   <none>        2379/TCP            3y32d
service/karmada-apiserver   LoadBalancer   10.254.52.163   10.80.12.18   5443:32258/TCP      3y32d
service/karmada-webhook     ClusterIP      10.254.61.17    <none>        443/TCP             3y32d

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/karmada-apiserver                 0/1     1            0           709d
deployment.apps/karmada-controller-manager        0/1     1            0           709d
deployment.apps/karmada-kube-controller-manager   0/1     1            0           709d
deployment.apps/karmada-scheduler                 1/1     1            1           709d
deployment.apps/karmada-webhook                   1/1     1            1           709d

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/karmada-apiserver-6dd844fdfd                 1         1         0       709d
replicaset.apps/karmada-controller-manager-7dbf7c6578        1         1         0       709d
replicaset.apps/karmada-kube-controller-manager-656cdc675f   1         1         0       709d
replicaset.apps/karmada-scheduler-764fbdcd6d                 1         1         1       709d
replicaset.apps/karmada-scheduler-7897cbfd9                  0         0         0       709d
replicaset.apps/karmada-webhook-6489787db4                   1         1         1       709d

NAME                    READY   AGE
statefulset.apps/etcd   1/1     709d

[zhzhuang-zju]

If the certificate has expired and you want to replace it manually, first find the CA that issued the expired certificate. Then, use the CA to issue a new certificate. Update the new certificate in the secret mounted by the component, and restart the component for the changes to take effect.

[pptfz]

Ok, I will try to operate it, but I have to make sure that the actual situation here can operate in this way to avoid other effects, thank you very much

[zhzhuang-zju]

The karmada-apiserver dynamically loads certificates, so it does not need to be restarted after the secret is updated.