From 2e3a7de020098e2b4fea6e07dfc5ba1e926ebda9 Mon Sep 17 00:00:00 2001 From: Camila Macedo Date: Wed, 10 Apr 2024 19:32:17 +0100 Subject: [PATCH] Proposal : Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding --- .../discontinue_usage_of_kube_rbac_proxy.md | 236 ++++++++++++++++++ 1 file changed, 236 insertions(+) create mode 100644 designs/discontinue_usage_of_kube_rbac_proxy.md diff --git a/designs/discontinue_usage_of_kube_rbac_proxy.md b/designs/discontinue_usage_of_kube_rbac_proxy.md new file mode 100644 index 00000000000..5d8aca31720 --- /dev/null +++ b/designs/discontinue_usage_of_kube_rbac_proxy.md @@ -0,0 +1,236 @@ +| Authors | Creation Date | Status | Extra | +|-----------------|---------------|---------------|-------| +| @camilamacedo86 | 07/04/2024 | Implementable | - | + +# Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding + +This proposal highlights the need to reassess the usage of [kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy) +in the default scaffold due to the evolving k8s infra, community feedback. Key considerations include the transition to a shared infrastructure requiring +all images to be published on [registry.k8s.io][registry.k8s.io], the deprecation +of Google Cloud Platform's [Container Registry](https://cloud.google.com/artifact-registry/docs/transition/prepare-gcr-shutdown), and the fact +that [kube-rbac-proxy][kube-rbac-proxy] is not yet part of the Kubernetes ecosystem umbrella. + +The dependency on a potentially discontinuable Google infrastructure, +**which is out of our control**, paired with the challenges of maintaining, +building, or promoting [kube-rbac-proxy][kube-rbac-proxy] images, +calls for a change. + +In this document is proposed replacing the [kube-rbac-proxy][kube-rbac-proxy] within +[Network Policies][k8s-doc-networkpolicies] follow-up for potentially enhancements +to protect the metrics endpoint combined with [cert-manager][cert-manager] and/or a new feature introduce +on [controller-runtime][cr-pr] which enable secure metrics serving over HTTPS. + +**For the future (when kube-rbac-proxy be part of the k8s umbrella)**, it is proposed the usage of the +[Plugins API provided by Kubebuilder](./../docs/book/src/plugins/plugins.md), +to create an [external plugin](./../docs/book/src/plugins/creating-plugins.md) +to properly integrate the solution with Kubebuilder and provide a helper to allow users to opt-in as please them. + +## Open Questions + +1. [Network Policies][k8s-doc-networkpolicies] is implemented by the cluster’s CNI. Are we confident that the proposed policy is supported by all the major CNIs in use? + +> Besides [Network Policies][k8s-doc-networkpolicies] being part of the core Kubernetes API, their enforcement relies on the CNI plugin installed in +the Kubernetes cluster. While support and implementation details can vary among CNIs, it seems that the most commonly used ones, +such as Calico, Cilium, WeaveNet, and Canal, shows to offer support for NetworkPolicies. +> +>Also, there was concern in the past because AWS did not support it. However, this changed, +>as detailed in their announcement: [Amazon VPC CNI now supports Kubernetes Network Policies](https://aws.amazon.com/blogs/containers/amazon-vpc-cni-now-supports-kubernetes-network-policies/). +> +>Moreover, under this proposal users still able to disable/enable this option as please them. + +1. NetworkPolicy is a simple firewall and does not provide authn/authz and encryption. + +> Yes, that's correct. NetworkPolicy acts as a basic firewall for pods within a Kubernetes cluster, controlling traffic flow at the IP address or port level. However, it doesn't handle authentication (authn), authorization (authz), or encryption directly likw kube-rbac-proxy solution. +> But, when combined with solutions like cert-manager for managing TLS certificates and the potential new features in controller-runtime [to serve metrics via tls and to use authentication and authorization][cr-pr] we can result in a very good level of security +> without relay in the third-part solution or have the need to still maintaining and promoting those images, which is one of the most critical issues that we need to address. + +1. Could not Kubebuilder maintainers use the shared infrastructure to continue building and promoting those images under the new `registry.k8s.io`? + +> We tried to do that, see [here](https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/k8s-staging-kubebuilder.yaml) the recipe implemented. +> However, it does not work because kube-rbac-proxy is not under the +> kubernetes umbrella. More over we experiment the GitHub Repository as alternative approach, see the [PR](https://github.com/kubernetes-sigs/kubebuilder/pull/3854) but seems +> that we are not allowed to use it. Nevertheless, both approaches does not show to sort out all motivations and requirements +> Ideally, Kubebuilder should not be responsible for maintain and promote third-part artifacts. + +1. However, is not Kubebuilder also building and promoting the binaries required to be used within [EnvTest](./../docs/book/src/reference/envtest.md) +feature implemented in controller-runtime? + +> Yes, but it also will need to change. Controller-runtime maintainers are looking for solutions to +> built those binaries inside its project since it seems part of its domain. This change is likely +> to be transparent to the community users. + +## Summary + +Starting with release `3.15.0`, Kubebuilder will no longer scaffold +new projects with [kube-rbac-proxy][kube-rbac-proxy]. +Existing users are encouraged to switch to images hosted by the project +on [quay.io](https://quay.io/repository/brancz/kube-rbac-proxy?tab=tags&tag=latest) **OR** +to adapt their projects to utilize [Network Policies][k8s-doc-networkpolicies], following the updated scaffold guidelines. + +For project updates, users can manually review scaffold changes or utilize the +provided [upgrade assistance helper](https://book.kubebuilder.io/reference/rescaffold). + +Communications and guidelines would be provided along with the release. + +## Motivation + +- **Infrastructure Reliability Concerns**: Kubebuilder’s reliance on Google's infrastructure, which may be discontinued +at their discretion, poses a risk to image availability and project reliability. [Discussion thread](https://kubernetes.slack.com/archives/CCK68P2Q2/p1711914533693319?thread_ts=1711913605.487359&cid=CCK68P2Q2) and issues: https://github.com/kubernetes/k8s.io/issues/2647 and https://github.com/kubernetes-sigs/kubebuilder/issues/3230 +- **Registry Changes and Image Availability**: The transition from `gcr.io` to [registry.k8s.io][registry.k8s.io] and +the [Container Registry][container-registry-dep] deprecation implies that **all** images provided so far by Kubebuilder +[here][kb-images-repo] will unassailable by **April 22, 2025**. [More info][container-registry-dep] and [slack ETA thread][slack-eta-thread] +- **Security and Endorsement Concerns**: [kube-rbac-proxy][kube-rbac-proxy] is a process to be part of +auth-sig for a long period, however, it not there yet. The Kubernetes Auth SIG’s review reveals that kube-rbac-proxy +must undergo significant updates to secure an official endorsement and to be supported, highlighting pressing concerns. +You can check the ongoing process and changes required by looking at the [project issue](https://github.com/brancz/kube-rbac-proxy/issues/238) +- **Evolving User Requirements and Deprecations**: The anticipated requirement for certificate management, potentially +necessitating cert-manager, underlines Kubebuilder's aim to simplify setup and reduce third-party dependencies. [More info, see issue #3524](https://github.com/kubernetes-sigs/kubebuilder/issues/3524) +- **Aim for a Transparent and collaborative infrastructure**: As an open-source project, Kubebuilder strives for +a community-transparent infrastructure that allows broader contributions. This goal aligns with our initiative +to migrate Kubebuilder CLI release builds from GCP to GitHub Actions and using Go-Releaser see [here](./../build/.goreleaser.yml), +or promoting infrastructure managed under the k8s-infra umbrella. +- **Community Feedback**: Some community members expressing a preference for its removal from the default scaffolding. [Issue 3482](https://github.com/kubernetes-sigs/kubebuilder/issues/3482) +- **Enhancing Service Monitor with Proper TLS/Certificate Usage Requested by Community:** [Issue #3657](https://github.com/kubernetes-sigs/kubebuilder/issues/3657). It is achievable with [kube-rbac-proxy][kube-rbac-proxy] OR [Network Policies][k8s-doc-networkpolicies] usage within [cert-manager][cert-manager]. + +### Goals + +- **Maximize Protection for the Metrics Endpoint without relay in third-part(s)**: Aim to provide the highest level of +protection achievable for the metrics endpoint without relying on new third-party dependencies or the need to build +and promote images from other projects. +- **Avoid Breaking Changes**: Ensure that users who generated projects with previous versions can still use the +new version with scaffold changes and are allowed to adapt their project at their convenience. +- **Sustainable Project Maintenance**: Ensure all projects scaffolded by Kubebuilder can be +maintained and supported by its maintainers. +- **Independence from Google Cloud Platform**: Move away from reliance on Google Cloud Platform, +considering the potential for unilateral shutdowns. +- **Kubernetes Umbrella Compliance**: Cease the promotion or endorsement of solutions +not yet endorsed by the Kubernetes umbrella organization mainly when it is used and shipped with the workload itself. +- **Promote Use of External Plugins**: Adhere to Kubebuilder's directive to avoid direct third-party +integrations, favoring the support of projects through the Kubebuilder API and [external plugins][external-plugins]. +This approach empowers users to add or integrate solutions with the Kubebuilder scaffold on their own, ensuring that +third-party project maintainers—who are more familiar with their solutions—can maintain and update +their integrations, as implementing it following the best practices to use their project, enhancing the user experience. +External plugins should reside within third-party repository solutions and remain up-to-date as part of those changes, +aligning with their domain of responsibility. +- **Flexible Network Policy Usage**: Allow users to opt-out of the default-enabled usage of [Network Policies][k8s-doc-networkpolicies] +if they prefer another solution, plan to deploy their solution with a vendor, or use a CNI that does not support NetworkPolicies. + +### Non-Goals + +- **Replicate kube-rbac-proxy Features or Protection Level**: It is not a goal to provide exactly the same features +or layer of protection as [kube-rbac-proxy][kube-rbac-proxy]. Since [Network Policies][k8s-doc-networkpolicies]operate differently +and do not offer the same kind of functionality as [kube-rbac-proxy][kube-rbac-proxy], achieving identical protection levels through +[Network Policies][k8s-doc-networkpolicies]alone is not feasible. + +However, incorporating NetworkPolicies, cert-manager, and/or the features introduced +in the [controller-runtime pull request #2407][cr-pr] can potentially address some of the security concerns that +kube-rbac-proxy handles. + +## Proposal + +### Phase 1: Transition to NetworkPolicies + +The immediate action outlined in this proposal is the replacement of [kube-rbac-proxy][kube-rbac-proxy] +with Kubernetes API NetworkPolicies. + +### Phase 2: Subsequent Enhancements: Cert-Manager and/or Controller-Runtime Integration + +Looking beyond the initial phase, this proposal envisions integrating cert-manager for TLS certificate management +and exploring synergies with new features in Controller Runtime, as demonstrated in [PR #2407](https://github.com/kubernetes-sigs/controller-runtime/pull/2407). + +These enhancements would introduce encrypted communication for metrics endpoints and potentially incorporate authentication mechanisms, +significantly elevating the security model employed by projects scaffolded by Kubebuilder. + +- **cert-manager**: Automates the management and issuance of TLS certificates, facilitating encrypted communication and, when configured with mTLS, adding a layer of authentication. + Currently, we leverage on cert-manager when webhooks are scaffold. So, the proposal idea would be to allow users enable the cert-manager for the metrics such as it is provided + and required for webhook feature. However, it MUST be optional. One of the goals of Kubebuilder is make it easier for new users, therefore new users should + not need to deal with cert-manager by default or have the need to install it to just quick start. + +- **Controller-Runtime Feature (PR #2407)[cr-pr]**: introduces options for metrics that allow serving metrics via TLS and using authentication and authorization. +It seems fit very well to our requirements. + +While this setup does not replicate [kube-rbac-proxy's][kube-rbac-proxy] method of leveraging Kubernetes RBAC for authentication +and authorization directly, it achieves a comparable level of security through a combination of network isolation and encrypted communication. + +### Phase 3: When kube-rbac-proxy be accepted under the umbrella + +Once kube-rbac-proxy is included in the Kubernetes umbrella, +Kubebuilder maintainers can support its integration through a [plugin](https://kubebuilder.io/plugins/plugins). +This enables a seamless way to incorporate kube-rbac-proxy into Kubebuilder scaffolds, +allowing users to run: + +```sh +kubebuilder init|edit --plugins="kube-rbac-proxy/v1" +``` + +So that, the plugin could use the [plugin/util](../pkg/plugin/util) lib provide +to comment (We can add a method like the [UncommentCode](https://github.com/kubernetes-sigs/kubebuilder/blob/72586d386cfbcaecea6321a703d1d7560c521885/pkg/plugin/util/util.go#L102)) +the patches in the `config/default/kustomization` and disable the default network policy used within +and [replace the code](https://github.com/kubernetes-sigs/kubebuilder/blob/72586d386cfbcaecea6321a703d1d7560c521885/pkg/plugin/util/util.go#L231) +in the `main.go` bellow with in order to not use the controller-runtime +feature instead. + +```go + ctrlOptions := ctrl.Options{ + MetricsFilterProvider: filters.WithAuthenticationAndAuthorization, + MetricsSecureServing: true, + } +``` + +### Proof of Concept + +- **(Phase 1)NetworkPolicies:** https://github.com/kubernetes-sigs/kubebuilder/pull/3853 +- Example of Controller-Runtime new feature to protect Metrics Endpoint: https://github.com/sbueringer/controller-runtime-tests/tree/master/metrics-auth + +### Risks and Mitigations + +#### Loss of Previously Promoted Images + +The transition to the new shared infrastructure for Kubernetes SIG projects has rendered us unable to automatically +build and promote images as before. The process only works for projects under the umbrella. +However, the k8s-infra maintainers are assisting in manually transferring these images +to the new [registry.k8s.io][registry.k8s.io]. + +To continue using kube-rbac-proxy, users will need to update their projects to reference images +from the new registry. This requires a project update and a new release, +ensuring the image references in the `config/default/manager_auth_proxy_patch.yaml` point +to a new place. + +Therefore, the best approach here for those still interested in using +kube-rbac-proxy seems to be to direct them to the images hosted +at [quay.io](https://quay.io/repository/brancz/kube-rbac-proxy?tab=tags&tag=latest), +which are maintained by the project itself and then, +we keep those images in the registry.k8s.io as a "contingent approach". + +Ensuring that these images will continue to be promoted under any infrastructure available to +Kubebuilder is not reliable or achievable for Kubebuilder maintainers. It is definitely out of our control. + +#### Impact of Google Cloud Platform Kubebuilder project + +Kubebuilder hasn't received any official notice regarding a shutdown of its project there so far, but there's a proactive move to transition away +from Google Cloud Platform services due to factors beyond our control. Open communication with our community is key as +we explore alternatives. It's important to note the [Container Registry Deprecation][container-registry-dep] results +in users no longer able to consume those images from the current location from **April 22, 2025**, +emphasizing the need to shift away from dependent images as soon as possible and communicate it extensively +through mailing lists and other channels to ensure community awareness and readiness. + +## Alternatives + +### Retain kube-rbac-proxy as an Opt-in Feature (Unsupported Feature) + +This alternative keeps kube-rbac-proxy out of default scaffolds, offering it as an optional plugin for users who choose +to integrate it. Clear communication will be crucial to inform users about the implications of using kube-rbac-proxy, +particularly the need to manage its dependencies. + +Additionally, even if opting for an alpha plugin for kube-rbac-proxy configuration, +the default scaffold should still aim for a supportable setup to ensure metrics protection. + +[kube-rbac-proxy]: https://github.com/brancz/kube-rbac-proxy +[external-plugins]: https://kubebuilder.io/plugins/external-plugins +[registry.k8s.io]: https://github.com/kubernetes/registry.k8s.io +[container-registry-dep]: https://cloud.google.com/artifact-registry/docs/transition/prepare-gcr-shutdown +[kb-images-repo]: https://console.cloud.google.com/gcr/images/kubebuilder/GLOBAL/kube-rbac-proxy +[slack-eta-thread]: https://kubernetes.slack.com/archives/CCK68P2Q2/p1712622102206909 +[cr-pr]: https://github.com/kubernetes-sigs/controller-runtime/pull/2407 +[k8s-doc-networkpolicies]: https://kubernetes.io/docs/concepts/services-networking/network-policies/ +[cert-manager]:https://cert-manager.io/ \ No newline at end of file