From b3a7965e3eb5a86a4b84dc9e2ee486f13e3df128 Mon Sep 17 00:00:00 2001 From: mtardy Date: Fri, 27 May 2022 18:37:43 +0200 Subject: [PATCH 01/16] Add the security checklist guide From the collaborative document with Savitha, Skybound and p4ck3t0, after many edits thanks to the collaborators on the PR. Co-authored-by: rschosser <88308339+rschosser@users.noreply.github.com> Co-authored-by: Cailyn Co-authored-by: Tim Bannister Co-authored-by: Rey Lejano Co-authored-by: Benjamin Koltermann <48812495+p4ck3t0@users.noreply.github.com> Co-authored-by: Skybound1 Co-authored-by: divya-mohan0209 --- .../concepts/security/security-checklist.md | 380 ++++++++++++++++++ 1 file changed, 380 insertions(+) create mode 100644 content/en/docs/concepts/security/security-checklist.md diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md new file mode 100644 index 0000000000000..8a90b65eadf60 --- /dev/null +++ b/content/en/docs/concepts/security/security-checklist.md @@ -0,0 +1,380 @@ +--- +title: Security Checklist +description: > + Baseline checklist for ensuring security in Kubernetes clusters. +content_type: concept +--- + + + +This checklist aims at providing a basic list of guidance with links to more +comprehensive documentation on each topic. It does not claim to be exhaustive +and is meant to evolve. + +The order of topics does not reflect an order of priority. + +{{< caution >}} +Checklists are **not** sufficient for attaining a good security posture on their +own. A good security posture requires constant attention and improvement, but a +checklist can be the first step on the never-ending journey towards security +preparedness. Some of the recommendations in this checklist may be too +restrictive or too lax for your specific security needs. Since Kubernetes +security is not "one size fits all", each category of checklist items should be +evaluated on its merits. +{{< /caution >}} + + + +## Authentication & Authorization + +- [ ] `system:masters` group is not used for user or component authentication after bootstrapping. +- [ ] The kube-controller-manager is running with `--use-service-account-credentials` + enabled. +- [ ] The root certificate is protected (either an offline CA, or a managed + online CA with effective access controls). +- [ ] Intermediate and leaf certificates have an expiry date no more than 3 + years in the future. +- [ ] A process exists for periodic access review, and reviews occur no more + than 24 months apart. + +See the [Role Based Access Control Good Practices](/docs/concepts/security/rbac-good-practices/) +for guidance related to authentication and authorization. + +After bootstrapping, neither users nor components should authenticate to the +Kubernetes API as `system:masters`. Similarly, running all of +kube-controller-manager as `system:masters` should be avoided. In fact, +`system:masters` should only be used as a break-glass mechanism, as opposed to +an admin user. + +## Network security + +- [ ] CNI plugins in-use supports network policies. +- [ ] Ingress and egress network policies are applied to all workloads in the + cluster. +- [ ] Default network policies within each namespace, selecting all pods, denying + everything, are in place. +- [ ] Use of a service mesh to encrypt all communications inside of the cluster. +- [ ] The Kubernetes API, kubelet API and etcd are not exposed publicly on Internet. +- [ ] Access from the workloads to the cloud metadata API is filtered. +- [ ] Use of LoadBalancer and ExternalIPs is restricted, see +[CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs](https://github.com/kubernetes/kubernetes/issues/97076) +and the [DenyServiceExternalIPs admission controller](/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips) +for further information. + +A number of [Container Network Interface (CNI) plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) +plugins provide the functionality to +restrict network resources that pods may communicate with. This is most commonly done +through [Network Policies](/docs/concepts/services-networking/network-policies/) +which provide a namespaced resource to define rules. Default network policies +blocking everything egress and ingress, in each namespace, selecting all the +pods, can be useful to adopt an allow list approach, ensuring that no workloads +is missed. + +Not all CNI plugins provide encryption in transit. If the chosen plugin lacks this +feature, an alternative solution could be to use a service mesh to provide that +functionality. + +The etcd datastore of the control plane should have controls to limit access and +not be publicly exposed on the Internet. Furthermore, mutual TLS (mTLS) should +be used to communicate securely with it. The certificate authority for this +should be unique to etcd. + +External Internet access to the Kubernetes API server should be restricted to +not expose the API publicly. Be careful as many managed Kubernetes distribution +are publicly exposing the API server by default. You can then use a bastion host +to access the server. + +The [kubelet](/docs/reference/command-line-tools-reference/kubelet/) API access +should be restricted and not publicly exposed, the defaults authentication and +authorization settings, when no configuration file specified with the `--config` +flag, are overly permissive. + +If a cloud provider is used for hosting Kubernetes, the access from pods to the cloud +metadata API `169.254.169.254` should also be restricted or blocked if not needed +because it may leak information. + +## Pod security + +- [ ] RBAC rights to `create`, `update`, `patch`, `delete` workloads is only granted if necessary. +- [ ] Appropriate pod security is enforced for all workloads. +- [ ] Memory and CPU limits are set for the workloads. +- [ ] For nodes that support it, Seccomp is enabled with appropriate syscalls + profile for programs. +- [ ] For nodes that support it, AppArmor or SELinux is enabled with appropriate + profile for programs. + +RBAC authorization is crucial but +[cannot be granular enough to have authorization on the Pods' resources](/docs/concepts/security/rbac-good-practices/#workload-creation) +(or on any resource that manages Pods). The only granularity is the API verbs +on the resource itself, for example, `create` on Pods. Without +additional admission, the authorization to create these resources allows direct +unrestricted access to the schedulable nodes of a cluster. + +[Pod Security Admission](/docs/concepts/security/pod-security-admission/) is an +admission controller to enable fine-grained authorization of Pod creation and +updates. It replaces the Pod Security Policy admission controller. +The [Pod Security Standards](/docs/concepts/security/pod-security-standards/) +define three different policies that restrict how fields can be set in the `PodSpec`. +For a hands-on tutorial on Pod Security, see the blog post +"[Kubernetes 1.23: Pod Security Graduates to Beta](/blog/2021/12/09/pod-security-admission-beta/)". + +{{< note >}} +[PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) +will be removed in version 1.25. +{{< /note >}} + +Pod Security admission `restricted` policy [can operate in several modes](/docs/concepts/security/pod-security-admission/#pod-security-admission-labels-for-namespaces), +`enforce`, `audit` or `warn` to help properly set the [security context](/docs/tasks/configure-pod-container/security-context/) +according to security best practices. Nevertheless, pods' [security context](/docs/tasks/configure-pod-container/security-context/) +should be separately investigated to limit the privileges and access pods may have. + +[Memory and CPU limits](/docs/concepts/configuration/manage-resources-containers/) +should be set in order to restrict the memory and CPU resources a pod can +consume on a node, and therefore prevent potential DoS attacks. It can be +enforced by an admission controller. + +### Enabling Seccomp + +Seccomp can improve the security of your workloads by reducing the Linux kernel +syscall attack surface available inside containers. The seccomp filter mode +leverages BPF to create an allow or deny list of specific syscalls, named +profiles. Those seccomp profiles can be enabled on individual workloads, +[a security tutorial is available](/docs/tutorials/security/seccomp/). In +addition, the [Kubernetes Security Profiles Operator](https://github.com/kubernetes-sigs/security-profiles-operator) +is a project to facilitate the management and use of seccomp in clusters. + +For historical context, please note that Docker has been using +[a default seccomp profile](https://docs.docker.com/engine/security/seccomp/) +to only allow a restricted set of syscalls since 2016 from +[Docker Engine 1.10](https://www.docker.com/blog/docker-engine-1-10-security/), +but Kubernetes is still not confining workloads by default. The default seccomp +profile can be found [in containerd](https://github.com/containerd/containerd/blob/main/contrib/seccomp/seccomp_default.go) +as well. Fortunately, [Seccomp Default](/blog/2021/08/25/seccomp-default/), a +new alpha feature to use a default seccomp profile for all workloads can now be +enabled and tested. + +{{< note >}} +Seccomp is only available on Linux nodes. +{{< /note >}} + +### Enabling AppArmor or SELinux + +#### AppArmor + +[AppArmor](https://apparmor.net/) is a Linux kernel security module that can +provide an easy way to implement Mandatory Access Control (MAC) and better +auditing through system logs. To [enable AppArmor in Kubernetes](/docs/tutorials/security/apparmor/), +at least version 1.4 is required. Like seccomp, AppArmor is also configured +through profiles, where each profile is either running in enforcing mode, which +blocks access to disallowed resources or complain mode, which only reports +violations. AppArmor profiles are enforced on a per-container basis, with an +annotation, allowing for processes to gain just the right privileges. + +{{< note >}} +AppArmor is only available on Linux nodes, and enabled in +[some Linux distributions](https://gitlab.com/apparmor/apparmor/-/wikis/home#distributions-and-ports). +{{< /note >}} + +#### SELinux + +[SELinux](https://github.com/SELinuxProject/selinux-notebook/blob/main/src/selinux_overview.md) is also a +Linux kernel security module that can provide a mechanism for supporting access +control security policies, including Mandatory Access Controls (MAC). SELinux +labels can be assigned to containers or pods +[via their `securityContext` section](/docs/tasks/configure-pod-container/security-context/#assign-selinux-labels-to-a-container). + +{{< note >}} +SELinux is only available on Linux nodes, and enabled in +[some Linux distributions](https://en.wikipedia.org/wiki/Security-Enhanced_Linux#Implementations). +{{< /note >}} + +## Pod placement + +- [ ] Pod placement is done in accordance with the tiers of sensitivity of the + application. +- [ ] Sensitive applications are running isolated on nodes or with specific + sandboxed runtimes. + +Pods that are on different tiers of sensitivity, for example, an application pod +and the Kubernetes API server, should be deployed onto separate nodes. The +purpose of node isolation is to prevent an application container breakout to +directly providing access to applications with higher level of sensitivity to easily +pivot within the cluster. This separation should be enforced to prevent pods +accidentally being deployed onto the same node. This could be enforced with the +following features: + +[Node Selectors](/docs/concepts/scheduling-eviction/assign-pod-node/) +: Key-value pairs, as part of the pod specification, that specify which nodes to +deploy onto. These can be enforced at the namespace and cluster level with the +[PodNodeSelector](/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) +admission controller. + +[PodTolerationRestriction](/docs/reference/access-authn-authz/admission-controllers/#podtolerationrestriction) +: An admission controller that allows administrators to restrict permitted +[tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/) within a +namespace. Pods within a namespace may only utilize the tolerations specified on +the namespace object annotation keys that provide a set of default and allowed +tolerations. + +[RuntimeClass](/docs/concepts/containers/runtime-class/) +: RuntimeClass is a feature for selecting the container runtime configuration. +The container runtime configuration is used to run a Pod's containers and can +provide more or less isolation from the host at the cost of performance +overhead. + +## Secrets + +- [ ] ConfigMaps are not used to hold confidential data. +- [ ] Encryption at rest is configured for the Secret API. +- [ ] A mechanism to inject secrets stored in third party storage is deployed + and available. +- [ ] Service account tokens are not mounted in pods that don't require them. +- [ ] [Bound service account token volume](/docs/reference/access-authn-authz/service-accounts-admin/#bound-service-account-token-volume) + is in-use instead of non-expiring tokens. + +Secrets required for pods should be stored within Kubernetes Secrets as opposed +to alternatives such as ConfigMap. Secret resources stored within etcd should +be [encrypted at rest](/docs/tasks/administer-cluster/encrypt-data/). + +Pods needing secrets should have these automatically mounted through volumes, +preferably stored in memory like with the [`emptyDir.medium` option](/docs/concepts/storage/volumes/#emptydir). +Mechanism can be used to also inject secrets from third-party storages as +volume, like the [Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/). +This should be done preferentially as compared to providing the pods service +account RBAC access to secrets. This would allow adding secrets into the pod as +environment variables or files. Please note that the environment variable method +might be more prone to leakage due to crash dumps in logs and the +non-confidential nature of environment variable in Linux, as opposed to the +permission mechanism on files. + +Service account tokens should not be mounted into pods that do not require them. This can be configured by setting +[`automountServiceAccountToken`](/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server) +to `false` either within the service account to apply throughout the namespace +or specifically for a pod. For Kubernetes v1.22 and above, use +[Bound Service Accounts](/docs/reference/access-authn-authz/service-accounts-admin/#bound-service-account-token-volume) +for time-bound service account credentials. + +## Images + +- [ ] Container images contain only the bare minimum. +- [ ] Container images are configured to be run as unprivileged user. +- [ ] References to container images are made by sha256 digests (rather than +tags) or the provenance of the image is validated by verifying the image's +digital signature at deploy time [via admission control](/docs/tasks/administer-cluster/verify-signed-images/#verifying-image-signatures-with-admission-controller). +- [ ] Container images are regularly scanned during creation and in deployment, and + known vulnerable software is patched. + +Container image should contain the bare minimum to run the program they +package. Preferably, only the program and its dependencies, building the image +from the minimal possible base. In particular, image used in production should not +contain shells or debugging utilities, as an +[ephemeral debug container](/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container) +can be used for troubleshooting. + +Build images to directly start with an unprivileged user by using the +[`USER` instruction in Dockerfile](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user). +The [Security Context](/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) +allows a container image to be started with a specific user and group with +`runAsUser` and `runAsGroup`, even if not specified in the image manifest. +However, the file permissions in the image layers might make it impossible to just +start the process with a new unprivileged user without image modification. + +Avoid using image tags to reference an image, especially the `latest` tag, the +image behind a tag can be easily modified in a registry. Prefer using the +complete `sha256` digest which is unique to the image manifest. This policy can be +enforced via an [ImagePolicyWebhook](/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook). +Image signatures can also be automatically [verified with an admission controller](/docs/tasks/administer-cluster/verify-signed-images/#verifying-image-signatures-with-admission-controller) +at deploy time to validate their authenticity and integrity. + +Scanning a container image can prevent critical vulnerabilities from being +deployed to the cluster alongside the container image. Image scanning should be +completed before deploying a container image to a cluster and is usually done +as part of the deployment process in a CI/CD pipeline. The purpose of an image +scan is to obtain information about possible vulnerabilities and their +prevention in the container image, such as a +[Common Vulnerability Scoring System (CVSS)](https://www.first.org/cvss/) +score. If the result of the image scans is combined with the pipeline +compliance rules, only properly patched container images will end up in +Production. + +## Admission controllers + +- [ ] An appropriate selection of admission controllers is enabled. +- [ ] A pod security policy is enforced by the Pod Security Admission or/and a + webhook admission controller. +- [ ] The admission chain plugins and webhooks are securly configured. + +Admission controllers can help to improve the security of the cluster. However, +they can present risks themselves as they extend the API server and +[should be properly secured](/blog/2022/01/19/secure-your-admission-controllers-and-webhooks/). + +The following list present a number of admission controllers that could be +considered to enhance the security posture of your cluster and application. It +includes controllers that may be referenced in other parts of this document: + +[`AlwaysPullImages`](/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages) +: Enforces the usage of the latest version of a tagged image and ensures that the deployer +has permissions to use the image. + +[`CertificateApproval`](/docs/reference/access-authn-authz/admission-controllers/#certificateapproval) +: Performs additional authorization checks to ensure the approving user has +permission to approve certificate request. + +[`CertificateSigning`](/docs/reference/access-authn-authz/admission-controllers/#certificatesigning) +: Performs additional authorization checks to ensure the signing user has +permission to sign certificate requests. + +[`CertificateSubjectRestriction`](/docs/reference/access-authn-authz/admission-controllers/#certificatesubjectrestriction) +: Rejects any certificate request that specifies a 'group' (or 'organization +attribute') of `system:masters`. + +[`DenyServiceExternalIPs`](/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips) +: Rejects all net-new usage of the `Service.spec.externalIPs` field. This is a mitigation for +[CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs](https://github.com/kubernetes/kubernetes/issues/97076). + +[`EventRateLimit`](/docs/reference/access-authn-authz/admission-controllers/#eventratelimit) +: Rate limits adding new Events to the API server. + +[`ImagePolicyWebhook`](/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook) +: Allows enforcing additional controls for images through webhooks. + +[`MutatingAdmissionWebhook`](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) +: Allows the use of custom controllers through webhooks, these controllers may +mutate requests that it reviews. + +[`NodeRestriction`](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) +: Restricts kubelet's permissions to only modify the pods API resources they own +or the node API ressource that represent themselves. It also prevents kubelet +from using the `node-restriction.kubernetes.io/` annotation, which can be used +by an attacker with access to the kubelet's credentials to influence pod +placement to the controlled node. + +[`PodNodeSelector`](/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) +: Allows controls of node selectors within namespaces and cluster-wide. + +[`PodSecurity`](/docs/reference/access-authn-authz/admission-controllers/#podsecurity) +: Replacement for Pod Security Policy, restricts security contexts of deployed +Pods. + +[`PodTolerationRestriction`](/docs/reference/access-authn-authz/admission-controllers/#podtolerationrestriction) +: Allows control of pod tolerations permitted for pods within a namespace. + +[`ResourceQuota`](/docs/reference/access-authn-authz/admission-controllers/#resourcequota) +: Enforces resource quotas to prevent over-usage of resources. + +[`ValidatingAdmissionWebhook`](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook) +: Allows the use of custom controllers through webhooks, these controllers do +not mutate requests that it reviews. + +{{< note >}} +Many of these admission controllers are [enabled by default](/docs/reference/access-authn-authz/admission-controllers/#which-plugins-are-enabled-by-default) by the API server. +{{< /note >}} + +## What's next + +- [RBAC Good Practices](/docs/concepts/security/rbac-good-practices/) for + further information on authorization. +- [Cluster Multi-tenancy guide](/docs/concepts/security/multi-tenancy/) for + configuration options recommendations and best practices on multi-tenancy. +- [Blog post "A Closer Look at NSA/CISA Kubernetes Hardening Guidance"](/blog/2021/10/05/nsa-cisa-kubernetes-hardening-guidance/#building-secure-container-images) + for complementary resource on hardening Kubernetes clusters. From 777d3969059eb635f19ed845d6dccbfef358a389 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mah=C3=A9?= Date: Wed, 31 Aug 2022 16:54:30 +0200 Subject: [PATCH 02/16] Remove warning on PodSecurityPolicy removal in 1.25 --- content/en/docs/concepts/security/security-checklist.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 8a90b65eadf60..69325d0f72ccf 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -118,10 +118,6 @@ define three different policies that restrict how fields can be set in the `PodS For a hands-on tutorial on Pod Security, see the blog post "[Kubernetes 1.23: Pod Security Graduates to Beta](/blog/2021/12/09/pod-security-admission-beta/)". -{{< note >}} -[PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) -will be removed in version 1.25. -{{< /note >}} Pod Security admission `restricted` policy [can operate in several modes](/docs/concepts/security/pod-security-admission/#pod-security-admission-labels-for-namespaces), `enforce`, `audit` or `warn` to help properly set the [security context](/docs/tasks/configure-pod-container/security-context/) From 5167ab5c884369430a1d562f41f97a48a8b9d8cb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mah=C3=A9?= Date: Wed, 31 Aug 2022 16:55:05 +0200 Subject: [PATCH 03/16] Use correct name for PodSecurityPolicy admission controller --- content/en/docs/concepts/security/security-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 69325d0f72ccf..2c847b23deb63 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -112,7 +112,7 @@ unrestricted access to the schedulable nodes of a cluster. [Pod Security Admission](/docs/concepts/security/pod-security-admission/) is an admission controller to enable fine-grained authorization of Pod creation and -updates. It replaces the Pod Security Policy admission controller. +updates. It replaces the PodSecurityPolicy admission controller. The [Pod Security Standards](/docs/concepts/security/pod-security-standards/) define three different policies that restrict how fields can be set in the `PodSpec`. For a hands-on tutorial on Pod Security, see the blog post From 949e499db3a7563496ab9a7be283a9bf979c9890 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mah=C3=A9?= Date: Wed, 31 Aug 2022 16:55:31 +0200 Subject: [PATCH 04/16] Rewrite the checklist item on minimal container images --- content/en/docs/concepts/security/security-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 2c847b23deb63..8d9aa0bd501e2 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -252,7 +252,7 @@ for time-bound service account credentials. ## Images -- [ ] Container images contain only the bare minimum. +- [ ] Minimize unnecessary content in container images. - [ ] Container images are configured to be run as unprivileged user. - [ ] References to container images are made by sha256 digests (rather than tags) or the provenance of the image is validated by verifying the image's From 7139aba9549f053a20eb212a67721c4b3204147e Mon Sep 17 00:00:00 2001 From: mtardy Date: Wed, 31 Aug 2022 17:17:56 +0200 Subject: [PATCH 05/16] Add some guidelines on how to read the doc --- content/en/docs/concepts/security/security-checklist.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 8d9aa0bd501e2..753ad63cfc404 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -11,7 +11,9 @@ This checklist aims at providing a basic list of guidance with links to more comprehensive documentation on each topic. It does not claim to be exhaustive and is meant to evolve. -The order of topics does not reflect an order of priority. +On how to read and use this document: +- The order of topics does not reflect an order of priority. +- Some checklist items are detailed in the paragraph below the list of each section. {{< caution >}} Checklists are **not** sufficient for attaining a good security posture on their From 0e81bfd8eff2d1149dfb9ebf3679632f6b3dcba2 Mon Sep 17 00:00:00 2001 From: mtardy Date: Wed, 31 Aug 2022 17:32:00 +0200 Subject: [PATCH 06/16] Detail and add info on the CPU and memory limit item --- .../docs/concepts/security/security-checklist.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 753ad63cfc404..985bd056fa2dd 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -99,7 +99,8 @@ because it may leak information. - [ ] RBAC rights to `create`, `update`, `patch`, `delete` workloads is only granted if necessary. - [ ] Appropriate pod security is enforced for all workloads. -- [ ] Memory and CPU limits are set for the workloads. +- [ ] Memory limit is set for the workloads with a limit equal or inferior to the request. +- [ ] CPU limit might be set on sensitive workloads. - [ ] For nodes that support it, Seccomp is enabled with appropriate syscalls profile for programs. - [ ] For nodes that support it, AppArmor or SELinux is enabled with appropriate @@ -128,8 +129,15 @@ should be separately investigated to limit the privileges and access pods may ha [Memory and CPU limits](/docs/concepts/configuration/manage-resources-containers/) should be set in order to restrict the memory and CPU resources a pod can -consume on a node, and therefore prevent potential DoS attacks. It can be -enforced by an admission controller. +consume on a node, and therefore prevent potential DoS attacks from malicious or +breached workloads. Such policy can be enforced by an admission controller. +Please note that CPU limits will throttle usage and thus can have unintended +effects on auto-scaling features or efficiency i.e. running the process in best +effort with the CPU resource available. + +{{< caution >}} +Memory limit superior to request can expose the whole node to OOM issues. +{{< /caution >}} ### Enabling Seccomp From 2f8388e83097b98862fc9747fc9728e28e10b639 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mah=C3=A9?= Date: Wed, 31 Aug 2022 17:35:03 +0200 Subject: [PATCH 07/16] Add precision about pod security with pod security standards Co-authored-by: Rey Lejano --- content/en/docs/concepts/security/security-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 985bd056fa2dd..a4cae6103b617 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -98,7 +98,7 @@ because it may leak information. ## Pod security - [ ] RBAC rights to `create`, `update`, `patch`, `delete` workloads is only granted if necessary. -- [ ] Appropriate pod security is enforced for all workloads. +- [ ] Appropriate Pod Security Standards policy is applied for all namespaces in `enforce` mode. - [ ] Memory limit is set for the workloads with a limit equal or inferior to the request. - [ ] CPU limit might be set on sensitive workloads. - [ ] For nodes that support it, Seccomp is enabled with appropriate syscalls From d40e9cfa8911b565098a008e97874b2d2de3991c Mon Sep 17 00:00:00 2001 From: mtardy Date: Wed, 31 Aug 2022 17:37:01 +0200 Subject: [PATCH 08/16] Remove an empty line --- content/en/docs/concepts/security/security-checklist.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index a4cae6103b617..63c561a1089c2 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -121,7 +121,6 @@ define three different policies that restrict how fields can be set in the `PodS For a hands-on tutorial on Pod Security, see the blog post "[Kubernetes 1.23: Pod Security Graduates to Beta](/blog/2021/12/09/pod-security-admission-beta/)". - Pod Security admission `restricted` policy [can operate in several modes](/docs/concepts/security/pod-security-admission/#pod-security-admission-labels-for-namespaces), `enforce`, `audit` or `warn` to help properly set the [security context](/docs/tasks/configure-pod-container/security-context/) according to security best practices. Nevertheless, pods' [security context](/docs/tasks/configure-pod-container/security-context/) From 63ae0a952156d914686f0b5124a3102a437a6cbc Mon Sep 17 00:00:00 2001 From: mtardy Date: Wed, 31 Aug 2022 17:38:42 +0200 Subject: [PATCH 09/16] Split checklist item and explanation --- content/en/docs/concepts/security/security-checklist.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 63c561a1089c2..a6e2e35eac774 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -58,10 +58,7 @@ an admin user. - [ ] Use of a service mesh to encrypt all communications inside of the cluster. - [ ] The Kubernetes API, kubelet API and etcd are not exposed publicly on Internet. - [ ] Access from the workloads to the cloud metadata API is filtered. -- [ ] Use of LoadBalancer and ExternalIPs is restricted, see -[CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs](https://github.com/kubernetes/kubernetes/issues/97076) -and the [DenyServiceExternalIPs admission controller](/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips) -for further information. +- [ ] Use of LoadBalancer and ExternalIPs is restricted. A number of [Container Network Interface (CNI) plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) plugins provide the functionality to @@ -95,6 +92,10 @@ If a cloud provider is used for hosting Kubernetes, the access from pods to the metadata API `169.254.169.254` should also be restricted or blocked if not needed because it may leak information. +For restricted LoadBalancer and ExternalIPs use, see +[CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs](https://github.com/kubernetes/kubernetes/issues/97076) +and the [DenyServiceExternalIPs admission controller](/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips) +for further information. ## Pod security - [ ] RBAC rights to `create`, `update`, `patch`, `delete` workloads is only granted if necessary. From c006a43f978974ecc93d4e36fe5601009d28a0af Mon Sep 17 00:00:00 2001 From: mtardy Date: Wed, 31 Aug 2022 17:51:51 +0200 Subject: [PATCH 10/16] Replace a wrong unicode space character --- content/en/docs/concepts/security/security-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index a6e2e35eac774..f7145ad9d1141 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -58,7 +58,7 @@ an admin user. - [ ] Use of a service mesh to encrypt all communications inside of the cluster. - [ ] The Kubernetes API, kubelet API and etcd are not exposed publicly on Internet. - [ ] Access from the workloads to the cloud metadata API is filtered. -- [ ] Use of LoadBalancer and ExternalIPs is restricted. +- [ ] Use of LoadBalancer and ExternalIPs is restricted. A number of [Container Network Interface (CNI) plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) plugins provide the functionality to From 239dc4c2fe26b603df7a2ea33dc71b9703d27140 Mon Sep 17 00:00:00 2001 From: mtardy Date: Wed, 31 Aug 2022 17:54:20 +0200 Subject: [PATCH 11/16] Fix a typo on the word securely --- content/en/docs/concepts/security/security-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index f7145ad9d1141..aa6252c2a1f10 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -308,7 +308,7 @@ Production. - [ ] An appropriate selection of admission controllers is enabled. - [ ] A pod security policy is enforced by the Pod Security Admission or/and a webhook admission controller. -- [ ] The admission chain plugins and webhooks are securly configured. +- [ ] The admission chain plugins and webhooks are securely configured. Admission controllers can help to improve the security of the cluster. However, they can present risks themselves as they extend the API server and From f14a7544e5130128cf0d8f57a2db98359221c023 Mon Sep 17 00:00:00 2001 From: mtardy Date: Wed, 31 Aug 2022 18:26:49 +0200 Subject: [PATCH 12/16] Rewrite the admission plugins list --- .../concepts/security/security-checklist.md | 70 +++++++++++-------- 1 file changed, 41 insertions(+), 29 deletions(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index aa6252c2a1f10..bff786c2499bf 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -314,13 +314,13 @@ Admission controllers can help to improve the security of the cluster. However, they can present risks themselves as they extend the API server and [should be properly secured](/blog/2022/01/19/secure-your-admission-controllers-and-webhooks/). -The following list present a number of admission controllers that could be +The following lists present a number of admission controllers that could be considered to enhance the security posture of your cluster and application. It -includes controllers that may be referenced in other parts of this document: +includes controllers that may be referenced in other parts of this document. -[`AlwaysPullImages`](/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages) -: Enforces the usage of the latest version of a tagged image and ensures that the deployer -has permissions to use the image. +This first group of admission controllers includes plugins +[enabled by default](/docs/reference/access-authn-authz/admission-controllers/#which-plugins-are-enabled-by-default), +consider to leave them enabled unless you know what you are doing: [`CertificateApproval`](/docs/reference/access-authn-authz/admission-controllers/#certificateapproval) : Performs additional authorization checks to ensure the approving user has @@ -334,20 +334,31 @@ permission to sign certificate requests. : Rejects any certificate request that specifies a 'group' (or 'organization attribute') of `system:masters`. -[`DenyServiceExternalIPs`](/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips) -: Rejects all net-new usage of the `Service.spec.externalIPs` field. This is a mitigation for -[CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs](https://github.com/kubernetes/kubernetes/issues/97076). - -[`EventRateLimit`](/docs/reference/access-authn-authz/admission-controllers/#eventratelimit) -: Rate limits adding new Events to the API server. - -[`ImagePolicyWebhook`](/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook) -: Allows enforcing additional controls for images through webhooks. +[`LimitRanger`](/docs/reference/access-authn-authz/admission-controllers/#limitranger) +: Enforce the LimitRange API constraints. [`MutatingAdmissionWebhook`](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) : Allows the use of custom controllers through webhooks, these controllers may mutate requests that it reviews. +[`PodSecurity`](/docs/reference/access-authn-authz/admission-controllers/#podsecurity) +: Replacement for Pod Security Policy, restricts security contexts of deployed +Pods. + +[`ResourceQuota`](/docs/reference/access-authn-authz/admission-controllers/#resourcequota) +: Enforces resource quotas to prevent over-usage of resources. + +[`ValidatingAdmissionWebhook`](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook) +: Allows the use of custom controllers through webhooks, these controllers do +not mutate requests that it reviews. + +The second group includes plugin that are not enabled by default but in general +availability state and recommended to improve your security posture: + +[`DenyServiceExternalIPs`](/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips) +: Rejects all net-new usage of the `Service.spec.externalIPs` field. This is a mitigation for +[CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs](https://github.com/kubernetes/kubernetes/issues/97076). + [`NodeRestriction`](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) : Restricts kubelet's permissions to only modify the pods API resources they own or the node API ressource that represent themselves. It also prevents kubelet @@ -355,26 +366,27 @@ from using the `node-restriction.kubernetes.io/` annotation, which can be used by an attacker with access to the kubelet's credentials to influence pod placement to the controlled node. -[`PodNodeSelector`](/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) -: Allows controls of node selectors within namespaces and cluster-wide. +The third group includes plugins that are not enabled by default but could be +considered for certain use cases: -[`PodSecurity`](/docs/reference/access-authn-authz/admission-controllers/#podsecurity) -: Replacement for Pod Security Policy, restricts security contexts of deployed -Pods. +[`AlwaysPullImages`](/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages) +: Enforces the usage of the latest version of a tagged image and ensures that the deployer +has permissions to use the image. -[`PodTolerationRestriction`](/docs/reference/access-authn-authz/admission-controllers/#podtolerationrestriction) -: Allows control of pod tolerations permitted for pods within a namespace. +[`ImagePolicyWebhook`](/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook) +: Allows enforcing additional controls for images through webhooks. -[`ResourceQuota`](/docs/reference/access-authn-authz/admission-controllers/#resourcequota) -: Enforces resource quotas to prevent over-usage of resources. + ## What's next From d4fcf2fc7c81001e48d1a3a0b0fe430083c63f0f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mah=C3=A9?= Date: Wed, 31 Aug 2022 18:29:43 +0200 Subject: [PATCH 13/16] Reword the secret injection suggestion --- content/en/docs/concepts/security/security-checklist.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index bff786c2499bf..2cfe814cc7681 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -232,8 +232,8 @@ overhead. - [ ] ConfigMaps are not used to hold confidential data. - [ ] Encryption at rest is configured for the Secret API. -- [ ] A mechanism to inject secrets stored in third party storage is deployed - and available. +- [ ] If appropriate, a mechanism to inject secrets stored in third-party storage + is deployed and available. - [ ] Service account tokens are not mounted in pods that don't require them. - [ ] [Bound service account token volume](/docs/reference/access-authn-authz/service-accounts-admin/#bound-service-account-token-volume) is in-use instead of non-expiring tokens. From a4305381fb19a326aac0d7d315f0cb6409031de3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mah=C3=A9?= Date: Wed, 31 Aug 2022 18:29:59 +0200 Subject: [PATCH 14/16] Reword the service mesh suggestion --- content/en/docs/concepts/security/security-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 2cfe814cc7681..f317c4bcae10b 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -55,7 +55,7 @@ an admin user. cluster. - [ ] Default network policies within each namespace, selecting all pods, denying everything, are in place. -- [ ] Use of a service mesh to encrypt all communications inside of the cluster. +- [ ] If appropriate, a service mesh is used to encrypt all communications inside of the cluster. - [ ] The Kubernetes API, kubelet API and etcd are not exposed publicly on Internet. - [ ] Access from the workloads to the cloud metadata API is filtered. - [ ] Use of LoadBalancer and ExternalIPs is restricted. From eb962b4c124212b0c3cf2a01a93aba9bc251cb7c Mon Sep 17 00:00:00 2001 From: mtardy Date: Thu, 1 Sep 2022 11:43:28 +0200 Subject: [PATCH 15/16] Rewrite the part on the Pod Security standards and admission --- .../concepts/security/security-checklist.md | 35 ++++++++++++------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index f317c4bcae10b..026683ff4c404 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -99,7 +99,7 @@ for further information. ## Pod security - [ ] RBAC rights to `create`, `update`, `patch`, `delete` workloads is only granted if necessary. -- [ ] Appropriate Pod Security Standards policy is applied for all namespaces in `enforce` mode. +- [ ] Appropriate Pod Security Standards policy is applied for all namespaces and enforced. - [ ] Memory limit is set for the workloads with a limit equal or inferior to the request. - [ ] CPU limit might be set on sensitive workloads. - [ ] For nodes that support it, Seccomp is enabled with appropriate syscalls @@ -114,18 +114,29 @@ on the resource itself, for example, `create` on Pods. Without additional admission, the authorization to create these resources allows direct unrestricted access to the schedulable nodes of a cluster. -[Pod Security Admission](/docs/concepts/security/pod-security-admission/) is an -admission controller to enable fine-grained authorization of Pod creation and -updates. It replaces the PodSecurityPolicy admission controller. The [Pod Security Standards](/docs/concepts/security/pod-security-standards/) -define three different policies that restrict how fields can be set in the `PodSpec`. -For a hands-on tutorial on Pod Security, see the blog post -"[Kubernetes 1.23: Pod Security Graduates to Beta](/blog/2021/12/09/pod-security-admission-beta/)". - -Pod Security admission `restricted` policy [can operate in several modes](/docs/concepts/security/pod-security-admission/#pod-security-admission-labels-for-namespaces), -`enforce`, `audit` or `warn` to help properly set the [security context](/docs/tasks/configure-pod-container/security-context/) -according to security best practices. Nevertheless, pods' [security context](/docs/tasks/configure-pod-container/security-context/) -should be separately investigated to limit the privileges and access pods may have. +define three different policies, privileged, baseline and restricted that limit +how fields can be set in the `PodSpec` regarding security. +These standards can be enforced at the namespace level with the new +[Pod Security](/docs/concepts/security/pod-security-admission/) admission, +enabled by default, or by third-party admission webhook. Please note that, +contrary to the removed PodSecurityPolicy admission it replaces, +[Pod Security](/docs/concepts/security/pod-security-admission/) +admission can be easily combined with admission webhooks and external services. + +Pod Security admission `restricted` policy, the most restrictive policy of the +[Pod Security Standards](/docs/concepts/security/pod-security-standards/) set, +[can operate in several modes](/docs/concepts/security/pod-security-admission/#pod-security-admission-labels-for-namespaces), +`warn`, `audit` or `enforce` to gradually apply the most appropriate +[security context](/docs/tasks/configure-pod-container/security-context/) +according to security best practices. Nevertheless, pods' +[security context](/docs/tasks/configure-pod-container/security-context/) +should be separately investigated to limit the privileges and access pods may +have on top of the predefined security standards, for specific use cases. + +For a hands-on tutorial on [Pod Security](/docs/concepts/security/pod-security-admission/), +see the blog post +[Kubernetes 1.23: Pod Security Graduates to Beta](/blog/2021/12/09/pod-security-admission-beta/). [Memory and CPU limits](/docs/concepts/configuration/manage-resources-containers/) should be set in order to restrict the memory and CPU resources a pod can From 9f5a35978f2de76df20ead8e87094d04f912041f Mon Sep 17 00:00:00 2001 From: mtardy Date: Thu, 1 Sep 2022 11:44:55 +0200 Subject: [PATCH 16/16] RBAC guide is presented as a checklist item --- content/en/docs/concepts/security/security-checklist.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/en/docs/concepts/security/security-checklist.md b/content/en/docs/concepts/security/security-checklist.md index 026683ff4c404..bcdacde86e2e8 100644 --- a/content/en/docs/concepts/security/security-checklist.md +++ b/content/en/docs/concepts/security/security-checklist.md @@ -38,9 +38,8 @@ evaluated on its merits. years in the future. - [ ] A process exists for periodic access review, and reviews occur no more than 24 months apart. - -See the [Role Based Access Control Good Practices](/docs/concepts/security/rbac-good-practices/) -for guidance related to authentication and authorization. +- [ ] The [Role Based Access Control Good Practices](/docs/concepts/security/rbac-good-practices/) + is followed for guidance related to authentication and authorization. After bootstrapping, neither users nor components should authenticate to the Kubernetes API as `system:masters`. Similarly, running all of