From da31c9390a0cdef8105e90194a9630b5be316b7e Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Wed, 2 Oct 2019 08:43:41 +0100 Subject: [PATCH 01/12] Initial draft --- .../20191002-seccomp-custom-profiles.md | 249 ++++++++++++++++++ 1 file changed, 249 insertions(+) create mode 100644 keps/sig-node/20191002-seccomp-custom-profiles.md diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md new file mode 100644 index 00000000000..a013bc2c15e --- /dev/null +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -0,0 +1,249 @@ +--- +title: seccomp-custom-profiles +authors: + - "@pjbgf" +owning-sig: sig-node +participating-sigs: + - sig-auth + - sig-apimachinery +reviewers: + - TBD +approvers: + - TBD +editor: TBD +creation-date: 2019-10-02 +last-updated: 2019-10-02 +status: provisional +see-also: + - "https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/seccomp.md" + - "https://github.com/kubernetes/enhancements/pull/1148" + - "https://github.com/kubernetes/enhancements/pull/1257" +replaces: + - "https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/seccomp.md" +superseded-by: {} +--- + +# seccomp-custom-profiles + +## Table of Contents + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [Usage Scenarios](#usage-scenarios) + - [Profile tampering protection](#profile-tampering-protection) + - [Rollout of profile changes](#rollout-of-profile-changes) + - [Starting containers with non-existent profile](#starting-containers-with-non-existent-profile) + - [User Stories](#user-stories) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Story 3](#story-3) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) + - [Graduation Criteria](#graduation-criteria) +- [Implementation History](#implementation-history) +- [Alternatives](#alternatives) +- [References](#references) + + +## Release Signoff Checklist + +**ACTION REQUIRED:** In order to merge code into a release, there must be an issue in [kubernetes/enhancements] referencing this KEP and targeting a release milestone **before [Enhancement Freeze](https://github.com/kubernetes/sig-release/tree/master/releases) +of the targeted release**. + +For enhancements that make changes to code or processes/procedures in core Kubernetes i.e., [kubernetes/kubernetes], we require the following Release Signoff checklist to be completed. + +Check these off as they are completed for the Release Team to track. These checklist items _must_ be updated for the enhancement to be released. + +- [ ] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR) +- [ ] KEP approvers have set the KEP status to `implementable` +- [ ] Design details are appropriately documented +- [ ] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input +- [ ] Graduation criteria is in place +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + +**Note:** Any PRs to move a KEP to `implementable` or significant changes once it is marked `implementable` should be approved by each of the KEP approvers. If any of those approvers is no longer appropriate than changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG-arch for cross cutting KEPs). + +**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone. + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://github.com/kubernetes/enhancements/issues +[kubernetes/kubernetes]: https://github.com/kubernetes/kubernetes +[kubernetes/website]: https://github.com/kubernetes/website + +## Summary + +A proposal to enhance the user-defined profiles support for seccomp. + + + +## Motivation + +The main motivation for this change is to bring wider seccomp adoption by making it easier for users to create and maintain their own seccomp profiles. +Use-defined profiles is a fundamental piece for users to get real value out of using seccomp, allowing them to decrease their cluster's attack surface. + +The current support relies on profiles being saved as files in the nodes. They need to exist across all cluster nodes where the pods using them may be scheduled. It is the user's responsibility to ensure the files are correctly saved on the nodes and their changes are synchronised as the profiles evolve. The scope of this proposal is to make that process more seamless. + +Kubernetes supports Seccomp in some capacity since version 1.3, but from then on this feature has been kept largely untouched. This change comes quite timely as there are now two other KEPs bringing enhancements on this space by [making this feature GA](https://github.com/kubernetes/enhancements/pull/1148) and [enabling it by default on audit mode](https://github.com/kubernetes/enhancements/pull/1257). + + +### Goals + +- Add support to load seccomp profiles from ConfigMaps. +- Provide a new mechanism for profiles to be sent to CRI without the use of files in the node's filesystem. +- Avoid breaking changes for Kubernetes api and user workloads. + + +### Non-Goals + +- Changes to make Seccomp GA. This is being covered by another [KEP](20190717-seccomp-ga.md). +- Changes to `PodSecurityPolicy`. +- Windows Support. + + + +## Proposal + +Add support to user-defined profiles being loaded from `ConfigMap` objects. The unstructured nature of this object type removes the potential tight coupling with OCI and downstream components, and the impact in case the seccomp profile format was to change in the future. + +Users will be able to create profiles in `ConfigMap` objects and refer to them as `configmap/`. The `ConfigMap` objects will have to be placed in the same namespace as where the Pod wills reside. Reusable cross namespaces user-defined profiles will not be supported at this point. + +The profile definition would be passed to the CRI as a serialised json object inside an dockerOpt object, in the same way that it is done currently for file based profiles, removing the need of files being saved and synchronised across nodes. + +``` +jsonSeccomp, _ := json.Marshal(profile.Spec) +return []dockerOpt{{"seccomp", string(jsonSeccomp), seccompProfileName}}, nil +``` + + +User-defined Seccomp profiles would be created this way: + +``` +apiVersion: v1 +kind: ConfigMap +metadata: + name: my-custom-profile +data: + profile.json: |- + { + "defaultAction": "SCMP_ACT_LOG" + } +``` + +This profile would be referenced as `configmap/my-custom-profile`. + + +### Usage Scenarios +Below goes a few points to clarify the expected behaviour around user-defined profiles: + +#### Profile tampering protection +At container start time the sha256 hash of the profile contents will be taken and saved within the `pod.status.containerStatuses.SecurityContext` object. +This enables users to verify whether the applied profile is the exact same as the profile saved on the given `ConfigMap` object. + +This will require changes to `ContainerStatus`: + + ``` + // ContainerStatus represents the status of a container. +type ContainerStatus struct { + ... + // Status of the container. + SecurityContext SecurityContextStatus + ... +} + + // SecurityContextStatus represents the security context status of a container. +type SecurityContextStatus struct { + // SeccompProfileHash of the SeccompProfile applied. + SeccompProfileHash string +} + ``` + +A similar approach would also be benefitial to AppArmor profiles. + + +#### Rollout of profile changes + +Seccomp profiles are applied at container startup time, therefore updating an existing user-defined profile will not cause any changes to the running containers that are using it. They will need to be restarted in order for the changes to take effect, which users will have to manually to so. + +At a later stage a new KEP will cover the automatic rollout of profile changes as an opt-in feature, taking into account container settings to ensure minimum availability impact. + +#### Starting containers with non-existent profile + +An error message should be returned stating the profile could not be found. Containers targeted for seccomp profiles should not be able to start if their profiles are deleted. + +If a running container has its policy deleted, nothing will happen until it has to be restarted, which in this case would lead to a failure. + + + +### User Stories + +#### Story 1 +As a user, I want to be able to create and maintain custom seccomp profiles without having to physically copy files across all my existing (and new) nodes. + + +#### Story 2 +As a user, I want to be able to create and maintain custom seccomp profiles by using kubectl, in the same way I do other Kubernetes objects. + + +#### Story 3 +As a user, I want to be able to determine whether custom seccomp profiles have been changed since applied to running containers. + + + +## Design Details + +### Test Plan + +Seccomp already has E2E tests, but the tests are guarded by the [Feature:Seccomp] tag and not run in the standard test suites. + +Prior to [being marked GA](20190717-seccomp-ga.md), the feature tag will be removed from the seccomp tests. + +Test coverage for configmap profiles will be added and be tested in isolation. + +E2E tests will be required to ensure profile changes won't impact running containers. + +### Upgrade / Downgrade Strategy + +No upgrade changes required - both localhost and configmap profiles will be supported. + +### Version Skew Strategy + +If applicable, how will the component handle version skew with other components? What are the guarantees? Make sure +this is in the test plan. + +Consider the following in developing a version skew strategy for this enhancement: +- Does this enhancement involve coordinating behavior in the control plane and in the kubelet? How does an n-2 kubelet without this feature available behave when this feature is used? +- Will any other components on the node change? For example, changes to CSI, CRI or CNI may require updating that component before the kubelet. + +### Graduation Criteria + +- API changes to map seccomp profiles to an additional `Kubernetes` type. +- E2E tests + + +## Implementation History +- 2019-10-02: Initial KEP + + +## Alternatives + +**Start deprecation process for `localhost/`.** The new `ConfigMapSeccompProfile` will better support custom profiles. Starting the deprecation process would signal to users what the end goal is. However, this can be started once the new approach GA's. + + +**Downstream seccomp support awareness.** Validation could be added to assert whether the Seccomp Profile could be applied by the downstream dependencies on a _per- node_ basis, and lead to a list of available profiles for each node. This would benefit clusters with heterogeneus nodes. It would also benefit the usage of the current `localhost/` profile, which an administrator has no way to tell which nodes have them and which ones don't. + +This can be treated as an incremental enhancement in the future, based on the community feedback. + + +## References +- [Original Seccomp Proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/seccomp.md) +- [Seccomp GA KEP](https://github.com/kubernetes/enhancements/pull/1148) +- [Seccomp enabled by default](https://github.com/kubernetes/enhancements/pull/1257) From ce0d940f5f12e784186caa31d4ea25d9cc453622 Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Wed, 2 Oct 2019 08:59:26 +0100 Subject: [PATCH 02/12] Add Version Skew Strategy --- .../20191002-seccomp-custom-profiles.md | 25 +++++++------------ 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index a013bc2c15e..7e7102664e0 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -81,7 +81,7 @@ Check these off as they are completed for the Release Team to track. These check ## Summary -A proposal to enhance the user-defined profiles support for seccomp. +A proposal to enhance user-defined profiles support for seccomp by allowing them to be stored as `ConfigMap` objects. @@ -114,9 +114,9 @@ Kubernetes supports Seccomp in some capacity since version 1.3, but from then on Add support to user-defined profiles being loaded from `ConfigMap` objects. The unstructured nature of this object type removes the potential tight coupling with OCI and downstream components, and the impact in case the seccomp profile format was to change in the future. -Users will be able to create profiles in `ConfigMap` objects and refer to them as `configmap/`. The `ConfigMap` objects will have to be placed in the same namespace as where the Pod wills reside. Reusable cross namespaces user-defined profiles will not be supported at this point. +Users will be able to create profiles in `ConfigMap` objects and refer to them as `configmap/`. Note that the `ConfigMap` objects will have to be placed in the same namespace as where the Pods will reside. Reusable cross namespaces user-defined profiles will not be supported at this point. -The profile definition would be passed to the CRI as a serialised json object inside an dockerOpt object, in the same way that it is done currently for file based profiles, removing the need of files being saved and synchronised across nodes. +The profile definition would be passed to the CRI as a serialised json object inside a dockerOpt object, in the same way that it is done currently for file based profiles, removing the need of files being saved and synchronised across nodes. ``` jsonSeccomp, _ := json.Marshal(profile.Spec) @@ -138,14 +138,13 @@ data: } ``` -This profile would be referenced as `configmap/my-custom-profile`. +And the profile would be referenced as `configmap/my-custom-profile`. ### Usage Scenarios -Below goes a few points to clarify the expected behaviour around user-defined profiles: #### Profile tampering protection -At container start time the sha256 hash of the profile contents will be taken and saved within the `pod.status.containerStatuses.SecurityContext` object. +Just before the container creation time the sha256 hash of the profile contents will be taken and saved within the `pod.status.containerStatuses.SecurityContext` object. This enables users to verify whether the applied profile is the exact same as the profile saved on the given `ConfigMap` object. This will require changes to `ContainerStatus`: @@ -171,15 +170,14 @@ A similar approach would also be benefitial to AppArmor profiles. #### Rollout of profile changes -Seccomp profiles are applied at container startup time, therefore updating an existing user-defined profile will not cause any changes to the running containers that are using it. They will need to be restarted in order for the changes to take effect, which users will have to manually to so. +Seccomp profiles are applied at container creation time, therefore updating an existing user-defined profile will not cause any changes to the running containers that are using it. They will need to be restarted in order for the changes to take effect, which users will have to manually do. -At a later stage a new KEP will cover the automatic rollout of profile changes as an opt-in feature, taking into account container settings to ensure minimum availability impact. #### Starting containers with non-existent profile An error message should be returned stating the profile could not be found. Containers targeted for seccomp profiles should not be able to start if their profiles are deleted. -If a running container has its policy deleted, nothing will happen until it has to be restarted, which in this case would lead to a failure. +If a running container has its policy deleted, nothing will happen until it has to be restarted, which in this case would lead to a failure as stated above. @@ -212,16 +210,11 @@ E2E tests will be required to ensure profile changes won't impact running contai ### Upgrade / Downgrade Strategy -No upgrade changes required - both localhost and configmap profiles will be supported. +No upgrade changes required - both localhost and configmap profiles will be supported in parallel. ### Version Skew Strategy -If applicable, how will the component handle version skew with other components? What are the guarantees? Make sure -this is in the test plan. - -Consider the following in developing a version skew strategy for this enhancement: -- Does this enhancement involve coordinating behavior in the control plane and in the kubelet? How does an n-2 kubelet without this feature available behave when this feature is used? -- Will any other components on the node change? For example, changes to CSI, CRI or CNI may require updating that component before the kubelet. +The new configmap based profiles will only be supported from this version on. Users will need to ensure that pods using the new feature won't be scheduled on nodes running older Kubelet versions. ### Graduation Criteria From 0483cd4da5caa0b25bef98826e6b06bf61866e72 Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Wed, 2 Oct 2019 09:18:57 +0100 Subject: [PATCH 03/12] Add reviewers --- keps/sig-node/20191002-seccomp-custom-profiles.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index 7e7102664e0..f14897da9ce 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -5,11 +5,10 @@ authors: owning-sig: sig-node participating-sigs: - sig-auth - - sig-apimachinery reviewers: - - TBD + - "@tallclair" approvers: - - TBD + - "@tallclair" editor: TBD creation-date: 2019-10-02 last-updated: 2019-10-02 @@ -18,9 +17,6 @@ see-also: - "https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/seccomp.md" - "https://github.com/kubernetes/enhancements/pull/1148" - "https://github.com/kubernetes/enhancements/pull/1257" -replaces: - - "https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/seccomp.md" -superseded-by: {} --- # seccomp-custom-profiles From 425bcc923470512c72b2d4c3a4347361699753c9 Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Wed, 2 Oct 2019 09:24:08 +0100 Subject: [PATCH 04/12] Fix mispelling. --- keps/sig-node/20191002-seccomp-custom-profiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index f14897da9ce..ef5144c3941 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -161,7 +161,7 @@ type SecurityContextStatus struct { } ``` -A similar approach would also be benefitial to AppArmor profiles. +A similar approach would also be beneficial to AppArmor profiles. #### Rollout of profile changes From 5cb06963a245c0b676d3fa689a2ecdee5afd1eed Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Wed, 2 Oct 2019 10:57:27 +0100 Subject: [PATCH 05/12] Add new seccomp profile type. --- keps/sig-node/20191002-seccomp-custom-profiles.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index ef5144c3941..7df2f8f979b 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -119,6 +119,15 @@ jsonSeccomp, _ := json.Marshal(profile.Spec) return []dockerOpt{{"seccomp", string(jsonSeccomp), seccompProfileName}}, nil ``` +The new config profiles would be mapped to an additional Kubernetes profile type: +``` +const ( + SeccompProfileUnconfined SeccompProfileType = "Unconfined" + SeccompProfileRuntime SeccompProfileType = "Runtime" + SeccompProfileLocalhost SeccompProfileType = "Localhost" + SeccompProfileConfigmap SeccompProfileType = "ConfigMap" +) +``` User-defined Seccomp profiles would be created this way: From 3a28d21da753420793391b9d27f5808bc78b1363 Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Tue, 15 Oct 2019 14:42:36 +0100 Subject: [PATCH 06/12] Minor changes --- keps/sig-node/20191002-seccomp-custom-profiles.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index 7df2f8f979b..8e5f3c32712 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -11,7 +11,7 @@ approvers: - "@tallclair" editor: TBD creation-date: 2019-10-02 -last-updated: 2019-10-02 +last-updated: 2019-10-15 status: provisional see-also: - "https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/seccomp.md" @@ -119,7 +119,7 @@ jsonSeccomp, _ := json.Marshal(profile.Spec) return []dockerOpt{{"seccomp", string(jsonSeccomp), seccompProfileName}}, nil ``` -The new config profiles would be mapped to an additional Kubernetes profile type: +The new config profiles would be mapped to an additional Kubernetes profile type, which is contingent on the GA API changes proposed in [#1148](https://github.com/kubernetes/enhancements/pull/1148): ``` const ( SeccompProfileUnconfined SeccompProfileType = "Unconfined" @@ -175,7 +175,8 @@ A similar approach would also be beneficial to AppArmor profiles. #### Rollout of profile changes -Seccomp profiles are applied at container creation time, therefore updating an existing user-defined profile will not cause any changes to the running containers that are using it. They will need to be restarted in order for the changes to take effect, which users will have to manually do. +Seccomp profiles are applied at container creation time, therefore updating an existing user-defined profile will not cause any changes to the running containers that are using it. +They will need to be restarted in order for the changes to take effect, which users will have to do manually. #### Starting containers with non-existent profile @@ -229,11 +230,12 @@ The new configmap based profiles will only be supported from this version on. Us ## Implementation History - 2019-10-02: Initial KEP +- 2019-10-15: Minor changes ## Alternatives -**Start deprecation process for `localhost/`.** The new `ConfigMapSeccompProfile` will better support custom profiles. Starting the deprecation process would signal to users what the end goal is. However, this can be started once the new approach GA's. +**Start deprecation process for `localhost/`.** The new `ConfigMapSeccompProfile` will better support custom profiles. Starting the deprecation process would signal users what the end goal is. However, this can be started once the new approach GA's. **Downstream seccomp support awareness.** Validation could be added to assert whether the Seccomp Profile could be applied by the downstream dependencies on a _per- node_ basis, and lead to a list of available profiles for each node. This would benefit clusters with heterogeneus nodes. It would also benefit the usage of the current `localhost/` profile, which an administrator has no way to tell which nodes have them and which ones don't. From 7bd205698f4eb7cadae200efd6bb7e2d065d6a01 Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Thu, 17 Oct 2019 08:13:19 +0100 Subject: [PATCH 07/12] Add alternative of using CRDs instead --- keps/sig-node/20191002-seccomp-custom-profiles.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index 8e5f3c32712..d6fe0e02a70 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -231,10 +231,19 @@ The new configmap based profiles will only be supported from this version on. Us ## Implementation History - 2019-10-02: Initial KEP - 2019-10-15: Minor changes +- 2019-10-17: Add alternative to use CRD instead of ConfigMap ## Alternatives +**Use CRD instead of ConfigMap.** The decision to use `ConfigMap` was to avoid unnecessary complexity. Below are some key points from the [official guidance](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#should-i-use-a-configmap-or-a-custom-resource) which clarifies the reasoning: + +- There is an existing, well-documented config file format. +- The entire config file will be stored into one key of a configMap. +- The main use of the config file is for the cluster to consume the file to pass it on to CRIs. +- Users could potentially perform rolling updates via Deployment, when the file is updated. + + **Start deprecation process for `localhost/`.** The new `ConfigMapSeccompProfile` will better support custom profiles. Starting the deprecation process would signal users what the end goal is. However, this can be started once the new approach GA's. From 097e699555e9c532f9e26db17322d7ccc274f4f3 Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Sun, 27 Oct 2019 17:24:47 +0000 Subject: [PATCH 08/12] Improvements based on feedback --- .../20191002-seccomp-custom-profiles.md | 84 ++++++++++++++++--- 1 file changed, 73 insertions(+), 11 deletions(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index d6fe0e02a70..6055551c9ed 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -11,7 +11,7 @@ approvers: - "@tallclair" editor: TBD creation-date: 2019-10-02 -last-updated: 2019-10-15 +last-updated: 2019-10-27 status: provisional see-also: - "https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/seccomp.md" @@ -39,10 +39,15 @@ see-also: - [Story 2](#story-2) - [Story 3](#story-3) - [Design Details](#design-details) + - [No persistence to disk](#no-persistence-to-disk) + - [Feature Gate](#feature-gate) - [Test Plan](#test-plan) - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) - [Version Skew Strategy](#version-skew-strategy) - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Alpha -> Beta Graduation](#alpha---beta-graduation) + - [Beta -> GA Graduation](#beta---ga-graduation) - [Implementation History](#implementation-history) - [Alternatives](#alternatives) - [References](#references) @@ -120,7 +125,8 @@ return []dockerOpt{{"seccomp", string(jsonSeccomp), seccompProfileName}}, nil ``` The new config profiles would be mapped to an additional Kubernetes profile type, which is contingent on the GA API changes proposed in [#1148](https://github.com/kubernetes/enhancements/pull/1148): -``` + +```golang const ( SeccompProfileUnconfined SeccompProfileType = "Unconfined" SeccompProfileRuntime SeccompProfileType = "Runtime" @@ -131,7 +137,7 @@ const ( User-defined Seccomp profiles would be created this way: -``` +```yaml apiVersion: v1 kind: ConfigMap metadata: @@ -154,7 +160,7 @@ This enables users to verify whether the applied profile is the exact same as th This will require changes to `ContainerStatus`: - ``` + ```golang // ContainerStatus represents the status of a container. type ContainerStatus struct { ... @@ -178,13 +184,16 @@ A similar approach would also be beneficial to AppArmor profiles. Seccomp profiles are applied at container creation time, therefore updating an existing user-defined profile will not cause any changes to the running containers that are using it. They will need to be restarted in order for the changes to take effect, which users will have to do manually. +However, the recommended approach for making profile changes is to create a new `ConfigMap` with the desired profile and then trigger a rolling-update, by changing the configmap the pods are pointing to. The old `ConfigMap` can be deleted once there are no running pods referring to it. + #### Starting containers with non-existent profile -An error message should be returned stating the profile could not be found. Containers targeted for seccomp profiles should not be able to start if their profiles are deleted. +When the Kubelet is creating a container using this seccomp profile type, it must validate that the container is referring to a `ConfigMap` that exists inside the same namespace. That `ConfigMap` should contain a `profile.json` file inside of it with a valid json value. If either of those evaluations is false, the container should not be created and an error should be logged instead. -If a running container has its policy deleted, nothing will happen until it has to be restarted, which in this case would lead to a failure as stated above. +Once the container is running, the `ConfigMap` holding the profile will not be re-evaluated. If that `ConfigMap` is deleted at this point, nothing will happen until the container is restarted, in which case the same rules as above will apply, leading to the container failing to start. +The Kubelet will only validate whether the contents of the `profile.json` file is a valid `json` file. The container runtime will be responsible for validating it further. The Kubelet will not validate or use any other files apart from `profile.json`. ### User Stories @@ -204,13 +213,40 @@ As a user, I want to be able to determine whether custom seccomp profiles have b ## Design Details +### No persistence to disk + +The profiles will not persisted to disk. The Kubelet will fetch the contents from `ConfigMap` and then returned to be later pass on to the OCI, as per code below: + +```golang +configMapName := strings.TrimPrefix(seccompProfile, "configmap/") +seccompCfg, err := apiclient.GetConfigMapWithRetry(client, podNamespace, configMapName) +... +profileData, ok := seccompCfg.Data[SeccompConfigMapFileName] +if !ok { + ... +} +if !isValidJson(profileData){ + ... +} + +return []dockerOpt{{"seccomp", profileData, ""}}, nil +``` + +### Feature Gate + +A new feature gate will be created to enable the use of Seccomp ConfigMaps, as per below: + +`--feature-gate=SeccompConfigMaps=true` + +The proposed default value for this feature gate will change through the [Graduation Process](#graduation-criteria). + ### Test Plan Seccomp already has E2E tests, but the tests are guarded by the [Feature:Seccomp] tag and not run in the standard test suites. Prior to [being marked GA](20190717-seccomp-ga.md), the feature tag will be removed from the seccomp tests. -Test coverage for configmap profiles will be added and be tested in isolation. +Unit tests will be created for both kubelet and API server to account for configmap profiles, taking into account scenarios mentioned at the [usage scenarios](#usage-scenarios) section. E2E tests will be required to ensure profile changes won't impact running containers. @@ -220,18 +256,44 @@ No upgrade changes required - both localhost and configmap profiles will be supp ### Version Skew Strategy -The new configmap based profiles will only be supported from this version on. Users will need to ensure that pods using the new feature won't be scheduled on nodes running older Kubelet versions. +Error handling for users trying to use the new `ConfigMap` based profiles on older versions of either Kubelet or API server is already implemented. Messages will differ depending on what component does not support the new feature: + +- **Not Supported at API Server** (i.e. before implementation of this proposal): +Pod creation will fail with the error message `must be a valid seccomp profile`. + +- **Supported at API Server but not at Kubelet** (i.e. Kubelet running on older version): +Pod creation will fail with the error message `unknown seccomp profile option: configmap/config-name`. + +The error message above may change once seccomp [becomes GA](20190717-seccomp-ga.md). ### Graduation Criteria -- API changes to map seccomp profiles to an additional `Kubernetes` type. -- E2E tests + +##### Alpha + +- API changes to map Built-in profiles to an additional `Kubernetes` type. +- Feature gate is disabled by default (i.e. `--feature-gate=SeccompConfigMaps=false`). +- Provide alpha level documentation. +- Unit tests. + +##### Alpha -> Beta Graduation + +- Feature gate is enabled by default (i.e. `--feature-gate=SeccompConfigMaps=true`). +- Provide beta level documentation. +- E2E tests. + +##### Beta -> GA Graduation + +- Provide comprehensive documentation. +- Feature gate is removed. +- Potential changes based on community feedback. ## Implementation History - 2019-10-02: Initial KEP - 2019-10-15: Minor changes - 2019-10-17: Add alternative to use CRD instead of ConfigMap +- 2019-10-27: Add improvements to Design, Version Skew Strategy and Graduation ## Alternatives @@ -244,7 +306,7 @@ The new configmap based profiles will only be supported from this version on. Us - Users could potentially perform rolling updates via Deployment, when the file is updated. -**Start deprecation process for `localhost/`.** The new `ConfigMapSeccompProfile` will better support custom profiles. Starting the deprecation process would signal users what the end goal is. However, this can be started once the new approach GA's. +**Start deprecation process for `localhost/`.** The proposed changes will better support custom profiles. Starting the deprecation process of the existing method would signal users what the end goal is. However, this can be started once the new approach GA's. **Downstream seccomp support awareness.** Validation could be added to assert whether the Seccomp Profile could be applied by the downstream dependencies on a _per- node_ basis, and lead to a list of available profiles for each node. This would benefit clusters with heterogeneus nodes. It would also benefit the usage of the current `localhost/` profile, which an administrator has no way to tell which nodes have them and which ones don't. From 95c053ab4d63842c7726e7e88dcfc533facee7dc Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Tue, 29 Oct 2019 16:40:26 +0000 Subject: [PATCH 09/12] Improve alternatives & add multiple files support --- .../20191002-seccomp-custom-profiles.md | 75 +++++++++++++------ 1 file changed, 54 insertions(+), 21 deletions(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index 6055551c9ed..60a4046b8ff 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -11,7 +11,7 @@ approvers: - "@tallclair" editor: TBD creation-date: 2019-10-02 -last-updated: 2019-10-27 +last-updated: 2019-10-29 status: provisional see-also: - "https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/seccomp.md" @@ -50,6 +50,11 @@ see-also: - [Beta -> GA Graduation](#beta---ga-graduation) - [Implementation History](#implementation-history) - [Alternatives](#alternatives) + - [Use CRD instead of ConfigMap](#use-crd-instead-of-configmap) + - [Reasons for ConfigMap](#reasons-for-) + - [Reasons against it](#reasons-against-it) + - [Start the deprecation process for localhost/<path>](#start-the-deprecation-process-for-) + - [Downstream seccomp support awareness](#downstream-seccomp-support-awareness) - [References](#references) @@ -115,9 +120,9 @@ Kubernetes supports Seccomp in some capacity since version 1.3, but from then on Add support to user-defined profiles being loaded from `ConfigMap` objects. The unstructured nature of this object type removes the potential tight coupling with OCI and downstream components, and the impact in case the seccomp profile format was to change in the future. -Users will be able to create profiles in `ConfigMap` objects and refer to them as `configmap/`. Note that the `ConfigMap` objects will have to be placed in the same namespace as where the Pods will reside. Reusable cross namespaces user-defined profiles will not be supported at this point. +Users will be able to create profiles in `ConfigMap` objects and refer to them as `configmap//`. Note that the `ConfigMap` objects will have to be placed in the same namespace as where the Pods will reside. Reusable cross namespaces user-defined profiles will not be supported at this point. -The profile definition would be passed to the CRI as a serialised json object inside a dockerOpt object, in the same way that it is done currently for file based profiles, removing the need of files being saved and synchronised across nodes. +The profile definition would be passed to the CRI as a serialised json object inside a `dockerOpt` object, in the same way that it is done currently for file based profiles, removing the need of files being saved and synchronised across nodes. ``` jsonSeccomp, _ := json.Marshal(profile.Spec) @@ -141,15 +146,24 @@ User-defined Seccomp profiles would be created this way: apiVersion: v1 kind: ConfigMap metadata: - name: my-custom-profile + name: webapi-seccomp data: - profile.json: |- - { - "defaultAction": "SCMP_ACT_LOG" - } + profile-block.json: |- + { + "defaultAction": "SCMP_ACT_ERRNO" + } + profile-complain.json: |- + { + "defaultAction": "SCMP_ACT_LOG" + } ``` -And the profile would be referenced as `configmap/my-custom-profile`. +The two profiles inside the `ConfigMap` above would be referenced respectively by: + +- `configmap/webapi-seccomp/profile-block.json` +- `configmap/webapi-seccomp/profile-complain.json` + +Neither the configmap nor the file inside of it needs to have a specific name. ### Usage Scenarios @@ -189,11 +203,11 @@ However, the recommended approach for making profile changes is to create a new #### Starting containers with non-existent profile -When the Kubelet is creating a container using this seccomp profile type, it must validate that the container is referring to a `ConfigMap` that exists inside the same namespace. That `ConfigMap` should contain a `profile.json` file inside of it with a valid json value. If either of those evaluations is false, the container should not be created and an error should be logged instead. +When the Kubelet is creating a container using this seccomp profile type, it must validate that the profile is valid. For a profile reference of `configmap/webapi-seccomp/profile-block.json`, the Kubelet will check for the existence of a `ConfigMap` named `web-seccomp` in the same namespace as the pod/container referencing it. It will also check whether that `ConfigMap` contains a file named `profile-block.json` and that it is a valid json. If any of those evaluations are false, the container will not be created and an error should be logged instead. Once the container is running, the `ConfigMap` holding the profile will not be re-evaluated. If that `ConfigMap` is deleted at this point, nothing will happen until the container is restarted, in which case the same rules as above will apply, leading to the container failing to start. -The Kubelet will only validate whether the contents of the `profile.json` file is a valid `json` file. The container runtime will be responsible for validating it further. The Kubelet will not validate or use any other files apart from `profile.json`. +The Kubelet will only validate whether the contents of the profile file is a valid `json` file. The container runtime will be responsible for validating it further. The Kubelet will not validate or use any other files apart from the one being referenced. ### User Stories @@ -218,10 +232,11 @@ As a user, I want to be able to determine whether custom seccomp profiles have b The profiles will not persisted to disk. The Kubelet will fetch the contents from `ConfigMap` and then returned to be later pass on to the OCI, as per code below: ```golang -configMapName := strings.TrimPrefix(seccompProfile, "configmap/") -seccompCfg, err := apiclient.GetConfigMapWithRetry(client, podNamespace, configMapName) +scmpWithouPrefix := strings.TrimPrefix(seccompProfile, "configmap/") +configAndFile := strings.Split(scmpWithouPrefix, "/") +seccompCfg, err := apiclient.GetConfigMapWithRetry(client, podNamespace, configAndFile[0]) ... -profileData, ok := seccompCfg.Data[SeccompConfigMapFileName] +profileData, ok := seccompCfg.Data[configAndFile[1]] if !ok { ... } @@ -279,6 +294,7 @@ The error message above may change once seccomp [becomes GA](20190717-seccomp-ga ##### Alpha -> Beta Graduation - Feature gate is enabled by default (i.e. `--feature-gate=SeccompConfigMaps=true`). +- Approach community for feedback. - Provide beta level documentation. - E2E tests. @@ -294,22 +310,39 @@ The error message above may change once seccomp [becomes GA](20190717-seccomp-ga - 2019-10-15: Minor changes - 2019-10-17: Add alternative to use CRD instead of ConfigMap - 2019-10-27: Add improvements to Design, Version Skew Strategy and Graduation +- 2019-10-29: Add improvements to alternatives and multiple files support ## Alternatives -**Use CRD instead of ConfigMap.** The decision to use `ConfigMap` was to avoid unnecessary complexity. Below are some key points from the [official guidance](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#should-i-use-a-configmap-or-a-custom-resource) which clarifies the reasoning: +### Use CRD instead of ConfigMap + +The decision to use `ConfigMap` was to avoid unnecessary complexity. However, this can be later reviewed through the Graduation process. The reasons considered whilst assessing the two different approaches were: + +#### Reasons for `ConfigMap` + +- Smaller implementation footprint. +- Official type for storing files. +- Allows for multiple files to be stored on a single object. +- Due to its free form nature, provides loose coupling with downstream dependencies (i.e. OCI). + +#### Reasons against it + +- The configmap won't be mounted into a container, so it won't be reusing any Kubelet code. +- A CRD could be cluster-scoped (although I'm on the fence about whether that's actually desirable). +- A dedicated type could be immutable, to avoid some uncertainty around updates. +- A fully-structured CRD could provide static type checking for early failure (although this creates the undesirable tight coupling to the OCI format). + + -- There is an existing, well-documented config file format. -- The entire config file will be stored into one key of a configMap. -- The main use of the config file is for the cluster to consume the file to pass it on to CRIs. -- Users could potentially perform rolling updates via Deployment, when the file is updated. +#### Start the deprecation process for `localhost/` +The proposed changes will better support custom profiles. Starting the deprecation process of the existing method would signal users what the end goal is. However, this can be started once the new approach GA's. -**Start deprecation process for `localhost/`.** The proposed changes will better support custom profiles. Starting the deprecation process of the existing method would signal users what the end goal is. However, this can be started once the new approach GA's. +#### Downstream seccomp support awareness -**Downstream seccomp support awareness.** Validation could be added to assert whether the Seccomp Profile could be applied by the downstream dependencies on a _per- node_ basis, and lead to a list of available profiles for each node. This would benefit clusters with heterogeneus nodes. It would also benefit the usage of the current `localhost/` profile, which an administrator has no way to tell which nodes have them and which ones don't. +Validation could be added to assert whether the Seccomp Profile could be applied by the downstream dependencies on a _per- node_ basis, and lead to a list of available profiles for each node. This would benefit clusters with heterogeneus nodes. It would also benefit the usage of the current `localhost/` profile, which an administrator has no way to tell which nodes have them and which ones don't. This can be treated as an incremental enhancement in the future, based on the community feedback. From 19e93315fb4c8d9f4cf2c2f1e480105e335480f7 Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Tue, 29 Oct 2019 17:02:30 +0000 Subject: [PATCH 10/12] Add explict reference to namespace --- keps/sig-node/20191002-seccomp-custom-profiles.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index 60a4046b8ff..b9d4b6d96ef 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -146,16 +146,13 @@ User-defined Seccomp profiles would be created this way: apiVersion: v1 kind: ConfigMap metadata: + namespace: secure-projects-ns name: webapi-seccomp data: profile-block.json: |- - { - "defaultAction": "SCMP_ACT_ERRNO" - } + { "defaultAction": "SCMP_ACT_ERRNO", ... } profile-complain.json: |- - { - "defaultAction": "SCMP_ACT_LOG" - } + { "defaultAction": "SCMP_ACT_LOG", ... } ``` The two profiles inside the `ConfigMap` above would be referenced respectively by: @@ -163,7 +160,7 @@ The two profiles inside the `ConfigMap` above would be referenced respectively b - `configmap/webapi-seccomp/profile-block.json` - `configmap/webapi-seccomp/profile-complain.json` -Neither the configmap nor the file inside of it needs to have a specific name. +The only restriction for using those profiles is that the `webapi-seccomp` object must reside in the same namespace as the pods/containers that refers to it, in this case `secure-projects-ns`. ### Usage Scenarios @@ -234,6 +231,7 @@ The profiles will not persisted to disk. The Kubelet will fetch the contents fro ```golang scmpWithouPrefix := strings.TrimPrefix(seccompProfile, "configmap/") configAndFile := strings.Split(scmpWithouPrefix, "/") +... seccompCfg, err := apiclient.GetConfigMapWithRetry(client, podNamespace, configAndFile[0]) ... profileData, ok := seccompCfg.Data[configAndFile[1]] From d9b66d7f2d99dc31c335da2aec91678d5af7c2de Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Fri, 24 Apr 2020 07:06:28 +0100 Subject: [PATCH 11/12] Update keps/sig-node/20191002-seccomp-custom-profiles.md Replace OCI with CRI Co-Authored-By: Sascha Grunert --- keps/sig-node/20191002-seccomp-custom-profiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index b9d4b6d96ef..5c593ae62f2 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -226,7 +226,7 @@ As a user, I want to be able to determine whether custom seccomp profiles have b ### No persistence to disk -The profiles will not persisted to disk. The Kubelet will fetch the contents from `ConfigMap` and then returned to be later pass on to the OCI, as per code below: +The profiles will not persisted to disk. The Kubelet will fetch the contents from `ConfigMap` and then returned to be later pass on to the CRI, as per code below: ```golang scmpWithouPrefix := strings.TrimPrefix(seccompProfile, "configmap/") From a94d0b82ef063cab51c16bd1339041e42998b070 Mon Sep 17 00:00:00 2001 From: Paulo Gomes Date: Fri, 24 Apr 2020 09:43:03 +0100 Subject: [PATCH 12/12] Fix grammar mistakes --- keps/sig-node/20191002-seccomp-custom-profiles.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/keps/sig-node/20191002-seccomp-custom-profiles.md b/keps/sig-node/20191002-seccomp-custom-profiles.md index 5c593ae62f2..19469a03e37 100644 --- a/keps/sig-node/20191002-seccomp-custom-profiles.md +++ b/keps/sig-node/20191002-seccomp-custom-profiles.md @@ -226,7 +226,7 @@ As a user, I want to be able to determine whether custom seccomp profiles have b ### No persistence to disk -The profiles will not persisted to disk. The Kubelet will fetch the contents from `ConfigMap` and then returned to be later pass on to the CRI, as per code below: +The profiles will not be persisted to disk. The Kubelet will fetch the contents from the `ConfigMap` which will later be passed onto the CRI, as per code below: ```golang scmpWithouPrefix := strings.TrimPrefix(seccompProfile, "configmap/") @@ -309,7 +309,7 @@ The error message above may change once seccomp [becomes GA](20190717-seccomp-ga - 2019-10-17: Add alternative to use CRD instead of ConfigMap - 2019-10-27: Add improvements to Design, Version Skew Strategy and Graduation - 2019-10-29: Add improvements to alternatives and multiple files support - +- 2020-04-24: Fix grammar ## Alternatives