diff --git a/keps/prod-readiness/sig-node/4265.yaml b/keps/prod-readiness/sig-node/4265.yaml new file mode 100644 index 000000000000..ae440ff55854 --- /dev/null +++ b/keps/prod-readiness/sig-node/4265.yaml @@ -0,0 +1,5 @@ +kep-number: 4265 +alpha: + approver: "@deads2k" +beta: + approver: "@deads2k" diff --git a/keps/sig-node/4265-proc-mount/README.md b/keps/sig-node/4265-proc-mount/README.md new file mode 100644 index 000000000000..bc8efc5a972f --- /dev/null +++ b/keps/sig-node/4265-proc-mount/README.md @@ -0,0 +1,488 @@ +# KEP-4265: add ProcMount option + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Story 3](#story-3) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta](#beta) + - [GA](#ga) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + +Currently the most container runtimes work is by masking and setting as read-only certain paths in `/proc`. +This is to prevent data from being exposed into a container that should not be. +However, there are certain use-cases where it is necessary to turn this off. + +In 1.12, this was introduced as the ProcMountType feature gate, and has languished since. This KEP is +a successor to (and heavily based on) https://github.com/kubernetes/community/pull/1934/, updated for the modern era. + +## Motivation + +For end-users who would like to run unprivileged containers using user namespaces _nested inside_ CRI containers, we need an option to have a `ProcMount`. +That is, we need an option to designate explicitly turn off masking and setting read-only of paths so that we can mount `/proc` in the nested container as an unprivileged user. + +Please see the following filed issues for more information: +- [opencontainers/runc#1658](https://github.com/opencontainers/runc/issues/1658#issuecomment-373122073) +- [moby/moby#36597](https://github.com/moby/moby/issues/36597) +- [moby/moby#36644](https://github.com/moby/moby/pull/36644) + +### Goals + +- Add the ProcMountType option to the pod API, allowing users to override the proc mount behavior in a container. + +### Non-Goals + +## Proposal + + +Add a new `string` type field named `ProcMountType` will hold the viable +options for `procMount` to the `SecurityContext` +definition. + +By default,`procMount` is `default`, aka the same behavior as today and the +paths are masked. + +This will look like the following in the spec: + +```go +type ProcMountType string + +const ( + // DefaultProcMount uses the container runtime default ProcType. Most + // container runtimes mask certain paths in /proc to avoid accidental security + // exposure of special devices or information. + DefaultProcMount ProcMountType = "Default" + + // UnmaskedProcMount bypasses the default masking behavior of the container + // runtime and ensures the newly created /proc the container stays in tact with + // no modifications. + UnmaskedProcMount ProcMountType = "Unmasked" +) + +procMount *ProcMountType +``` + +This requires changes to the CRI runtime integrations so that kubelet will add the specific `unmasked` option. + +Note: Pod Security Admission will also need to be taught how to handle this option. +[Luckily, this has been done after the feature was introduced.](https://github.com/kubernetes/kubernetes/pull/103340) + +Unmasking the paths in `/proc` option really only makes sense for when a user is nesting unprivileged containers with user namespaces as it will allow more information +than is necessary to the program running in the container spawned by kubernetes. + +It should be noted that this is different that the host /proc. It is still a newly mounted /proc just the container runtimes will not mask the paths. + +Since the only use case for this option is to run unprivileged nested containers, this option should only be allowed or used if the user in the container is not `root`. +This can be easily enforced with `MustRunAs`. + +Since the user inside is still unprivileged, doing things to `/proc` would be off limits regardless, since linux user support already prevents this. + +### User Stories (Optional) + +#### Story 1 + +As a cluster admin, I would like a way to nest containers within containers and get access to proc entries as if the top level container was running +in the host context. + +#### Story 2 + +As a cluster admin, I may want to [genuinetools/img](https://github.com/genuinetools/img) inside a kubernetes container. +That program then launches sub-containers that take advantage of user namespaces and re-mask /proc and set /proc as read-only. + +#### Story 3 + +As a kubernetes user, I may want to build containers from within a kubernetes container. +See [this article for more information](https://github.com/jessfraz/blog/blob/master/content/post/building-container-images-securely-on-kubernetes.md). + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + + + +## Design Details + +### Test Plan + + + +[ ] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- ``: `` - `` + +##### Integration tests + + + + + +- : + +##### e2e tests + +- test/e2e_node + +additional tests should be added to e2e_node suite to test the adherence of the ProcMount field field + +### Graduation Criteria + + + +#### Alpha + +- Feature implemented behind a feature flag + +#### Beta + +- Add e2e tests for the feature + +#### GA + +- Allowing time for feedback + + +### Upgrade / Downgrade Strategy + + +Turn off the feature gate to turn off the feature. + +### Version Skew Strategy + + +It has been in support by kube-apiserver and kubelet since 1.12, so version skew is no longer a worry. + +## Production Readiness Review Questionnaire + +### Feature Enablement and Rollback + +###### How can this feature be enabled / disabled in a live cluster? + +- [x] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: ProcMountType + - Components depending on the feature gate: kubelet, kube-apiserver + +###### Does enabling the feature change any default behavior? + +No, only allows a user to access the Unmasked ProcMountType + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + +Yes + +###### What happens if we reenable the feature if it was previously rolled back? + +Nothing unexpected. + +###### Are there any tests for feature enablement/disablement? + +No + +### Rollout, Upgrade and Rollback Planning + +###### How can a rollout or rollback fail? Can it impact already running workloads? + +Inconsistent feature gate enablement is the only way. All supported versions of CRIs (CRI-O and containerd) have support for this feature. + +###### What specific metrics should inform a rollback? + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + +No, though since its a fairly contained feature I would be surprised if there are issues. + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + +No + +### Monitoring Requirements + +###### How can an operator determine if the feature is in use by workloads? + +Can check feature gates enabled on kubelet and kube-apiserver. + +###### How can someone using this feature know that it is working for their instance? + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [x] Other (treat as last resort) + - Details: successfully create a pod with a container that uses ProcMountType + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + +N/A + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [x] Other (treat as last resort) + - Details: general health of pods that use this feature. + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + +### Dependencies + +###### Does this feature depend on any specific services running in the cluster? + +- A CRI implementation that supports this feature + - All supported versions currently do. + +### Scalability + +###### Will enabling / using this feature result in any new API calls? + +No + +###### Will enabling / using this feature result in introducing new API types? + +ProcMountType in the pod spec + +###### Will enabling / using this feature result in any new calls to the cloud provider? + +No + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + +Negligible + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + +No + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + +No + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + +Potentially a malicious user given access and running with a root container in the host context could mess with the host processes, but PSA has already +been configured to fix this. + +### Troubleshooting + +###### How does this feature react if the API server and/or etcd is unavailable? + +No effect + +###### What are other known failure modes? + +Malicious user, protected by PSA. + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +2018-05-07: k/community update opened +2018-05-27: k/kubernetes PR merged with support. +2023-10-02: KEP opened and targeted at Beta + +## Drawbacks + + +## Alternatives + +## Infrastructure Needed (Optional) diff --git a/keps/sig-node/4265-proc-mount/kep.yaml b/keps/sig-node/4265-proc-mount/kep.yaml new file mode 100644 index 000000000000..13eba5cc770a --- /dev/null +++ b/keps/sig-node/4265-proc-mount/kep.yaml @@ -0,0 +1,39 @@ +title: ProcMountType +kep-number: 4265 +authors: + - "@haircommander" +owning-sig: sig-node +participating-sigs: + - sig-auth +status: implementable +creation-date: 2018-05-07 +reviewers: +approvers: + - "@mrunalp" + +replaces: + - "https://github.com/kubernetes/community/pull/1934/" + +# The target maturity stage in the current dev cycle for this KEP. +stage: beta + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.29" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.12" + beta: "v1.29" + stable: "v1.31" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: ProcMountType + components: + - kube-apiserver + - kubelet +disable-supported: true +metrics: