Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Rootless Docker #12359

Merged
merged 1 commit into from
Aug 30, 2021
Merged

Support Rootless Docker #12359

merged 1 commit into from
Aug 30, 2021

Conversation

AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda commented Aug 26, 2021

Fixes #10836 ("add support for rootless Docker").

Support for rootless Podman (issue #8719) is not covered in this PR.

Requirements

Usage

$ dockerd-rootless-setuptool.sh install -f
$ docker context use rootless
$ minikube start --driver=docker --container-runtime=containerd
...
* Preparing Kubernetes v1.22.1 on containerd 1.4.9 ...
  - Generating certificates and keys ...
  - Booting up control plane ...
  - Configuring RBAC rules ...
* Configuring CNI (Container Networking Interface) ...
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-78fcd69978-d5gmt           1/1     Running   0          26s
kube-system   etcd-minikube                      1/1     Running   0          33s
kube-system   kindnet-ss6xw                      1/1     Running   0          27s
kube-system   kube-apiserver-minikube            1/1     Running   0          33s
kube-system   kube-controller-manager-minikube   1/1     Running   0          32s
kube-system   kube-proxy-wltqb                   1/1     Running   0          27s
kube-system   kube-scheduler-minikube            1/1     Running   0          32s
kube-system   storage-provisioner                1/1     Running   0          35s

The --container-runtime flag needs to be set to "containerd".
CRI-O can be also supported later.

Code reading guide

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 26, 2021
@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 26, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: AkihiroSuda
To complete the pull request process, please assign medyagh after the PR has been reviewed.
You can assign the PR to them by writing /assign @medyagh in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@azhao155
Copy link
Contributor

Thanks @AkihiroSuda!!

Requirements:
- Install rootless Docker 20.10 or later, see https://rootlesscontaine.rs/getting-started/docker/
- Enable cgroup v2 delegation, see https://rootlesscontaine.rs/getting-started/common/cgroup2/

Usage: `minikube start --driver=docker --container-runtime=containerd`.
The `--container-runtime` flag needs to be set to "containerd".
CRI-O can be also supported later.

Closes issue 10836 ("add support for rootless Docker").

Support for rootless Podman (issue 8719) is not covered in this commit.

---

Code reading guide:
- `deploy/kicbase/Dockerfile`: updated to install fuse-overlayfs and containerd-fuse-overlayfs, which is used
  instead of `overlayfs` snapshotter

- `deploy/kicbase/entrypoint`: updated to verify cgroup v2 delegation.
  Mostly from https://github.com/kubernetes-sigs/kind/blob/8a83ee46b28a80ccd47a85e24294b3e149361947/images/base/files/usr/local/bin/entrypoint

- `cmd/minikube/cmd/start_flags.go`: updated to set `KubeletInUserNamespace` feature gate when rootless

- `pkg/drivers/kic/oci`: updated to use port forwarding, because rootless container IPs are not reachable from the host

- `pkg/minikube/cruntime`: updated to generate `/etc/containerd/config.toml` with rootless support.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@medyagh
Copy link
Member

medyagh commented Aug 30, 2021

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Aug 30, 2021
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12359) |
+----------------+----------+---------------------+
| minikube start | 54.2s    | 51.2s               |
| enable ingress | 32.7s    | 34.7s               |
+----------------+----------+---------------------+

Times for minikube start: 57.4s 49.6s 54.6s 55.1s 54.1s
Times for minikube (PR 12359) start: 50.1s 53.5s 51.9s 50.6s 50.1s

Times for minikube ingress: 33.0s 33.5s 33.0s 31.9s 32.4s
Times for minikube (PR 12359) ingress: 31.9s 33.4s 41.4s 34.3s 32.4s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12359) |
+----------------+----------+---------------------+
| minikube start | 22.9s    | 22.7s               |
| enable ingress | 32.7s    | 29.2s               |
+----------------+----------+---------------------+

Times for minikube ingress: 37.0s 34.5s 27.5s 28.0s 36.5s
Times for minikube (PR 12359) ingress: 27.5s 27.0s 36.0s 28.5s 27.0s

Times for minikube start: 23.4s 22.3s 21.9s 23.2s 23.7s
Times for minikube (PR 12359) start: 23.5s 22.1s 21.7s 22.7s 23.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12359) |
+----------------+----------+---------------------+
| minikube start | 40.8s    | 41.4s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube (PR 12359) start: 29.4s 44.2s 43.9s 45.4s 44.3s
Times for minikube start: 29.0s 43.3s 44.3s 43.6s 43.6s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_crio TestAddons/parallel/HelmTiller (gopogh) 1.26 (chart)
Docker_Linux_containerd TestAddons/parallel/HelmTiller (gopogh) 1.27 (chart)
Docker_Linux TestAddons/parallel/HelmTiller (gopogh) 1.27 (chart)
KVM_Linux TestAddons/parallel/HelmTiller (gopogh) 1.27 (chart)
KVM_Linux_crio TestAddons/parallel/HelmTiller (gopogh) 1.29 (chart)
Docker_Windows TestStartStop/group/newest-cni/serial/EnableAddonAfterStop (gopogh) 1.69 (chart)
Docker_Windows TestStartStop/group/newest-cni/serial/FirstStart (gopogh) 1.69 (chart)
Docker_Windows TestStartStop/group/newest-cni/serial/Stop (gopogh) 1.69 (chart)
KVM_Linux_containerd TestAddons/parallel/HelmTiller (gopogh) 1.89 (chart)
Docker_macOS TestAddons/parallel/HelmTiller (gopogh) 2.00 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/AddonExistsAfterStop (gopogh) 5.08 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 5.08 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 5.08 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/VerifyKubernetesImages (gopogh) 5.08 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/AddonExistsAfterStop (gopogh) 5.93 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh) 5.93 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/UserAppExistsAfterStop (gopogh) 5.93 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/VerifyKubernetesImages (gopogh) 5.93 (chart)
Docker_Linux_containerd_arm64 TestNetworkPlugins/group/bridge/NetCatPod (gopogh) 6.67 (chart)
Docker_Linux_crio TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 6.92 (chart)
Docker_Linux TestKubernetesUpgrade (gopogh) 7.59 (chart)
Docker_Windows TestKubernetesUpgrade (gopogh) 9.42 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 10.17 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/Pause (gopogh) 13.56 (chart)
Docker_Windows TestStartStop/group/newest-cni/serial/VerifyKubernetesImages (gopogh) 17.80 (chart)
Docker_Windows TestAddons/parallel/GCPAuth (gopogh) 18.42 (chart)
Docker_Windows TestStartStop/group/newest-cni/serial/SecondStart (gopogh) 18.49 (chart)
Docker_Windows TestFunctional/parallel/LoadImageFromFile (gopogh) 18.57 (chart)
Docker_Linux_crio TestStartStop/group/no-preload/serial/Stop (gopogh) 24.53 (chart)
Docker_Windows TestAddons/parallel/HelmTiller (gopogh) 31.58 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@medyagh
Copy link
Member

medyagh commented Aug 30, 2021

@TestAddons/parallel/HelmTiller seem to be failing only on this PR and not on master

 Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.16.12@sha256:6003775d503546087266eda39418d221f9afb5ccfe35f637c32a1161619a3f9c": rpc error: code = Unknown desc = Error reading manifest sha256:6003775d503546087266eda39418d221f9afb5ccfe35f637c32a1161619a3f9c in gcr.io/kubernetes-helm/tiller: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

but it might be a problem not ralted to this PR since I can not pull that image manually either

@medyagh
Copy link
Member

medyagh commented Aug 30, 2021

@TestAddons/parallel/HelmTiller seem to be failing only on this PR and not on master

 Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.16.12@sha256:6003775d503546087266eda39418d221f9afb5ccfe35f637c32a1161619a3f9c": rpc error: code = Unknown desc = Error reading manifest sha256:6003775d503546087266eda39418d221f9afb5ccfe35f637c32a1161619a3f9c in gcr.io/kubernetes-helm/tiller: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

but it might be a problem not ralted to this PR since I can not pull that image manually either

ok I confirm the failure is not related to this PR (see #12377)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add support for rootless Docker
6 participants