Fix delete command for paused kic driver with containerd/crio runtime #11504

ilya-zuyev · 2021-05-25T01:56:59Z

This PR fixes an issue when delete operation hangs forever for paused minikube when (container-runtime=containerd or container-runtime=crio) and driver=docker
Fixes #11332

Before:

➜  minikube git:(master) m start --container-runtime=containerd
😄  minikube v1.20.0 on Darwin 11.3.1
✨  Automatically selected the docker driver. Other choices: virtualbox, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
📦  Preparing Kubernetes v1.20.2 on containerd 1.4.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default



➜  minikube git:(master) m pause
⏸️  Pausing node minikube ... 
⏯️  Paused 1 containers in: kube-system, kubernetes-dashboard, storage-gluster, istio-operator


➜  minikube git:(master) m delete 
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
[hangs]
^C
➜  minikube git:(master)

After:

➜  minikube git:(ilyaz/fix_delete_paused) ✗ m start --container-runtime=containerd
😄  minikube v1.20.0 on Darwin 11.3.1
✨  Automatically selected the docker driver. Other choices: virtualbox, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
📦  Preparing Kubernetes v1.20.2 on containerd 1.4.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default


➜  minikube git:(ilyaz/fix_delete_paused) ✗ m pause
⏸️  Pausing node minikube ... 
⏯️  Paused 1 containers in: kube-system, kubernetes-dashboard, storage-gluster, istio-operator


➜  minikube git:(ilyaz/fix_delete_paused) ✗ m delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /Users/izuyev/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

k8s-ci-robot · 2021-05-25T01:57:00Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

ilya-zuyev · 2021-05-25T01:57:08Z

/ok-to-test

k8s-ci-robot · 2021-05-25T01:57:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ilya-zuyev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ilya-zuyev]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

minikube-pr-bot · 2021-05-25T02:43:50Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 47.3s    | 46.7s               |
| enable ingress | 36.3s    | 38.4s               |
+----------------+----------+---------------------+

Times for minikube ingress: 34.2s 34.7s 42.7s 35.2s 34.8s
Times for minikube (PR 11504) ingress: 42.2s 34.7s 42.3s 37.7s 35.3s

Times for minikube start: 46.8s 46.8s 47.5s 49.8s 45.7s
Times for minikube (PR 11504) start: 46.4s 46.7s 46.6s 46.4s 47.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 21.7s    | 21.7s               |
| enable ingress | 35.9s    | 33.9s               |
+----------------+----------+---------------------+

Times for minikube start: 22.8s 21.1s 22.1s 21.2s 21.5s
Times for minikube (PR 11504) start: 23.1s 20.9s 21.9s 21.2s 21.4s

Times for minikube ingress: 41.0s 38.5s 35.5s 31.4s 33.0s
Times for minikube (PR 11504) ingress: 34.0s 38.0s 32.0s 33.5s 32.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 42.8s    | 45.1s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 32.3s 47.1s 44.0s 46.9s 43.7s
Times for minikube (PR 11504) start: 47.2s 43.4s 43.4s 48.7s 43.0s

minikube-pr-bot · 2021-05-25T06:37:12Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 48.0s    | 46.8s               |
| enable ingress | 36.1s    | 35.2s               |
+----------------+----------+---------------------+

Times for minikube ingress: 33.7s 40.7s 33.8s 38.1s 34.2s
Times for minikube (PR 11504) ingress: 35.2s 33.7s 34.2s 37.7s 35.2s

Times for minikube start: 52.4s 48.1s 47.0s 45.9s 46.6s
Times for minikube (PR 11504) start: 48.6s 47.1s 45.4s 46.9s 45.8s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 21.9s    | 21.5s               |
| enable ingress | 30.2s    | 31.8s               |
+----------------+----------+---------------------+

Times for minikube start: 23.2s 21.7s 22.0s 21.4s 21.3s
Times for minikube (PR 11504) start: 21.4s 21.3s 21.7s 21.1s 22.1s

Times for minikube ingress: 29.0s 29.5s 30.0s 29.5s 33.0s
Times for minikube (PR 11504) ingress: 37.0s 29.5s 33.0s 30.0s 29.5s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 41.4s    | 45.3s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 30.7s 46.9s 43.2s 43.3s 43.0s
Times for minikube (PR 11504) start: 43.3s 47.3s 46.9s 42.2s 46.9s

minikube-pr-bot · 2021-05-26T02:07:54Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 47.4s    | 48.9s               |
| enable ingress | 34.4s    | 34.6s               |
+----------------+----------+---------------------+

Times for minikube (PR 11504) start: 50.6s 47.8s 49.5s 49.9s 47.0s
Times for minikube start: 47.7s 47.7s 49.5s 46.0s 46.3s

Times for minikube ingress: 33.7s 34.2s 33.7s 34.3s 35.8s
Times for minikube (PR 11504) ingress: 35.3s 35.3s 33.2s 34.8s 34.2s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 22.6s    | 21.9s               |
| enable ingress | 30.8s    | 29.5s               |
+----------------+----------+---------------------+

Times for minikube start: 23.0s 22.0s 22.3s 23.1s 22.6s
Times for minikube (PR 11504) start: 22.1s 21.3s 22.2s 21.8s 21.9s

Times for minikube ingress: 36.5s 28.5s 29.0s 31.0s 29.0s
Times for minikube (PR 11504) ingress: 29.5s 28.5s 29.0s 32.0s 28.5s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 42.9s    | 45.8s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 32.4s 43.5s 47.2s 43.4s 48.2s
Times for minikube (PR 11504) start: 46.8s 46.8s 43.3s 48.0s 44.1s

minikube-pr-bot · 2021-05-26T21:21:58Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 49.5s    | 46.9s               |
| enable ingress | 36.0s    | 35.2s               |
+----------------+----------+---------------------+

Times for minikube start: 51.8s 47.3s 48.7s 47.7s 52.2s
Times for minikube (PR 11504) start: 47.8s 46.8s 46.3s 47.6s 46.3s

Times for minikube (PR 11504) ingress: 33.8s 36.3s 35.3s 35.9s 34.8s
Times for minikube ingress: 33.9s 41.8s 33.8s 36.3s 34.3s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 22.4s    | 21.9s               |
| enable ingress | 32.5s    | 32.9s               |
+----------------+----------+---------------------+

Times for minikube start: 22.6s 21.3s 23.0s 22.0s 23.1s
Times for minikube (PR 11504) start: 22.4s 21.9s 22.4s 20.8s 22.0s

Times for minikube ingress: 29.5s 33.5s 33.0s 33.5s 33.0s
Times for minikube (PR 11504) ingress: 36.5s 28.0s 35.6s 33.0s 31.5s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 44.1s    | 43.6s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 44.4s 45.1s 43.6s 43.6s 43.7s
Times for minikube (PR 11504) start: 43.9s 42.9s 43.4s 43.9s 43.7s

minikube-pr-bot · 2021-05-26T21:52:18Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 53.5s    | 56.1s               |
| enable ingress | 35.6s    | 39.0s               |
+----------------+----------+---------------------+

Times for minikube start: 53.0s 56.4s 51.2s 53.7s 53.2s
Times for minikube (PR 11504) start: 54.7s 53.6s 58.0s 59.4s 54.6s

Times for minikube ingress: 35.3s 33.8s 37.1s 37.0s 34.9s
Times for minikube (PR 11504) ingress: 40.9s 33.9s 43.4s 43.1s 33.8s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 23.3s    | 23.0s               |
| enable ingress | 29.1s    | 29.2s               |
+----------------+----------+---------------------+

Times for minikube start: 23.3s 24.2s 23.7s 23.3s 21.9s
Times for minikube (PR 11504) start: 23.5s 23.0s 24.6s 21.8s 21.9s

Times for minikube ingress: 27.5s 26.6s 32.5s 28.1s 31.0s
Times for minikube (PR 11504) ingress: 30.5s 29.0s 25.5s 30.0s 31.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 46.9s    | 45.9s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 49.3s 45.0s 48.6s 43.3s 48.6s
Times for minikube (PR 11504) start: 43.7s 44.2s 43.7s 48.0s 49.7s

ilya-zuyev · 2021-05-26T23:12:56Z

http://storage.googleapis.com/minikube-builds/logs/11504/773500b/KVM_Linux_containerd.html#pass_TestNetworkPlugins%2fgroup%2fauto%2fStart
Test fixed

medyagh · 2021-05-26T23:53:45Z

cmd/minikube/cmd/delete.go

+func unpauseIfNeeded(profile *config.Profile) error {
+	// there is a known issue with removing kicbase container with paused containerd containers inside
+	// unpause it before we delete it
+	if profile.Config.KubernetesConfig.ContainerRuntime != "containerd" {


crio also has this problem., we can check if Not Docker

cmd/minikube/cmd/delete.go

minikube-pr-bot · 2021-05-27T00:03:13Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 46.8s    | 47.9s               |
| enable ingress | 36.9s    | 38.1s               |
+----------------+----------+---------------------+

Times for minikube start: 47.9s 46.3s 47.4s 47.0s 45.4s
Times for minikube (PR 11504) start: 46.1s 50.3s 47.1s 48.4s 47.6s

Times for minikube ingress: 36.7s 42.2s 35.2s 34.7s 35.7s
Times for minikube (PR 11504) ingress: 35.2s 34.2s 42.7s 42.2s 36.3s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 21.7s    | 22.1s               |
| enable ingress | 36.3s    | 35.3s               |
+----------------+----------+---------------------+

Times for minikube start: 21.0s 21.8s 21.9s 21.3s 22.4s
Times for minikube (PR 11504) start: 21.3s 22.1s 21.5s 22.4s 23.0s

Times for minikube (PR 11504) ingress: 34.0s 34.5s 37.0s 37.5s 33.5s
Times for minikube ingress: 38.5s 36.5s 32.5s 34.5s 39.5s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 44.5s    | 44.1s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 43.7s 44.1s 47.5s 43.2s 44.0s
Times for minikube (PR 11504) start: 43.5s 43.2s 47.0s 43.7s 42.9s

minikube-pr-bot · 2021-05-27T06:12:23Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 49.9s    | 48.7s               |
| enable ingress | 35.2s    | 35.2s               |
+----------------+----------+---------------------+

Times for minikube (PR 11504) start: 49.7s 46.6s 48.9s 47.0s 51.5s
Times for minikube start: 52.2s 47.9s 50.7s 48.0s 50.4s

Times for minikube (PR 11504) ingress: 34.8s 34.2s 34.2s 37.8s 35.2s
Times for minikube ingress: 42.3s 33.8s 33.2s 33.8s 33.2s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 22.8s    | 22.5s               |
| enable ingress | 30.7s    | 30.5s               |
+----------------+----------+---------------------+

Times for minikube start: 23.0s 22.5s 23.2s 22.9s 22.5s
Times for minikube (PR 11504) start: 22.1s 22.9s 22.5s 22.9s 22.0s

Times for minikube ingress: 32.0s 32.0s 29.0s 29.0s 31.5s
Times for minikube (PR 11504) ingress: 28.5s 28.5s 32.5s 31.0s 32.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 11504) |
+----------------+----------+---------------------+
| minikube start | 42.0s    | 41.7s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 31.8s 44.3s 47.2s 43.2s 43.7s
Times for minikube (PR 11504) start: 31.1s 43.5s 43.7s 43.5s 46.9s

unpause before delete

4265887

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 25, 2021

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 25, 2021

k8s-ci-robot requested review from prasadkatti and tstromberg May 25, 2021 01:57

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 25, 2021

update logs

017bd42

trigger bulid

4567aaa

ilya-zuyev added 2 commits May 26, 2021 11:02

remove log files

9bd097f

remove log files

86c654e

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 26, 2021

ilya-zuyev added 2 commits May 26, 2021 12:49

do not unapuse for docker/crio

773500b

simplify the code

2a7f0dc

ilya-zuyev marked this pull request as ready for review May 26, 2021 20:59

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 26, 2021

fix quoting

2628f8b

medyagh requested changes May 26, 2021

View reviewed changes

medyagh reviewed May 26, 2021

View reviewed changes

cmd/minikube/cmd/delete.go Show resolved Hide resolved

Merge branch 'master' into ilyaz/fix_delete_paused

2244e26

apply to crio as well

1b562da

ilya-zuyev changed the title ~~Fix delete command for paused kic driver with containerd runtime~~ Fix delete command for paused kic driver with containerd/crio runtime May 27, 2021

trigger build to update github status

eff0bc7

sharifelgamal merged commit 3430150 into kubernetes:master Jun 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix delete command for paused kic driver with containerd/crio runtime #11504

Fix delete command for paused kic driver with containerd/crio runtime #11504

ilya-zuyev commented May 25, 2021 •

edited

Loading

k8s-ci-robot commented May 25, 2021

ilya-zuyev commented May 25, 2021

k8s-ci-robot commented May 25, 2021

minikube-pr-bot commented May 25, 2021

minikube-pr-bot commented May 25, 2021

minikube-pr-bot commented May 26, 2021

minikube-pr-bot commented May 26, 2021

minikube-pr-bot commented May 26, 2021

ilya-zuyev commented May 26, 2021 •

edited

Loading

medyagh May 26, 2021

ilya-zuyev May 27, 2021

minikube-pr-bot commented May 27, 2021

minikube-pr-bot commented May 27, 2021

Fix delete command for paused kic driver with containerd/crio runtime #11504

Fix delete command for paused kic driver with containerd/crio runtime #11504

Conversation

ilya-zuyev commented May 25, 2021 • edited Loading

k8s-ci-robot commented May 25, 2021

ilya-zuyev commented May 25, 2021

k8s-ci-robot commented May 25, 2021

minikube-pr-bot commented May 25, 2021

minikube-pr-bot commented May 25, 2021

minikube-pr-bot commented May 26, 2021

minikube-pr-bot commented May 26, 2021

minikube-pr-bot commented May 26, 2021

ilya-zuyev commented May 26, 2021 • edited Loading

medyagh May 26, 2021

Choose a reason for hiding this comment

ilya-zuyev May 27, 2021

Choose a reason for hiding this comment

minikube-pr-bot commented May 27, 2021

minikube-pr-bot commented May 27, 2021

ilya-zuyev commented May 25, 2021 •

edited

Loading

ilya-zuyev commented May 26, 2021 •

edited

Loading