Minikube start fails on 0.26.0, 0.26.1, 0.27.0, 0.28.0 (Win 10 + VirtualBox)

[javajon]

BUG REPORT

Environment:

Minikube version: v0.27.0

• OS: Microsoft Windows [Version 10.0.16299.431]
• VM Driver: virtualbox
• ISO version : "Boot2DockerURL": "file://C:/Users/User/.minikube/cache/iso/minikube-v0.26.0.iso"
• Install tools: minikube-windows-amd64.exe renamed to minikube.exe
• Others: VirtualBox Version 5.2.12 r122591 (Qt5.6.2)

What happened:

$ minikube start
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0517 00:24:48.732109   17904 start.go:276] Error starting cluster:  kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
 output: [init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[certificates] Using the existing ca certificate and key.
                                                                                                    [WARNING Swap]: running with swap on is not supported. Please disable swap
Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.
                                                          [certificates] Using the existing apiserver certificate and key.
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [minikube-sonarqube] and IPs [192.168.99.100]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/var/lib/localkube/certs/"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 114.505841 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node minikube-sonarqube as master by adding a label and a taint
                                                          : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1

$ minikube ip
192.168.99.100

Minikube status does report running:

minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

However, pods remain forever pending

$ kubectl get pods -n kube-system
NAME                                    READY     STATUS    RESTARTS   AGE
heapster-dn52t                          0/1       Pending   0          13m
influxdb-grafana-4pblq                  0/2       Pending   0          13m
kubernetes-dashboard-5498ccf677-g8hsg   0/1       Pending   0          13m
registry-npd79                          0/1       Pending   0          13m
storage-provisioner                     0/1       Pending   0          13m

What you expected to happen:

No errors when starting
Pods should be starting
minikube dashboard should function

How to reproduce it (as minimally and precisely as possible):

minikube start on above system.

Output of minikube logs (if applicable):

Possible notable excerpts from: VBoxHardening.log

31a8.2b90: NtOpenDirectoryObject failed on \Driver: 0xc0000022
...
4398.a98: supR3HardenedWinReSpawn: NtSetInformationThread/ThreadHideFromDebugger failed: 0xc0000022 (harmless)
...
2aa0.2fac: supR3HardenedMonitor_LdrLoadDll: returns rcNt=0x0 hMod=00007ff80c270000 'C:\WINDOWS\System32\winmm.dll'
2aa0.2eec: '\Device\HarddiskVolume4\Windows\System32\tzres.dll' has no imports
2aa0.2eec: supHardenedWinVerifyImageByHandle: -> 22900 (\Device\HarddiskVolume4\Windows\System32\tzres.dll)
2aa0.2eec: supR3HardenedWinVerifyCacheInsert: \Device\HarddiskVolume4\Windows\System32\tzres.dll
2aa0.2eec: supR3HardenedMonitor_NtCreateSection: NtMapViewOfSection failed on 0000000000000e48 (hFile=0000000000000e84) with 0xc0000022 -> STATUS_TRUST_FAILURE
2aa0.2eec: supR3HardenedScreenImage/NtCreateSection: cache hit (Unknown Status 22900 (0x5974)) on \Device\HarddiskVolume4\Windows\System32\tzres.dll [avoiding WinVerifyTrust]
2aa0.2eec: supR3HardenedMonitor_NtCreateSection: NtMapViewOfSection failed on 0000000000000e84 (hFile=0000000000000e48) with 0xc0000022 -> STATUS_TRUST_FAILURE

Anything else do we need to know:

Something went wrong after version 0.25.2

Five other colleagues I work with have also reported similar problems on Windows 10 with VirtualBox. They have all reverted back to 0.25.2. I was hoping 0.27.0 would resolve the issue.

Tried minikube delete and a restart. Deleted all of VirtualBox and .kube .minikube and .VirtualBox files. Fresh install of VirtualBox, and minikube and still same problem on versions 0.26.0, 0.26.1 and 0.27.0

Workaround: Revert minikube.exe to version 0.25.2 and all returns to normal working order.

[javajon]

Perhaps related: #2804 #2786

[javajon]

The root of this problem appears to be related to the minikube profile feature and its current breakage is covered in #2717

If you notice above the log reports a node name of "minikube-sonarqube" instead of the default "minikube". This is because I had been previously working with a profile name set with minikube profile minikube-sonarqube. This profile setting is sticky. When switching from Minikube version 0.25.2 to a newer version this profile was still set (as confirmed by minikube config view) and calling minikube start, even without the -p switch will assume this previous profile and node name.

If I switch the profile to the default minikube profile minikube then with 0.27.0 perform a minikube start everything appears to function normal and healthy.

Since version 0.26.0, 0.26.1 and now 0.27.0 there is a definite problem with using profiles and that is confirmed with #2717. To replicate the error:

minikube profile test
minkube start

and you will see:

[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node test as master by adding a label and a taint
                                                          : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1

Workaround: The above error does not occur with version 0.25.2 as profiles appear to work with that previous release.

Perhaps this can be closed as supportive information for #2717?

(An enhancement could be to change the start message from "Starting local Kubernetes v1.10.0 cluster..." to "Starting local Kubernetes v1.10.0 cluster with profile [profile name]...")

[Robot684]

@javajon same issue with me,This problem exists in v0.26.x and later versions。

[playniuniu]

Same issue on mac osx 10.13.4

only with minikube start --vm-driver=hyperkit --bootstrapper=localkube works

[javajon]

Minikube version 0.28.0 produces same error:

$ minikube version
minikube version: v0.28.0
$ minikube start --kubernetes-version v1.10.0 --cpus 4 --memory 8000 --disk-size 80g --profile "openfaas"
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0613 21:32:59.715055   10924 start.go:294] Error starting cluster:  kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1
$ minikube profile openfaas
minikube profile was successfully set to openfaas
$ kubectl get pods -n kube-system
NAME                                        READY     STATUS    RESTARTS   AGE
default-http-backend-59868b7dd6-8v47n       0/1       Pending   0          16m
heapster-76djh                              0/1       Pending   0          16m
influxdb-grafana-zlcxm                      0/2       Pending   0          16m
kubernetes-dashboard-5498ccf677-6vp4b       0/1       Pending   0          16m
nginx-ingress-controller-67956bf89d-fgmgp   0/1       Pending   0          16m
registry-7mssp                              0/1       Pending   0          16m
storage-provisioner                         0/1       Pending   0          16m

[javajon]

As found also in #2574 the use of deprecated --bootstrapper localkube appears to avoid the problem.

[kairen]

Hi @javajon, I just to make sure that run Minikube v0.28.0 on Windows 10, I also tried to use Hyper-V and Virtualbox(5.2.12) driver to start VM, but it can be performed successfully.

Hyper-V snapshots:

Virtualbox snapshots:

Could you remove the config dir and try again?

$ rm -r $HOME/.minikube
$ minikube.exe start

I use Git bash to run this commands.

[javajon]

@kairen To be clearer I simplified this defect into a new report #2952 with minimal and reproducible steps. Including your suggestion of adding steps to delete ~/.kube and ~/.minikube.

This issue is the usage of profiles.

[quan01994]

I have same error .
it's my minikube version

it's my VBoxHardening.log

[tstromberg]

Dupe of #2574