Storage provisioner CrashLoopBackOff: error getting server version dial tcp 10.96.0.1:443: i/o timeout

[punkerpunker]

What Happened?

Trying to start minikube cluster with --driver=none, getting storage-provisioner CrashLoopBackOff.
Running minikube behind the proxy.

Command to start:
minikube start --driver=none --kubernetes-version=1.20.8 --apiserver-ips 127.0.0.1 --apiserver-name localhost

😄  minikube v1.22.0 on Centos 8.4.2105
    ▪ MINIKUBE_HOME=/home/glebvazhenin
    ▪ KUBECONFIG=/home/glebvazhenin/.kube/config
🎉  minikube 1.25.1 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.25.1
✨  Using the none driver based on user configuration
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=40, Memory=128306MB, Disk=20030MB) ...
ℹ️  OS release is CentOS Linux 8
🌐  Found network options:
    ▪ HTTP_PROXY=http://proxy.mlan:3128
    ▪ HTTPS_PROXY=http://proxy.mlan:3128
    ▪ NO_PROXY=localhost, xcat-mn1, 127.0.0.1, repo-obs, repo-manual, repo-smt, .mlan, .ulan, .hlan, 10., 192.168., dockerio.badoo.com, .d3, .d4+, 10.10.211.99, 10.96.0.1, datascience10.mlan
    ▪ http_proxy=http://proxy.mlan:3128
    ▪ https_proxy=http://proxy.mlan:3128
    ▪ no_proxy=localhost, xcat-mn1, 127.0.0.1, repo-obs, repo-manual, repo-smt, .mlan, .ulan, .hlan, 10., 192.168., dockerio.badoo.com, .d3, .d4
🐳  Preparing Kubernetes v1.20.8 on Docker 19.03.13 ...
    ▪ env HTTP_PROXY=http://proxy.mlan:3128
    ▪ env HTTPS_PROXY=http://proxy.mlan:3128
    ▪ env NO_PROXY=localhost, xcat-mn1, 127.0.0.1, repo-obs, repo-manual, repo-smt, .mlan, .ulan, .hlan, 10., 192.168., dockerio.badoo.com, .d3, .d4+, 10.10.211.99, 10.96.0.1, datascience10.mlan
    ▪ env NO_PROXY=localhost, xcat-mn1, 127.0.0.1, repo-obs, repo-manual, repo-smt, .mlan, .ulan, .hlan, 10., 192.168., dockerio.badoo.com, .d3, .d4
    > kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubeadm: 37.40 MiB / 37.40 MiB [-----------] 100.00% 118.55 MiB p/s 500ms
    > kubelet: 108.85 MiB / 108.85 MiB [----------] 100.00% 108.09 MiB p/s 1.2s
    > kubectl: 38.36 MiB / 38.36 MiB [-------------] 100.00% 38.66 MiB p/s 1.2s
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner

❗  /home/glebvazhenin/.local/bin/kubectl is version 1.23.4, which may have incompatibilites with Kubernetes 1.20.8.
    ▪ Want kubectl v1.20.8? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Pods status:
kubectl get pods -A

kube-system   coredns-74ff55c5b-zpt5z                 1/1     Running            0          5m5s
kube-system   etcd-datascience10                      1/1     Running            0          5m18s
kube-system   kube-apiserver-datascience10            1/1     Running            0          5m18s
kube-system   kube-controller-manager-datascience10   1/1     Running            0          5m18s
kube-system   kube-proxy-l2fkg                        1/1     Running            0          5m5s
kube-system   kube-scheduler-datascience10            1/1     Running            0          5m18s
kube-system   storage-provisioner                     0/1     CrashLoopBackOff   4          5m19s

Storage-provisioner status:
kubectl -n kube-system logs storage-provisioner -f

F0218 23:04:03.105273       1 main.go:39] error getting server version: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

Tried both latest version of minikube (v 1.25.1) and the one suggested here. Also tried this solution, but doesn't seem to work for me. When trying to create any kind of PVC - it hangs with status Pending. With driver=docker everything works for me, but trying my best with driver=none since nvidia gpu support is needed.

I am stuck here, will appreciate any help. Thank you!

Attach the log file

log.txt

Operating System

Other

Driver

No response

[mgabeler-lee-6rs]

I saw another error in concert with this one, which I think may actually share a root cause: #13726

FWIW, the workaround for that issue (restart minikube) also cleared this issue for me

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[klaases]

Hi @punkerpunker, looks like this issue was resolved in #13726

If not, please feel free to re-open the issue by commenting with /reopen. This issue will be closed as additional information was unavailable and some time has passed.

Additional information that may be helpful:

• Whether the issue occurs with the latest minikube release

• The exact minikube start command line used

• Attach the full output of minikube logs, run minikube logs --file=logs.txt to create a log file

Thank you for sharing your experience!