kubelet: Exit with error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

[jiulongzaitian]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version): v0.24.1

• OS (e.g. from /etc/os-release):
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): none
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
• Install tools:
  curl
• Others:
  The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo " v0.24.1";
echo "OS:";

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

echo "";
echo "VM driver":  none
grep DriverName ~/.minikube/machines/minikube/config.json
echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:
when i exec

minikube start --vm-driver=none --extra-config=kubelet.cgroup-driver=systemd

Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
===================
WARNING: IT IS RECOMMENDED NOT TO RUN THE NONE DRIVER ON PERSONAL WORKSTATIONS
	The 'none' driver will run an insecure kubernetes apiserver as root that may leave the host vulnerable to CSRF attacks

Loading cached images from config file.

i can not get any node resource, such as:

kubectl get node
No resources found.

when i use minikube logs , i find:

localkube[14702]: kubelet: Exit with error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

--extra-config=kubelet.cgroup-driver=systemd don`t work

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Output of minikube logs (if applicable):

Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.401103   30646 cadvisor_linux.go:170] Failed to register cAdvisor on port 4194, retrying. Error: listen tcp 0.0.0.0:4194: bind: address already in use
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.401878   30646 container_manager_linux.go:252] container manager verified user specified cgroup-root exists: /
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.401895   30646 container_manager_linux.go:257] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s}
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.401959   30646 container_manager_linux.go:288] Creating device plugin handler: false
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.402033   30646 kubelet.go:273] Adding manifest file: /etc/kubernetes/manifests
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.402048   30646 kubelet.go:283] Watching apiserver
Jan 03 01:49:48 banwagong localkube[30646]: E0103 01:49:48.404259   30646 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": unable to create inotify: inotify_init: too many open files
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.407842   30646 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.407862   30646 kubelet.go:517] Hairpin mode set to "hairpin-veth"
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.410692   30646 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.414365   30646 docker_service.go:207] Docker cri networking managed by kubernetes.io/no-op
Jan 03 01:49:48 banwagong localkube[30646]: kubelet: Exit with error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.619260   30646 feature_gate.go:156] feature gates: map[]
Jan 03 01:49:48 banwagong localkube[30646]: E0103 01:49:48.619305   30646 server.go:210] unable to register configz: register config "kubeletconfig" twice
Jan 03 01:49:48 banwagong localkube[30646]: E0103 01:49:48.619313   30646 server.go:268] unable to register KubeletConfiguration with configz, error: register config "kubeletconfig" twice
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.619323   30646 server.go:276] --require-kubeconfig is deprecated. Set --kubeconfig without using --require-kubeconfig.
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.626314   30646 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.626338   30646 client.go:95] Start docker client with request timeout=2m0s
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.627429   30646 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.631805   30646 server.go:289] --cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.635147   30646 manager.go:149] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct"
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.640010   30646 manager.go:157] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.640165   30646 manager.go:166] unable to connect to CRI-O api service: Get http://%2Fvar%2Frun%2Fcrio.sock/info: dial unix /var/run/crio.sock: connect: no such file or directory
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.640590   30646 fs.go:139] Filesystem UUIDs: map[3437f1a0-f850-4f1b-8a7c-819c5f6a29e4:/dev/sda2 ad1361f7-4ab4-4252-ba00-5d4e5d8590fb:/dev/sda1]
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.640610   30646 fs.go:140] Filesystem partitions: map[tmpfs:{mountpoint:/dev/shm major:0 minor:18 fsType:tmpfs blockSize:0} /dev/sda2:{mountpoint:/ major:8 minor:2 fsType:ext4 blockSize:0} /dev/sda1:{mountpoint:/boot major:8 minor:1 fsType:ext3 blockSize:0}]
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.641592   30646 manager.go:216] Machine: {NumCores:2 CpuFrequency:2399994 MemoryCapacity:1057562624 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:fec6fa7192834fd7b32553bd0105eedb SystemUUID:FEC6FA71-9283-4FD7-B325-53BD0105EEDB BootID:51aed273-d999-43dd-bc48-7a36d02cab38 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:18 Capacity:528781312 Type:vfs Inodes:129097 HasInodes:true} {Device:/dev/sda2 DeviceMajor:8 DeviceMinor:2 Capacity:22770499584 Type:vfs Inodes:1411680 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:397803520 Type:vfs Inodes:102400 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:107374182400 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:23622320128 Scheduler:deadline}] NetworkDevices:[{Name:eth0 MacAddress:aa:aa:00:0b:f7:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:1092198400 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]} {Id:1 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.642574   30646 manager.go:222] Version: {KernelVersion:3.10.0-514.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.12.6 DockerAPIVersion:1.24 CadvisorVersion: CadvisorRevision:}
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.643398   30646 cadvisor_linux.go:170] Failed to register cAdvisor on port 4194, retrying. Error: listen tcp 0.0.0.0:4194: bind: address already in use
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.644302   30646 container_manager_linux.go:252] container manager verified user specified cgroup-root exists: /
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.644315   30646 container_manager_linux.go:257] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s}
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.644373   30646 container_manager_linux.go:288] Creating device plugin handler: false
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.644461   30646 kubelet.go:273] Adding manifest file: /etc/kubernetes/manifests
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.644482   30646 kubelet.go:283] Watching apiserver
Jan 03 01:49:48 banwagong localkube[30646]: E0103 01:49:48.644861   30646 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": unable to create inotify: inotify_init: too many open files
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.650264   30646 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.650285   30646 kubelet.go:517] Hairpin mode set to "hairpin-veth"
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.652586   30646 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.656586   30646 docker_service.go:207] Docker cri networking managed by kubernetes.io/no-op
Jan 03 01:49:48 banwagong localkube[30646]: kubelet: Exit with error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.860878   30646 feature_gate.go:156] feature gates: map[]
Jan 03 01:49:48 banwagong localkube[30646]: E0103 01:49:48.860934   30646 server.go:210] unable to register configz: register config "kubeletconfig" twice
Jan 03 01:49:48 banwagong localkube[30646]: E0103 01:49:48.860944   30646 server.go:268] unable to register KubeletConfiguration with configz, error: register config "kubeletconfig" twice
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.860954   30646 server.go:276] --require-kubeconfig is deprecated. Set --kubeconfig without using --require-kubeconfig.
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.873127   30646 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.873153   30646 client.go:95] Start docker client with request timeout=2m0s
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.874131   30646 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
Jan 03 01:49:48 banwagong localkube[30646]: W0103 01:49:48.880358   30646 server.go:289] --cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly
Jan 03 01:49:48 banwagong localkube[30646]: I0103 01:49:48.883819   30646 manager.go:149] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct"

Anything else do we need to know:

[g00nix]

It fails in my case with the same error:

failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

I am running minikube start --vm-driver=none on Fedora 29.

[tnguyen14]

I am seeing the same error, running on RHEL 7.6. Does anyone now what the issue might be?