Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman driver does not handle filesystems using dev mapper #2113

Closed
papanito opened this issue Mar 9, 2021 · 6 comments · Fixed by #2122
Closed

podman driver does not handle filesystems using dev mapper #2113

papanito opened this issue Mar 9, 2021 · 6 comments · Fixed by #2122
Assignees
@aojea
Labels
area/provider/podman Issues or PRs related to podman kind/bug Categorizes issue or PR as related to a bug.

Comments

@papanito
Copy link

papanito commented Mar 9, 2021

Basically the same issue as #1999 but podman instead docker

What happened:

I run kind create cluster on Manjaro linux using podman, but the control plane failed to start.

What you expected to happen:

A kind k8s cluster to be created successfully

How to reproduce it (as minimally and precisely as possible):

  • Install podman (instead of docker)
  • Run kind create cluster on Majaro

Anything else we need to know?:

  • I have firewalld enabled and backend is FirewallBackend=nftables
  • Filesystem is btrfs and podman is configured accordingly
    /etc/containers/storage.conf 
    [storage]

# Default Storage Driver
driver = "btrfs"
mountopt = "nodev"
...

Environment:

  • kind version: (use kind version): kind v0.10.0 go1.15.7 linux/amd64
  • Kubernetes version: (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"archive", BuildDate:"2021-01-13T22:07:54Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"}
  • Docker version: (use sudo podman info): 
    host:
arch: amd64
buildahVersion: 1.19.4
cgroupManager: systemd
cgroupVersion: v2
conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.26-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 0e155c83aa739ef0a0540ec9f9d265f57f68038b'
cpus: 8
distribution:
    distribution: manjaro
    version: unknown
eventLogger: journald
hostname: clawfinger
idMappings:
    gidmap: null
    uidmap: null
kernel: 5.10.18-1-MANJARO
linkmode: dynamic
memFree: 779718656
memTotal: 16644816896
ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 0.18-1
    path: /usr/bin/crun
    version: |-
    crun version 0.18
    commit: 808420efe3dc2b44d6db9f1a3fac8361dde42a95
    spec: 1.0.0
    +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
remoteSocket:
    path: /run/podman/podman.sock
security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
slirp4netns:
    executable: ""
    package: ""
    version: ""
swapFree: 19034157056
swapTotal: 19293794304
uptime: 186h 52m 52.68s (Approximately 7.75 days)
registries:
search:
- docker.io
- registry.fedoraproject.org
- quay.io
- registry.access.redhat.com
- registry.centos.org
store:
configFile: /etc/containers/storage.conf
containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
graphDriverName: overlay
graphOptions:
    overlay.mountopt: nodev
graphRoot: /var/lib/containers/storage
graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
imageStore:
    number: 2
runRoot: /var/run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 3.0.0
Built: 1613921386
BuiltTime: Sun Feb 21 16:29:46 2021
GitCommit: c640670e85c4aaaff92741691d6a854a90229d8d
GoVersion: go1.16
OsArch: linux/amd64
Version: 3.0.1
  • OS (e.g. from /etc/os-release): 
    NAME="Manjaro Linux"
ID=manjaro
ID_LIKE=arch
BUILD_ID=rolling
PRETTY_NAME="Manjaro Linux"
ANSI_COLOR="32;1;24;144;200"
HOME_URL="https://manjaro.org/"
DOCUMENTATION_URL="https://wiki.manjaro.org/"
SUPPORT_URL="https://manjaro.org/"
BUG_REPORT_URL="https://bugs.manjaro.org/"
LOGO=manjarolinux

Details

sudo kind create cluster
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✗ Starting control-plane 🕹️ 
ERROR: failed to create cluster: failed to init node with kubeadm: command "podman exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0309 09:04:57.415942     129 initconfiguration.go:201] loading configuration from "/kind/kubeadm.conf"
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta2, Kind=JoinConfiguration
[init] Using Kubernetes version: v1.20.2
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0309 09:04:57.504023     129 certs.go:110] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0309 09:04:57.651104     129 certs.go:474] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kind-control-plane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.89.0.6 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0309 09:04:58.003404     129 certs.go:110] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0309 09:04:58.610077     129 certs.go:474] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0309 09:04:58.719639     129 certs.go:110] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0309 09:04:58.906658     129 certs.go:474] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kind-control-plane localhost] and IPs [10.89.0.6 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kind-control-plane localhost] and IPs [10.89.0.6 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0309 09:04:59.688660     129 certs.go:76] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0309 09:04:59.912575     129 kubeconfig.go:101] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0309 09:05:00.050283     129 kubeconfig.go:101] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0309 09:05:00.256262     129 kubeconfig.go:101] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0309 09:05:00.538438     129 kubeconfig.go:101] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I0309 09:05:00.637596     129 kubelet.go:63] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0309 09:05:00.780926     129 manifests.go:96] [control-plane] getting StaticPodSpecs
I0309 09:05:00.781474     129 certs.go:474] validating certificate period for CA certificate
I0309 09:05:00.781596     129 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0309 09:05:00.781609     129 manifests.go:109] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I0309 09:05:00.781617     129 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0309 09:05:00.781624     129 manifests.go:109] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I0309 09:05:00.781633     129 manifests.go:109] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0309 09:05:00.790010     129 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
I0309 09:05:00.790044     129 manifests.go:96] [control-plane] getting StaticPodSpecs
I0309 09:05:00.790422     129 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0309 09:05:00.790438     129 manifests.go:109] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I0309 09:05:00.790442     129 manifests.go:109] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0309 09:05:00.790453     129 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0309 09:05:00.790457     129 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0309 09:05:00.790460     129 manifests.go:109] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I0309 09:05:00.790467     129 manifests.go:109] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0309 09:05:00.791431     129 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
I0309 09:05:00.791447     129 manifests.go:96] [control-plane] getting StaticPodSpecs
I0309 09:05:00.791692     129 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0309 09:05:00.792144     129 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
I0309 09:05:00.792799     129 local.go:74] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0309 09:05:00.792809     129 waitcontrolplane.go:87] [wait-control-plane] Waiting for the API server to be healthy
I0309 09:05:00.793460     129 loader.go:379] Config loaded from file:  /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0309 09:05:00.794506     129 round_trippers.go:445] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I0309 09:05:01.295140     129 round_trippers.go:445] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I0309 09:05:01.795652     129 round_trippers.go:445] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I0309 09:05:02.295555     129 round_trippers.go:445] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I0309 09:05:02.795619     129 round_trippers.go:445] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I0309 09:05:03.295850     129 round_trippers.go:445] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 
...
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

	Unfortunately, an error has occurred:
		timed out waiting for the condition

	This error is likely caused by:
		- The kubelet is not running
		- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

	If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
		- 'systemctl status kubelet'
		- 'journalctl -xeu kubelet'

	Additionally, a control plane component may have crashed or exited when started by the container runtime.
	To troubleshoot, list all containers using your preferred container runtimes CLI.

	Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
		- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
		Once you have found the failing container, you can inspect its logs with:
		- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'

couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:114
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:151
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:850
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
k8s.io/kubernetes/cmd/kubeadm/app.Run
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
	_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:204
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1374
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:151
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:850
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
k8s.io/kubernetes/cmd/kubeadm/app.Run
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
	_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:204
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1374
@papanito papanito added the kind/bug Categorizes issue or PR as related to a bug. label Mar 9, 2021
@BenTheElder
Copy link
Member

We need the cluster logs to debug this further, as discussed in the previous issue.

This setup is quite unstable for running kubernetes:

  • podman is less stable (the project), podman support in kind is still experimental
  • no one is testing btrfs and kubernetes or containerd regularly in any tool
  • cgroupsv2 is still quite new in the container space, and poorly tested. We have CI for this ourselves, but we only test on fedora, with a less bleeding edge kernel

@BenTheElder
Copy link
Member

Found your logs in #1999 (comment)

Mar 09 09:12:30 kind-control-plane kubelet[777]: F0309 09:12:30.541441 777 kubelet.go:1350] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/lib/kubelet": could not find device with major: 0, minor: 27 in cached partitions map

This happens when /dev/mapper is not mounted. We don't have detection for btrfs storage driver in podman currently cc @aojea

The workaround config in #1999 (comment) will solve that particular issue in the meantim.

@BenTheElder BenTheElder mentioned this issue Mar 9, 2021
Closed
@BenTheElder BenTheElder changed the title Manjaro Linux, podman and btrfs: failed to create cluster: failed to init node with kubeadm #1999 podman driver does not handle filesystems using dev mapper Mar 9, 2021
@BenTheElder BenTheElder added the area/provider/podman Issues or PRs related to podman label Mar 9, 2021
@papanito
Copy link
Author

papanito commented Mar 10, 2021 •
edited
Loading

Thanks @BenTheElder sorry attached the logs to the wrong issue 🤦 For completeness I've attached them here
kind-logs.zip

@papanito
Copy link
Author

papanito commented Mar 10, 2021 •
edited
Loading

cgroupsv2 is still quite new in the container space, and poorly tested. We have CI for this ourselves, but we only test on fedora, with a less bleeding edge kernel

If you think there is some help needed in testing, glad to assist.

@BenTheElder
Copy link
Member

I'm not sure if there's sufficient interest in upstream Kubernetes to maintain coverage for btrfs etc. I don't think even even OpenShift uses btrfs for containers.

Does #1999 (comment) solve your issue? Or are there further issues even with /dev/mapper mounted?

@aojea aojea mentioned this issue Mar 11, 2021
Merged
@aojea
Copy link
Contributor

aojea commented Mar 11, 2021

If you think there is some help needed in testing, glad to assist.

@papanito if you can test if this fixes the problem :)
#2122

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aojea aojea

Labels
area/provider/podman Issues or PRs related to podman kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

podman: mount dev mapper for btrfs and zfs storage aojea/kind
3 participants
@BenTheElder @aojea @papanito