rkt 1.0 not working when /var/lib/rkt is on btrfs

[trusch]

I tried running a container under debian stable and failed.
It seems like it fails copying the requiered systemd utils (especially journald).

I'm working on a freshly installed debian jessie. The only special things about my setup are a kernel from the backports repo, and that my root filesystem is a btrfs.

When executing a container the following output occurs and the container freezes and can only be stopped by killing it the hard way (kill -9)

➜  rkt-v1.0.0  sudo ./rkt --debug run quay.io/coreos/alpine-sh
image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.0.0
image: using image from local store for image name quay.io/coreos/alpine-sh
stage0: Preparing stage1
stage0: Writing image manifest
stage0: Loading image sha512-a2fb8f390702d3d9b00d2ebd93e7dd1c241ee7bc752b362635c57b29aabb1a48
stage0: Writing image manifest
stage0: Writing pod manifest
stage0: Setting up stage1
stage0: Wrote filesystem to /var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d
stage0: Pivoting to filesystem /var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d
stage0: Execing /init
networking: loading networks from /etc/rkt/net.d
networking: loading network default with type ptp
Spawning container rkt-4804ea60-b292-4e20-8b90-7c7cb50f675d on /var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/stage1/rootfs.
Press ^] three times within 1s to kill container.
Failed to create directory /var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/stage1/rootfs/sys/fs/selinux: Read-only file system
Failed to create directory /var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/stage1/rootfs/sys/fs/selinux: Read-only file system
/etc/localtime is not a symlink, not updating container timezone.
systemd 222 running in system mode. (-PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
Detected virtualization systemd-nspawn.
Detected architecture x86-64.

Welcome to Linux!

Initializing machine ID from container UUID.
Failed to install release agent, ignoring: File exists
[  OK  ] Created slice -.slice.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice system.slice.
         Starting Journal Service...
[  OK  ] Created slice system-prepare\x2dapp.slice.
[  OK  ] Started Pod shutdown.
         Starting Pod shutdown...
[  OK  ] Started alpine-sh Reaper.
         Starting alpine-sh Reaper...
systemd-journald.service: Main process exited, code=killed, status=9/KILL
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[DEPEND] Dependency failed for Prepare minim...nment for chrooted applications.
[DEPEND] Dependency failed for Application=a... Image=quay.io/coreos/alpine-sh.
alpine-sh.service: Job alpine-sh.service/start failed with result 'dependency'.
alpine-sh.service: Triggering OnFailure= dependencies.
prepare-app@opt-stage2-alpine\x2dsh-rootfs.service: Job prepare-app@opt-stage2-alpine\x2dsh-rootfs.service/start failed with result 'dependency'.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'signal'.
systemd-journald.service: Service has no hold-off time, scheduling restart.
[  OK  ] Stopped Journal Service.
         Starting Journal Service...
[  OK  ] Reached target rkt apps target.
shutdown.service: Unit not needed anymore. Stopping.
         Stopping alpine-sh Reaper...
[  OK  ] Reached target Halt.
[  OK  ] Stopped alpine-sh Reaper.
shutdown.service: Unit not needed anymore. Stopping.
         Stopping Pod shutdown...
Shutting down.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Sending SIGKILL to PID 3 (systemd-journal).

Some system stats:

➜  rkt-v1.0.0  uname -a && mount
Linux asi 4.3.0-0.bpo.1-amd64 #1 SMP Debian 4.3.3-7~bpo8+1 (2016-01-19) x86_64 GNU/Linux
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=1004931,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=1612152k,mode=755)
/dev/mapper/asi--vg-root on / type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda2 on /boot type ext2 (rw,relatime,stripe=4)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=806076k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
overlay on /var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/stage1/rootfs type overlay (rw,relatime,lowerdir=/var/lib/rkt/cas/tree/deps-sha512-edba7b045eec6d4973400beda30556b6fb615d50d22e92756ddbd745b6d6f05a/rootfs,upperdir=/var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/overlay/deps-sha512-edba7b045eec6d4973400beda30556b6fb615d50d22e92756ddbd745b6d6f05a/upper,workdir=/var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/overlay/deps-sha512-edba7b045eec6d4973400beda30556b6fb615d50d22e92756ddbd745b6d6f05a/work)
overlay on /var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/stage1/rootfs/opt/stage2/alpine-sh/rootfs type overlay (rw,relatime,lowerdir=/var/lib/rkt/cas/tree/deps-sha512-82a099e560a596662b15dec835e9adabab539cad1f41776a30195a01a8f2f22b/rootfs,upperdir=/var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/overlay/deps-sha512-82a099e560a596662b15dec835e9adabab539cad1f41776a30195a01a8f2f22b/upper/alpine-sh,workdir=/var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/overlay/deps-sha512-82a099e560a596662b15dec835e9adabab539cad1f41776a30195a01a8f2f22b/work/alpine-sh)
nsfs on /var/lib/rkt/pods/run/4804ea60-b292-4e20-8b90-7c7cb50f675d/netns type nsfs (rw)

[alban]

I tested the pre-built release rkt-v1.0.0 on Debian 8.1 (Jessie) with the image ami-02b78e1f from https://wiki.debian.org/Cloud/AmazonEC2Image/Jessie and it worked for me.

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux ip-172-31-25-10 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux

$ dpkg -l |grep systemd  #  On the host
ii  libsystemd0:amd64              215-17+deb8u1                amd64        systemd utility library
ii  systemd                        215-17+deb8u1                amd64        system and service manager
ii  systemd-sysv                   215-17+deb8u1                amd64        system and service manager - SysV links

$ dpkg -l |grep journal  #  On the host (no results)

At a first glance, I don't see how the kernel version or btrfs would make a difference. So it needs more investigation.

can only be stopped by killing it the hard way (kill -9)

Does pressing ^]^]^] (control + ']' 3 times) work?

[trusch]

My systemd version differs a little bit:

➜  ~  dpkg -l |grep systemd
ii  libsystemd0:amd64              215-17+deb8u3                amd64        systemd utility library
ii  systemd                        215-17+deb8u3                amd64        system and service manager
ii  systemd-sysv                   215-17+deb8u3                amd64        system and service manager - SysV links

The /etc/os-release is identical to the one provided by you.

Pressing ^]^]^] doesn't work. When the third press is done, I get a newline but the process does not exit.

What information could I provide to help debugging this issue? My knowledge about the internals of rkt are not so good, so debugging this by myself is not very promising.

[squall0gd]

@trusch rkt 1.0 on Debian 8.0 is working fine for me (as well kvm flavor and coreos flavor)

As far as I know, systemd features depend on kernel and its configuration. Are you able to reproduce this situation on kernel from non-backport repo?

My /etc/os-release:

PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Current systemd:

ii  libpam-systemd:amd64                  215-17+deb8u3                        amd64        system and service manager - PAM module
ii  libsystemd0:amd64                     215-17+deb8u3                        amd64        systemd utility library
ii  systemd                               215-17+deb8u3                        amd64        system and service manager
ii  systemd-sysv                          215-17+deb8u3                        amd64        system and service manager - SysV links

Kernel version

#1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17)

[asbachb]

I've the same issue with arch linux, systemd 229

When I start the container with --no-overlay it seems to work.

[zmedico]

When I start the container with --no-overlay it seems to work.

That solved the problem for me too, when /var/lib/rkt is btrfs (on gentoo). If I mount ext4 on /var/lib/rkt, then I don't need the --no-overlay option. So, it seems to be a bad interaction between overlay and btrfs. As I recall, docker refuses to use the overlay graph driver when /var/lib/docker is btrfs, so it may be for the same reason.

[asbachb]

I can't test it with ext4, but can confirm that I'm using btrfs too.

[trusch]

I reproduced the problem today on a fresh debian testing installation on a btrfs.

rkt version: 1.1.0

Kernel:
Linux carbon 4.3.0-1-amd64 #1 SMP Debian 4.3.5-1 (2016-02-06) x86_64 GNU/Linux

Systemd:
ii libsystemd0:amd64 229-2 amd64 systemd utility library
ii systemd 229-2 amd64 system and service manager
ii systemd-container 229-2 amd64 systemd container/nspawn tools
ii systemd-sysv 229-2 amd64 system and service manager - SysV links

[jellonek]

rkt will not work (at least for now) if /var/lib/rkt is on btrfs, so mount tmpfs on this directory or use other filesystem for it.
It's known issue of overlayfs on btrfs, on which AIR some redhat engineers are working from the kernel side.

[mischief]

this is fixed in torvalds/linux@de17e79, which is available in linux 4.5.2 (and CoreOS >=1029.0.0)

[philips]

@trusch Based on mischief's comments can you get this working now?

[jellonek]

#2600 related.

[jonboulle]

Please re-open if you can still reproduce on Linux 4.5.2+. We're also planning on adding some checks for these conditions in #2619. Thanks!