Not able to start Docker daemon inside LXC container in OpenWRT image build using Yocto.

[satishnaidu]

Required information

• Distribution: OpenWRT
• Distribution version: Linux 4.14
• Device: Raspberry Pi 3
• The output of

  • lxc-start --version : 2.0.8

  • lxc-checkconfig
    --- Namespaces ---
    Namespaces: enabled
    Utsname namespace: enabled
    Ipc namespace: enabled
    Pid namespace: enabled
    User namespace: enabled
    Network namespace: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: enabled
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
FUSE (for use with lxcfs): enabled

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: missing
File capabilities: enabled

• uname -a:
  Linux LEDE 4.14.39 O.S. Systems pending fixes #1 SMP Thu Jul 12 00:36:38 UTC 2018 armv7l GNU/Linux

• cat /proc/self/cgroup:
  root@LEDE:/home/root# cat /proc/self/cgroup
  1:cpuset,cpu,cpuacct,blkio,devices,freezer,net_cls,perf_event,net_prio:/

• cat /proc/1/mounts
  root@LEDE:/home/root# cat /proc/1/mounts
  /dev/root / ext4 rw,noatime,data=ordered 0 0
  devtmpfs /dev devtmpfs rw,relatime,size=470104k,nr_inodes=117526,mode=755 0 0
  proc /proc proc rw,nosuid,nodev,noexec,noatime 0 0
  sysfs /sys sysfs rw,nosuid,nodev,noexec,noatime 0 0
  cgroup /sys/fs/cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset,cpu,cpuacct,blkio,devices,freezer,net_cls,perf_event,net_prio,clone_children 0 0
  tmpfs /tmp tmpfs rw,nosuid,nodev,noatime 0 0
  tmpfs /dev tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
  devpts /dev/pts devpts rw,nosuid,noexec,relatime,mode=600,ptmxmode=000 0 0
  debugfs /sys/kernel/debug debugfs rw,noatime 0 0

Issue description

Not able to start docker inside LXC Ubuntu container, even though I enabled cgroup configuration in container config file.
I can start docker on host OpenWRT image, but when I try to start docker inside LXC container, it is failed with error "Device's cgroup isn't mounted".

I raised this issue with LXC Github, they responded, it's because of "Mounting all cgroups into a single hierarchy" in OpenWRT system.
lxc/lxc#2483 (comment).

NOTE: In raspbian stretch armhf architecture, I am able to run docker inside LXC without any issues, only facing issues with OpenWRT image.

Do we have any solution on OpenWRT to mount cgroups as multiple hierarchies inside LXC container, in order to run docker inside LXC.

Error message:
root@c1:/# dockerd -s vfs
INFO[0000] libcontainerd: new containerd process, pid: 18
WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support cgroup memory limit
WARN[0001] Unable to find cpu cgroup in mounts
WARN[0001] Unable to find blkio cgroup in mounts
WARN[0001] Unable to find cpuset cgroup in mounts
WARN[0001] mountpoint for pids not found
Error starting daemon: Devices cgroup isn't mounted

Steps to reproduce

1. lxc-start -n c1 --logfile test.log --logpriority DEBUG ( Ubuntu container)
2. lxc-attach -n c1
3. apt-get update , apt-get install docker.io
4. dockerd -s vfs

Information to attach

• Container configuraiton : /var/lib/lxc/c1/config

##Template used to create this container: /usr/share/lxc/templates/lxc-download
##Parameters passed to the template:
##Template script checksum (SHA-1): 740c51206e35463362b735e68b867876048a8baf
##For additional config options, please look at lxc.container.conf(5)

##Uncomment the following line to support nesting containers:
##lxc.include = /usr/share/lxc/config/nesting.conf
##(Be aware this has security implications)

##Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.arch = linux32

##Container specific configuration
lxc.rootfs = /var/lib/lxc/c1/rootfs
lxc.rootfs.backend = dir
lxc.utsname = c1

Network configuration

lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up

Cgroup configuration

lxc.aa_profile = unconfined
lxc.mount.auto = proc:rw sys:rw cgroup:rw
lxc.autodev = 1
lxc.cgroup.devices.allow = a
lxc.cap.drop =

##lxc.mount.entry = proc proc proc nosuid,nodev,noexec 0 0
##lxc.mount.entry = sysfs sys sysfs nosuid,nodev,noexec 0 0

[satishnaidu]

Hi Team,

I am able to resolve this issue to run Docker inside LXC container on OpenWRT by resolving cgroup issues using the steps below:

• Create a fstab file at container roots /var/lib/lxc/<container_name>/fstab
• Adding following cgroup configuration inside fstab
  cgroup /sys/fs/cgroup cgroup defaults 0 0
• Login to inside the container lxc-attach -n <container_name>
• mount /sys/fs/cgroup
• dockerd -s vfs &
• docker run hello-world

Please let me know if there is a better way to do this or any configuration to avoid manual steps.

Thanks,
Satish Kumar Andey,

[SeriousM]

@satishnaidu sorry but I don't understand all your commands.
would you be so kind to write the full commands down?

[oxr463]

@satishnaidu were you able to figure this out?