[Bug] Cannot run Docker inside MicroVM

[WeeJeWel]

The Docker service does not start inside a MicroVM.

To Reproduce

1. Start a Firecracker MicroVM. Tested using ubuntu xenial and debian jessie.
2. Get internet working.
3. Follow the Docker Engine installation guide.
4. Docker won't start:

$ journalctl -fn 100 -u docker

...
dockerd[4984]: failed to start daemon: Error initializing network controller: Error creating default "bridge" network: package not installed

Expected behaviour

The Docker service should start, and I should be able to run docker run hello-world.

Environment

• Firecracker version 1.2.0
• Host and guest kernel versions https://s3.amazonaws.com/spec.ccfc.min/img/aarch64/debian_with_ssh_and_balloon/kernel/vmlinux.bin
• Rootfs used https://s3.amazonaws.com/spec.ccfc.min/img/aarch64/debian_with_ssh_and_balloon/fsfiles/debian.rootfs.ext4
• Architecture aarch64

Checks

• Have you searched the Firecracker Issues database for similar problems?
• Have you read the existing relevant Firecracker documentation?
• Are you certain the bug being reported is a Firecracker issue?

[JonathanWoollett-Light]

Thank you for letting us know. We will look into this.

[dianpopa]

Hi @WeeJeWel

This does not look to be related to Firecracker but rather a general OS error where some additional setup might be needed.

However, seems that that the "bridge-utils" package needs to be installed as a dependency of docker. Did you do that? Can you check what sudo modinfo bridge gives you?

[WeeJeWel]

Thank you for replying @dianpopa.

I've followed the Getting Started, downloaded kernel and rootfs.

# Set up networking
ip addr add 172.16.0.2/24 dev eth0
ip link set eth0 up
ip route add default via 172.16.0.1 dev eth0
echo "nameserver 8.8.8.8" > /etc/resolv.conf

# Set up apt
mkdir -p /var/lib/dpkg/{info,alternatives}
touch /var/lib/dpkg/status
mkdir /var/lib/dpkg/updates/
apt update
DEBIAN_FRONTEND=noninteractive apt-get -yq install apt-utils
apt install -y iputils-ping net-tools

# Try bridge module
$ modinfo bridge
-bash: modinfo: command not found

$ apt install kmod

$ modinfo bridge
modinfo: ERROR: Module alias bridge not found.

$ apt install bridge-utils
$ modinfo bridge
modinfo: ERROR: Module alias bridge not found.

$ reboot
$ modinfo bridge
modinfo: ERROR: Module alias bridge not found.

$ modprobe bridge
modprobe: FATAL: Module bridge not found in directory /lib/modules/4.14.174+

$ uname -a
Linux ubuntu-fc-uvm 4.14.174+ #14 SMP Mon Nov 23 20:44:24 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

Unfortunately I don't know how to get the bridge module to load. By the way, /lib/modules/ does not exist.

[dianpopa]

Firecracker's guest configs do not allow for loadable modules, that is why /lib/modules does not exist. The modules are statically compiled into the kernel (for performance purposes).

I spent some time replicating this and it seems that some modules needed to be compiled into the kernel. More specifically,
CONFIG_BRIDGE and CONFIG_NETFILTER_XT_MATCH_ADDRTYPE

Also I needed to do this about iptables:

update-alternatives --set iptables /usr/sbin/iptables-legacy

I also discovered later that 4.14 has some runc incompatibility since it is too old so I actually compiled a 5.10 docker enabled kernel that you can find here: https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/aarch64/kernels/vmlinux-docker-5.10.bin and which seems to do the job since I was able to run docker run hello-world :D.

GIve it a try and let me know if this also works for you.

[WeeJeWel]

@dianpopa That works! 🎉

You're an absolute wizard, thank you!

[chinchaun]

I also discovered later that 4.14 has some runc incompatibility since it is too old so I actually compiled a 5.10 docker enabled kernel that you can find here: https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/aarch64/kernels/vmlinux-docker-5.10.bin and which seems to do the job since I was able to run docker run hello-world :D.

hey @dianpopa can you share the kernel config or the script that you used to build that one? I would like to build one but for an x86_64, and try to run on ubuntu 18 and 20, thanks!

[felipecruz91]

Hi @dianpopa, do you happen to have the kernel built for x86_64?

[dianpopa]

Hi @felipecruz91 and @chinchaun

Unfortunately I do not have an x86_64 docker enabled kernel yet. Let me look for the config I used for building the arm guest kernel and get back to you.

Thanks!

[dianpopa]

You can find a working x86_64 guest kernel here: https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux-5.10.bin.

The working config is actually the one from our resources.

Let me know if this works.

[BasToTheMax]

You can find a working x86_64 guest kernel here: https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux-5.10.bin.

This worked, thanks for sharing this kernel image :)

[ashwin31]

You can find a working x86_64 guest kernel here: https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux-5.10.bin.

The working config is actually the one from our resources.

Let me know if this works.

this is not working for me. I tried with firecracker 1.9. is there a specific version it is compatible with?
do we have official kernel which supports docker listed out somewhere ?

[Manciukic]

The latest CI kernel supports everything needed to run Docker on the provided Ubuntu image, see #4791