[WIP] Embeded containerd, shim and runc

[chanwit]

This PR introduces an embedded containerd mode for ignited by preparing static containerd and runc binaries to ship together with ignited.

[] always check if go.sum breaks CI. If so, delete go.sum then do go mod tidy again.
[] TODO ignited would start and shutdown the embedded containerd automatically.
[] upgrade containerd to 1.3.0-rc.0 in hope of to not break the client

[luxas]

This maybe is doable, but I'd invest in better deb/rpm packaging depending on the official containerd packages instead of running containerd in-process. The whole idea is re-using an existing tool you have installed (containerd)

[chanwit]

I was thinking the same. But I found that docker ships with containerd.io and always cause the package conflict when I tries installing a standard containerd package.

I'll leave options for users to choose:

1. external containerd runtime on their host.
2. embedded containerd shipped by us
3. docker

All should be switchable via a flag.
WDYT?

[chanwit]

Also the current containerd client version built into ignite is 1.3.0-beta.2. But the stable version people would install is 1.2.8. It's causing a problem to me (at least on my machine).

[stealthybox]

Also the current containerd client version built into ignite is 1.3.0-beta.2. But the stable version people would install is 1.2.8. It's causing a problem to me (at least on my machine).

Yes, this is a significant problem that makes defaulting to containerd problematic.
Most people have docker installed, but we can't upgrade containerd underneath users.

If we want users to play with ignite, we might provide privileged container environments with controlled versions of containerd or docker.

• depends on user being able to run a container
• skips installing deps
• protects the host
• wraps all VM's in a PID-namespace that can easily be destroyed

[chanwit]

If we want users to play with ignite, we might provide privileged container environments with controlled versions of containerd or docker.

Could we achieve this with runc?

[stealthybox]

We could provide examples for ctr / crictl / podman / rkt / runc.
Most people will use docker on their laptop so it should be the first one to worry about.

I don't think we need to concerned we building 1 generic install solution.
People can install the containerd rc on their hosts if they want, and we can document that.
I'm just thinking about how we help prevent problems for users who already have other tools installed.

[stealthybox]

I get devicemapper errors when trying to run ignite vms on containerd inside of a privileged docker container:

docker run -it --rm \
   -v ~/Repos/ignite:/root/ignite -v /var/lib/containerd \
  --privileged --cap-add ALL \
  test-containerd
root@f851c24da535:~# containerd |& cat >/dev/null &
[1] 11
root@f851c24da535:~# ignite/bin/ignite run weaveworks/ignite-ubuntu
INFO[0000] Docker image "weaveworks/ignite-ubuntu:latest" not found locally, pulling... 
INFO[0054] Starting image import...                     
INFO[0056] Imported OCI image "weaveworks/ignite-ubuntu:latest" (226.5 MB) to base image with UID "36e1a62e6a29f519" 
INFO[0056] Docker image "weaveworks/ignite-kernel:4.19.47" not found locally, pulling... 
INFO[0062] Imported OCI image "weaveworks/ignite-kernel:4.19.47" (49.6 MB) to kernel image with UID "9f23830375005e64" 
INFO[0062] Removed VM with name "long-sun" and ID "030304567fa435dc" 
FATA[0062] command ["dmsetup" "create" "ignite-030304567fa435dc"] exited with "device-mapper: reload ioctl on ignite-030304567fa435dc  failed: No such device\nCommand failed\n": exit status 1 
root@f851c24da535:~# dmsetup ls
ignite-030304567fa435dc-base	(253:1)
ignite-e64cdaedef0a2846	(253:6)
ignite-7dc6ee39e00e7826	(253:24)
ignite-e64cdaedef0a2846-base	(253:5)
ignite-7dc6ee39e00e7826-base	(253:23)
cryptswap	(253:0)

It looks like the base device is created and then it fails from there.

Dockerfile / build.sh (click to expand)

# test-containerd/Dockerfile
FROM ubuntu:latest

WORKDIR /root
RUN apt-get update
RUN apt-get install -y \
     wget

# ignite deps minus containerd
RUN apt-get install -y --no-install-recommends \
     binutils \
     dmsetup \
     git \
     openssh-client

ENV ctd_version="1.3.0-rc.0"
ENV ctd_platform="linux-amd64"
ENV ctd_release="https://github.com/containerd/containerd/releases/download/v${ctd_version}/containerd-${ctd_version}.${ctd_platform}.tar.gz"

RUN wget "${ctd_release}" -O release.tgz -q \
  && tar -xzf release.tgz \
  && rm release.tgz \
  && mv bin/* /usr/bin/ \
  && rmdir bin

# test-containerd/build.sh
#!/bin/sh
docker build \
  -t "test-containerd" \
  .

[darkowlzz]

Hi,
We had a discussion about the above failure to run ignite inside a privileged docker container in the weekly meeting. I spent some time reproducing and investigating the issue. Here are some findings:

• The device mapper error seems to be related to the docker issues created loop devices do not appear in container even when run privileged moby/moby#27886 and New devices on host are not exposed to privileged container moby/moby#16160 . The devices on host are not exposed to privileged container. As a workaround, people have mounted /dev to the container. I tried to same with the test-containerd image from above, but had some error running the container: container init caused \"open /dev/console: input/output error\"": unknown. Instead of /dev in container, I tried mounting /dev from host to /hostdev and got the container running. Started containerd and changed the ignite snapshot device in code to use path /hostdev/mapper instead of /dev/mapper. With that, the device mapper error "no such device" got resolved.
• Running a few times, I observed the same device mapper error "no such device" again. But if I manually run the dmsetup create command, it worked. Added a few seconds of sleep in ignite before creating the snapshot device and that made it work. It could be that there's some delay for the device to be available in the container.
• Continuing the VM creation, next failure:

FATA[0005] command ["resize2fs" "/hostdev/mapper/ignite-e8c39e136be2c064"] exited with "resize2fs 1.44.1 (24-Mar-2018)\nopen: No such file or directory while opening /hostdev/mapper/ignite-e8c39e136be2c064\n": exit status 1 

This is resize2fs command run after e2fsck. Again, running the command manually worked. Added some sleep before the command execution made it work.

• Continuing the VM run, encountered errors related to missing cni binaries. Those are not added in the above Dockerfile. Adding them resolved the cni issues. This was followed by failure due to missing iptables and runc binaries. Added them and the VM started successfully.

root@1e4324483442:~# ignite/bin/ignite run weaveworks/ignite-ubuntu:latest
INFO[0000] containerd image "weaveworks/ignite-ubuntu:latest" not found locally, pulling... 
INFO[0020] Starting image import...                     
INFO[0023] Imported OCI image "weaveworks/ignite-ubuntu:latest" (226.5 MB) to base image with UID "736f7ec47d97abe0" 
INFO[0023] containerd image "weaveworks/ignite-kernel:4.19.47" not found locally, pulling... 
INFO[0028] Imported OCI image "weaveworks/ignite-kernel:4.19.47" (49.6 MB) to kernel image with UID "d856f1fa8ebd91e7" 
INFO[0039] Pulling image "darkowlzz/ignite:dev"...      
INFO[0045] Networking is handled by "cni"               
INFO[0045] Started Firecracker VM "5c1d2e3595e62a8d" in a container with ID "ignite-5c1d2e3595e62a8d" 
root@533a5559c995:~# ignite/bin/ignite ps
VM ID			IMAGE				KERNEL					SIZE	CPUS	MEMORY		CREATED		STATUS		IPS		PORTS	NAME
5c1d2e3595e62a8d	weaveworks/ignite-ubuntu:latest	weaveworks/ignite-kernel:4.19.47	4.0 GB	1	512.0 MB	5m14s ago	Up 5m14s	10.61.0.2	damp-cherry

Hope this helps.