making root filesystems

[Joe-Degs]

couple of links to people talking about building kernels and making root filesystems. No mention of OCI images but this would be useful in the making of the userspaces.

Rob's talks on compiling simple linux systems

• compilling simplest linux system
• building linux via automated scripts
• toybox vs busybox

Github repos of setting up userspace linux

• mkroot - script for simple linux build
• gokrazy - bootstraps go programs on raspberry pis

[bxffour]

Alright. I'd look into them then we can have a meeting next week to discuss the way forward. What do you think?

[bxffour]

Okay, after conducting thorough research and investigation, I have gained a deeper understanding of the components involved and even gained insights into how an initialization process for this system might look. When it comes to creating root filesystems for a Linux system, there are numerous approaches, some of which can be overwhelming, while others are deceptively simple. However, fortunately for us, Firecracker has specific requirements for its root drive, so as long as we meet those criteria, we should be fine. Whether you choose to use tools like mkroot, gokrazy, or deconstructed Docker images, as long as you provide it as a virtual drive, it will work. Nevertheless, I am particularly enthusiastic about supporting tooling related to OCI image-based root filesystems due to the ecosystem surrounding it. Creating your own becomes as effortless as writing a Dockerfile.

So, what prompted me to explore this idea further was a friend of mine who wanted to create something unique for his blog. I suggested adding interactive elements to it, such as providing a shell or some form of interface for his users to experiment with. That's when I proposed Firecracker as a backend for this project, aiming to make our tooling useful for his goals. I envision a scenario where a user can package everything into a Docker image and simply hand it over to us, and we take care of the rest. I believe that OCI image-backed root filesystem solutions would provide the optimal experience in such cases. I also delved into fly.io's init, which appears to be a perfect fit for a workload like this.

Here's the basic concept: we flash the init and its configuration onto a drive, similar to how we handle other Firecracker root filesystems, and mount the init as the root drive (e.g., '/dev/vda'). Then, we mount the actual rootfs as an additional drive (e.g., '/dev/vdb') and launch Firecracker. The purpose of the init is to perform initial setups and then pivot_root/chroot into '/dev/vdb'. Consequently, we don't need to install and configure something like rcinit or systemd into our root filesystem. We can simply provide it as is, add a few configurations, and it will boot seamlessly.

That's the progress I've made so far. What are your thoughts on this?"

[Joe-Degs]

fully support the idea of going with OCI image based rootfs, simpler to get images -- and various kinds of them.

another issue that's been eating away at my brain is how we manage the running system: can we add userspace software after initial setup or do we need to specify everything that needs to run in the machine in the machine's config (dockerfile in this case). and is the system going to be flexible enough to use as a server (akin to how cloud virtual machines are used) or is it essentially just going to be a runner for docker images?

[bxffour]

fully support the idea of going with OCI image based rootfs, simpler to get images -- and various kinds of them.

another issue that's been eating away at my brain is how we manage the running system: can we add userspace software after initial setup or do we need to specify everything that needs to run in the machine in the machine's config (dockerfile in this case). and is the system going to be flexible enough to use as a server (akin to how cloud virtual machines are used) or is it essentially just going to be a runner for docker images?

Generally I want us to make this as flexible as possible. We're building tools that's going to be the core of other tools and stuff we're going to do.
So about the userspace software I think we can do both. We can even do something like what vagrant does so we specifically the packages you might need installed then we do it at setup time. We can also explore running commands at runtime through SSH. But the point is it has to be flexible enough to solve whatever problem we might have in future. Also I intend for them to be used as servers or runners for docker files. So fly's init might only be used for transfigured oci images but for traditional server like workloads we go with a regular init system or something of our own making. We don't have to lock ourselves in a box rn

[bxffour]

This is a script to unpack OCI bundles

skopeo copy docker://docker.io/alpine:latest oci://$(pwd)/alpine:latest

./unpack.sh alpine outpath

#!/usr/bin/env bash

oci_image_path=$1
rootfs=$2

blobs="$oci_image_path"/blobs/sha256
manifest_digest=$(cat "$oci_image_path"/index.json | jq '.manifests[0].digest' | grep -oP '(?<=:).*(?=")')
manifest="$blobs"/"$manifest_digest"

readarray -t layers <<< "$(cat "$manifest" | jq '.layers[].digest' | grep -oP '(?<=:).*(?=")')"
for el in "${layers[@]}"; do
  tar xf "$blobs"/"$el" -C "$rootfs"
done

[bxffour]

Create run.json file

{
  "ImageConfig": {
    "Cmd": ["/bin/bash"],
    "Env": ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],
    "WorkingDir": "/",
    "User": "root"
  },
  "IPConfigs": [],
  "TTY": true,
  "Hostname": "bxffour",
  "Mounts": [],
  "RootDevice": "/dev/vdb",
  "EtcResolv": {
    "Nameservers": ["8.8.8.8", "8.8.4.4"]
  },
  "EtcHosts": []
}

Setting up the init

cargo build --release
fallocate -l 100M tmpinit
mkfs.ext2 tmpinit
mkdir initmount
mount -o loop,noatime tmpinit initmount
mkdir initmount/fly
cp target/x86_64-unknown-linux-musl/release/init initmount/fly/
cp run.json initmount/fly/
umount initmount

[bxffour]

To run the vm using firectl assuming tmpinit as the init drive and rootfs.ext4 as the rootdrive

firectl --root-drive tmpinit --add-drive rootfs.ext4:rw --vsock-device ./v.sock:3 --kernel-opts "ro console=ttyS0 noapic reboot=k panicOD=1 pci=off nomodules init=/fly/init"

[bxffour]

cni config for the modified firecracker

/etc/cni/conf.d/fcnet.conflist

{
  "name": "fcnet",
  "cniVersion": "0.4.0",
  "plugins": [
    {
      "type": "ptp",
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.127.0/24",
        "resolvConf": "/etc/resolv.conf"
      }
    },
    {
      "type": "firewall"
    },
    {
      "type": "tc-redirect-tap"
    }
  ]
}