[Spike] Investigate if kcrypt KMS server can be auto-discoverable

[jimmykarily]

There are use cases, in which the cluster doesn't have stable internet connectivity (or no internet connectivity at all). In those cases, encryption will either work locally or a KMS server will be deployed somewhere locally accessible (locked up in a room or something).

It would be nice if a "moving" KMS could be used. That would be a machine that runs the KMS server but doesn't necessarily have a static IP address. As soon as it connects to the same network, the nodes should be able to discover it and decrypt their disks.

There are various service auto-discovery network protocols that could help there. Let's do some research and see what is possible.

[jimmykarily]

This library seems to do what it promises: https://github.com/hashicorp/mdns

We could check the domain of the kcrypt server in the Kairos config and if the top level is .local (or something like that) we could look up the IP address using mdns.

The automatic IP address discovery on the server side in the example in the README seems to pick the 127.0.1.1 address which is not useful (at least on my system where I tested it). We might need to implement some logic to find the IP address of the correct interface.

[jimmykarily]

For this specific scenario, we could have an implementation of the kcrypt challenger server that doesn't need to run inside Kubernetes. For example, it could be a standalone application (golang binary?) that stores the passphrases in a flat file encrypted with pgp. When the application starts with will ask for a passphrase to decrypt the file and load the passphrases in memory and will then start an mdns server to reply to any node that does an mdns query.

Combined with some automated way to decrypt the file (e.g. a yubikey), the server could be something as simple as a raspberry pi running on a battery, or even a mobile app.

So when the person with the KMS device is around (connected to the network), Kairos nodes boot up and decrypt their disks. When that person is off (drinking Piña colada on a beach in Foinikouda for example 🍹), the cluster is safe and encrypted with no way to decrypt. Only works if that person remembered to shut everything down before leaving of course.

[jimmykarily]

Initially let's implement this on the server that runs in Kubernetes. The problem here is which IP address should the server advertise over mdns. In Kubernetes, "things" that run inside Pods, become accessible to the outside world through Services. Our server should first discover it's own IP address by looking up for the service that exposes it and then advertise that IP address over mdns.

That's a very fine plan but here we are talking about a machine that moves around, possibly changing IP addresses. Let's assume that this machine is running a k3s/kind cluster with the KMS server running on it. These "local" clusters usually setup a new network interface that doesn't change IP address when the machine reconnects to a network. That IP address is the one assigned to Load Balanced services.

The way we usually setup such "local" clusters to work is by binding some host ports to the ports of the cluster container. So we have the host's network interface that might change IP address and the cluster container's interface which doesn't change address.

I don't think there is a way to know the host's IP address from withing the cluster in a default setup.
I'll think of some solution but any ideas are welcome.

[jimmykarily]

One solution would be to leave the current implementation of the KMS untouched and just implement a new utility that can run outside the cluster and act as an mdns server that advertises a local IP address and port which is bound to the container's IP address.

In other words, instead of the KMS server advertising it self over the network, we will have someone else doing so. Let me play with it and see if it works.

[jimmykarily]

WIP branches:

• https://github.com/kairos-io/kcrypt-challenger/tree/2069-mdns-kms
• https://github.com/kairos-io/tpm-helpers

[jimmykarily]

Created a little tool to expose the server: https://github.com/kairos-io/simple-mdns-server

[jimmykarily]

I think this Spike can be considered complete. There is a PR here:
kairos-io/kcrypt-challenger#45

with instructions on how to set it up. This tool: https://github.com/kairos-io/simple-mdns-server is used in the instructions, to advertise the KMS over mdns but maybe this can also be achieved with existing tools like avahi-publish-service but I didn't try.

TODO:

• The mdns utility should find the IP address everytime it receives a request right before the response. That's because the IP address may have changed since last time.
• Don't rely on ".local" in the settings to do a lookup. After all, in mdns the lookup is based on the service type not the domain (like it happens in dns). We can add a boolean setting mDNS: true to do an mdns lookup and if it fails we could revert to the server address (as it was up to now).

In any case. This is working. Let's see if there are additional requirements before we release with this feature (e.g. require no settings for the server at all and do a lookup if install.encrypted_partitions is set?)

Let's move the discussion to the implementation ticket here: #2087