Support user-data in minimal ISO image (Was: Add cloud-init service to minimal ISO)

[wmertens]

There are other cloud providers than AWS that support the userdata system.

Since the minimal ISO works everywhere, I propose that the ISO gets services.cloud-init.enabled = true;. If there is no config source, nothing will happen, and thus the minimal ISO can be used to bootstrap on cloud services.

[wmertens]

Looking at http://cloudinit.readthedocs.io/en/latest/topics/examples.html and the fact that the AWS ones can just be bash scripts, there is no real standard. It seems that you can provide any data you like.

So for a NixOS deployment-on-a-cloud-using-ISO scenario, what would be good information to expect in user-data? I'm thinking:

• Some way to specify disk configuration
• SSH key(s) for root
• configuration.nix
• …anything else?

It would be kind of cool to have user-data just be a Nix file and it provides attributes for all the above, which would be evaluated in a way that would allow partitioning the disk and installing to /mnt.

Maybe a derivation that builds an install script that is run when it built successfully?

/cc @Mic92 @copumpkin

[nlewo]

I'm currently using cloud-init to push ssh-keys but I think it would be better to push a configuration.nix and rebuild the system.
There is a cloud-init module to write a file and another one to run a command. So, I think via user-data, we could write a configuration.nix file a run nixos-rebuild switch. But I never tryied.

Do you know the size of the cloud-init closure?
Which cloud provider or IAAS stack are you using?

[wmertens]

* when running `nix-shell -p cloud-init` on a recent minimal ISO (300MB) it wanted to download 60MB * cloud provider: Vultr * the problem with just writing configuration.nix is that you need to have an already-nixos image, which is not available on all cloud providers. The ISO works almost anywhere.

…

On Wed, Apr 18, 2018 at 11:10 AM lewo ***@***.***> wrote: I'm currently using cloud-init to push ssh-keys but I think it would be better to push a configuration.nix and rebuild the system. There is a cloud-init module to write a file and another one to run a command. So, I think via user-data, we could write a configuration.nix file a run nixos-rebuild switch. But I never tryied. Do you know the size of the cloud-init closure? Which cloud provider or IAAS stack are you using? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#39076 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADWluIeekwClERvtY8L5YQMUYdYz7k9ks5tpwMLgaJpZM4TYkgY> .

[coretemp]

@wmertens Good idea. When work on this is done, clarifying the documentation by mentioning
http://cloudinit.readthedocs.io/en/latest/ would be helpful too then. Perhaps including a list of cloud providers in the documentation which support cloudinit would also be useful. Perhaps the cloudinit project makes such a list available.

[edolstra]

Adding 60 MB to the minimal ISO is not really compatible with it being minimal...

[edolstra]

I see that cloud-init has a 721 MiB closure size. That's quite a lot for a glorified version of curl http://169.254.169.254/.

[wmertens]

Indeed, which is why I think it's better to just get the user-data and parse it manually if it's there.

…

On Wed, Apr 18, 2018 at 3:25 PM Eelco Dolstra ***@***.***> wrote: I see that cloud-init has a 721 MiB closure size. That's quite a lot for a glorified version of curl http://169.254.169.254/. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39076 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADWln1gkC_fieJbhPaL2SrMztiwIAzLks5tpz7ZgaJpZM4TYkgY> .

[wmertens]

Sorry I see now that this wasn't clear from the issue; part of the conversation was in the PR.

So I think cloud-init is quite bloated and a no-go; we should just get and parse the data ourselves.

So this issue is now about "what data should we support and how do we implement this?".

It would actually be fun to be able to make a solution that wouldn't even require an installation; if we make something that derives a script that is auto-run, then single-shot services could be implemented this way. Quick boot from ISO, bind-mount disk to store, install extra deps and run service.

[edolstra]

Maybe we can factor out and extend the metadata stuff from nixos/modules/virtualisation/amazon-image.nix and nixos/modules/virtualisation/amazon-init.nix into a separate module.

BTW I do have to wonder about the security implications of fetching stuff from 169.254.169.254 in arbitrary environments.

[copumpkin]

Yeah, I'd generally support the idea of expanding the scope of the current "configure from userdata" implementation (I've wanted it on Packet.net, gcloud, etc.), but cloud-init never felt like a good fit to me for NixOS, since it tries to stick a pseudo-declarative interface on top of something that's actually declarative, but there's a lot of bad fits. That's largely why I didn't write a custom NixOS module for cloud-init (apart from it being terribly documented) and instead put the userdata handling natively into a NixOS module.

[wmertens]

@edolstra good point about the security implications - it's a link-local, non-routable address, so if the ISO is booted somewhere and there is a malicious player on the same network and it can fingerprint the OS as NixOS it could conceivably return the correct payload to turn it into a controlled instance while pretending to be a normal server. So perhaps this should go on a separate cloud-installer ISO image instead.

@copumpkin the way I read it, you parse the SSH key from the user data and that's about it, right? I'm rusty on my AWS, can we assume that for every cloud provider that supports cloud-init, we can provide fully custom payload for the user-data?

I'm thinking these might be the steps:

• Check if a route for 169.254.169.254 was provided via DHCP
• Get user-data, check if Nix expression, if not, ignore
• Merge expression with our nixos-cloud-init module; build some attribute, e.g. install
• If build succeeded, run install as root

[copumpkin]

@wmertens no, on AWS, you can include an arbitrary NixOS config in userdata and the instance will configure itself to match the specified config at startup time.

[wmertens]

@copumpkin right - but I'd like for us to come up with a system that can be used to configure both ISO and AMI use cases. I mean, in theory you could make a system service that then installs to disk, but more specialized install vs configure attributes would be easier to use and maybe use less memory to evaluate.

Some use cases:

• AWS configuration of booted AMI
• Deploy of system using ISO followed by boot from disk
• Deploy of service using only ISO, no install to disk

There seem to be a lot of ways that userdata is made available, see http://cloudinit.readthedocs.io/en/latest/topics/datasources.html

• http://169.254.169.254/ (most popular)
• http://100.100.100.200/ (alibaba and others)
• floppy injection (RH)
• iso injection (VMware, Azure) (not sure if that will work with another iso)

All of these sources seem to provide a way to get a Nix configuration text to the VM. Some also provide extra data like network and SSH keys etc. So it seems to me that a Nix build script that is provided with this extra data can then build scripts for installation or configuration, which are then run as part of the bootup.

Then it's up to deciding if it can go in the generic minimal ISO, or if it should be a separate ISO.

I'm certainly not proposing to support all things immediately, my use case is the 169.254.169.254 endpoint with SSH and install.

[copumpkin]

Oh yes, so that's the thing I'm saying I want too.

I just don't really see the benefit of switching over to cloud-init, with all the other baggage it brings with it. I've written custom modules for it, intensely disliked the state of its code and documentation, and adding a list of two or three such datasources never seemed like the end of the world to me. I don't want YAML in my userdata, which it would force me to put there. Its notion of declarative users seems like a bad fit for our notion of it, unless you want to add a big translation layer and get rid of all its attempts to add users, etc.

[eliasp]

Did anyone already look into Ignition?

• Developed/maintained by CoreOS
• Comes as a single Go binary
• Works with user-data provided configurations, but also supports other platforms/mechanism
• Designed to be a declarative provisioning tool, not a configuration/execution tool like cloud-init

In general it seems to be designed quite well (having learnt from the issues of cloud-init), comes with a proper specification which is used to validate and provide helpful error context.

[Mic92]

Ignition does not look like it is compatible with NixOS: https://coreos.com/ignition/docs/latest/examples.html (it modifies configuration itself). I also suspect it makes many assumption about running on CoreOS, that do not apply on other platforms.

[copumpkin]

Yeah, NixOS by default is notch or two above pretty much all declarative config systems, so I'm not sure why we'd take a step back down the declarative ladder to be compatible with something that doesn't map nicely to what we do. We have a "NixOS config in userdata" system today that works and we've been using for years. It's not ideal to have to know the various platform-specific endpoints to hit, but it's not awful either.

[ghost]

Sorry for the noise; just wanted to say this is a really cool idea and I was wondering what the current status of this is.

[Mic92]

It's not enabled by default, but you could build your own ISOs that have it: https://github.com/nix-community/nixos-generators

[wmertens]

@Mic92 @copumpkin so the conclusion here is: the current system based on cloud-init is good enough for NixOS use, but its closure size is too big for the minimal iso, so it won't get added.

Is that correct?

[Mic92]

@wmertens Yes. Maybe someone could provide a nix-community project to build variants of NixOS ISOs that we do not have in nixpkgs itself.

[flokli]

Other distros also recognized the cloud-init package and its dependencies is way too much bloat. There is flamingo, meant to be a lightweight replacement for cloud-init in Atomic Host, but is somewhat abandoned.

There's also clearlinux/micro-config, and https://github.com/coreos/coreos-cloudinit.

Some distros seem to be moving to https://github.com/coreos/ignition, which is also a (small) Go binary, but I'm not sure if this can be a replacement, and how well it'd integrate with NixOS.

[Mic92]

Some distros seem to be moving to coreos/ignition, which is also a (small) Go binary, but I'm not sure if this can be a replacement, and how well it'd integrate with NixOS.

Ignition does not look too bad after grepping through the code. They only reference to /usr/ is /usr/lib/ignition and but can be overwritten exporting SYSTEM_CONFIG_DIR. All executables are looked up through PATH rather than absolute paths...

I think the main issue is that it requires systemd in initrd, which is not what we do right now.

[flokli]

I meant "I'm not sure if this can be a replacement", as Ignition uses another format for its files, and probably we want to keep support for the cloud-init format.

clearlinux/micro-config also doesn't look too complicated to get working.

[Mic92]

It looks better. One problem I see: Enabling it by default in our minimal iso is a security risk because if an attacker controls the link-local address they can execute arbitrary code. In cloud environment this is fine, but I don't want to expose every user running our installer to that.

[flokli]

Yeah, it probably shouldn't be part of the minimal installer, but it'd be nice to have that tooling available, and emit a NixOS Server|Cloud image with cloud-init support enabled, that could be used by cloud providers.

[OddBloke]

Hey folks, I'm an upstream cloud-init dev and just happened across this thread. I just wanted to respond to one specific comment:

Enabling it by default in our minimal iso is a security risk because if an attacker controls the link-local address they can execute arbitrary code. In cloud environment this is fine, but I don't want to expose every user running our installer to that.

cloud-init has started shipping a systemd generator precisely to handle this case: if it doesn't find that it's running on a cloud platform (by examining DMI data, generally, so this only works on Intel platforms; don't know if that's a concern for NixOS) then it will entirely disable cloud-init for the boot: https://github.com/canonical/cloud-init/blob/master/systemd/cloud-init-generator.tmpl (which relies on https://github.com/canonical/cloud-init/blob/master/tools/ds-identify).

(The version of cloud-init currently in NixOS is too old to include it, but I know that @bcdarwin was looking into updating it earlier today: bcdarwin@8e126a5)

[Mic92]

@OddBloke I have updated cloud-init here: #95746
If you have time for testing it, we can get it into the next release.

[stale]

I marked this as stale due to inactivity. → More info

[mbrgm]

@OddBloke Did you make any progress following @Mic92's PR?

[stale]

I marked this as stale due to inactivity. → More info

[msgilligan]

Some form of cloud-init support (doesn't have to be in minimal image) would allow fast/easy creation of NixOS instances with Lima, see: lima-vm/lima#430

[bam80]

Canonical has adopted cloud-init config in it's Ubuntu Server Installer since version 20.04:
https://ubuntu.com/server/docs/install/autoinstall

It's not enabled by default, but you could build your own ISOs that have it: https://github.com/nix-community/nixos-generators

If we can't (or don't want, for some unclear to me reason) do the same by default, I would love if we document at least how one could generate the featured installer .iso, if needed.

I'm new to Nix and the cloud-init support in it's installer seems like a very hidden subtle possibility to me.
Second day I'm struggling and still have no clear answer if it's possible, at all.
While the installing is among the very first things one would do, and shouldn't bring such an obstacles..

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/bluetooth-keyboard-in-stage-1/19259/25

[physics-enthusiast]

Maybe this recent project of mine may be relevant?