Support userdata privacy beyond cloudinit

[randomvariable]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]
#1490 added support for preventing users reading secrets essential to the cluster via EC2 userdata, by shifting a version of the userdata to AWS Secrets Manager and injecting a boot hook using multi-part MIME to download the real user-data.

This is a cloud-init specific solution which will not work with other bootstrap providers such as Ignition.

The aim is therefore to find a bootstrap provider agnostic form of userdata privacy.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-aws version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[randomvariable]

cc @dongsupark

[randomvariable]

We can think about the original use case though. What mechanisms are there for mechanisms to provide secrets in userdata. I guess these fall into:

• Using s3
• Presigned http URLs

either of those passed into ignition.config.merge

The reason we're using the boothook is that we need to reconstruct the user-data as there's a 10KB limit on AWS Secrets Manager. If we also supported S3 as a mechanism, might become simpler.

[dongsupark]

I think we should try out S3.
That's how the WIP PR kubernetes-sigs/cluster-api#3437 makes use of ignition:
set up a S3 bucket, upload ignition configs to the bucket, and make ignition download the configs.
(still tests needed)

Where should be the ignition code then?
AWS provider repo, or the main cluster-api repo?

[randomvariable]

I think the S3 code should perhaps live in here and work in a similar way to the current method, in that other infrastructure providers can use their own object storage then for the final pass to the userdata.

Either that or you make an S3 backend compulsory which may mean deploying minio and making assumptions around network connectivity between the management and workload clusters.

Long term, we can also think about chaining a "bootstrap generator" and a "bootstrap injector" as separate things for v1alpha4.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[randomvariable]

/lifecycle frozen

[invidian]

I've made some recent work on CABPK (kubernetes-sigs/cluster-api#4172), which adds support to generate bootstrap data in Ignition format.

I also have a WIP branch for CAPA, which adds remaining bits to support provisioning clusters running Flatcar (#1979). My work involves:

• Adding bucket field to AWSClusterSpec. If enabled property is set, S3 bucket will be created with cluster name as a name, where Ignition data will be stored.
  On cluster removal, bucket will be removed. This might be further extended to allow pre-existing buckets and to skip the cleanup.
• Adding ignition field next to cloudInit field in AWSMachineSpec. If ignition.enabled: true is set OR we detect, that CABPK produced bootstrap data in ignition format (new feature added in ✨ Add support for generating bootstrap data in Ignition format to CABPK cluster-api#4172), then different code-path is taken, which places bootstrap Ignition data in S3 and generates small Ignition snippet which will be used as a user data for created EC2 instance.
  When machine finishes provisioning, user data is removed from S3, similar to how Secret Manager handling is done.
• New cluster template is added, which contains Flatcar-specific configuration (AMIs, AWS-specific systemd configuration etc.)

I'd really appreciate some feedback on this suggestion. I'm right now finishing this work, cleaning up, I'll try to also add IAM support, some tests etc before I can open the PR.

[JoelSpeed]

I certainly like the sound of this update, the small ignition snippet style and loading from S3 is certainly a common pattern so makes sense to keep to that.

Ignition data in S3 and generates small Ignition snippet which will be used as a user data for created EC2 instance.

Out of interest, can we account here for generating a v2 vs v3 ignition snippet? Have you thought about this already?

Another suggestion I would have is to try and see what other providers can do in this area, perhaps there's some discussion that could be had more centrally about this so we establish a common pattern across the different clouds.

[invidian]

Out of interest, can we account here for generating a v2 vs v3 ignition snippet? Have you thought about this already?

Yes, similar to CABPK, I use v2 Ignition for Flatcar by default, with easy to add v3 support.

Another suggestion I would have is to try and see what other providers can do in this area, perhaps there's some discussion that could be had more centrally about this so we establish a common pattern across the different clouds.

I think other cloud providers will be simpler to implement. E.g. Azure and GCP puts userdata directly in instance metadata, without any sort of middleware service. I'd expect them to work out of the box (or almost) with proposed CABPK changes. Only AWS is special due to userdata size limitation.

[invidian]

Created #2271 with initial implementation to resolve #1979 and possibly this issue.

[randomvariable]

/close
in favour of kubernetes-sigs/cluster-api#5294

[k8s-ci-robot]

@randomvariable: Closing this issue.

In response to this:

/close
in favour of kubernetes-sigs/cluster-api#5294

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.