Early Boot Config, apiclient route for elastic load balancer

[peteristhegreat]

What I'd like:

A route in the early boot config and apiclient for the elastic load balancer api.

(Similar to Amazon EC2 - IMDSv1 HTTP API, but for the elastic load balancer api.)
https://github.com/bottlerocket-os/bottlerocket/blob/develop/README.md#amazon-ecs-settings

Here is a snippet from a user-data bash script coming from CloudFormation for another linux box that we are porting to bottlerocket:

        #!/bin/bash
        INSTANCE_ID=$(/usr/bin/ec2-metadata -i | /usr/bin/cut -d ' ' -f 2)
        /usr/bin/aws --region ${AWS::Region} elbv2 register-targets \
                --target-group-arn ${MyALBTargetGroup} \
                --targets Id=$INSTANCE_ID

It doesn't look like there is an equivalent aws agent for elbv2, so just a direct call to the AWS API could do it. (either the rest route or thru AWS CLI).

Proposed way to do this in early boot config via the user-data toml file (going thru CloudFormation):

       UserData: !Base64
         'Fn::Sub': |
          [settings.kubernetes]
          cluster-name = "${MyClusterName}"
          api-server = "${Cluster.Endpoint}"
          cluster-certificate = "${Cluster.CertificateAuthorityData}"
          [settings.elbv2.register-targets]
          "${MyALBTargetGroup}" = "INSTANCE_ID"

If there is a better way to imply this instance-id instead of INSTANCE_ID... that would work, too.

And this does the a call to the api below (similar to the aws cli call above):
https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_RegisterTargets.html

Related to:

https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/api/ecs-settings-applier/src/ecs.rs

Any alternatives you've considered:

https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html

[webern]

Thank you for opening your first Bottlerocket issue and providing links and context.

The way we expect users to interact with things like the AWS CLI (or SDKs) would be via a container. In this case, it sounds like something that should happen early in the boot so we'd suggest using a bootstrap container that contains the desired logic. If you want to go this route, let us know if you need any assistance.

[peteristhegreat]

So we made progress on a workaround with a awscli+alpine image and using the bootstrap-containers setting. Here goes:

In ECR we saved a new alpine image with the following:

FROM alpine
RUN apk add aws-cli cloud-utils-ec2metadata
ENTRYPOINT ["sh", "/.bottlerocket/bootstrap-containers/current/user-data"]

Our new CloudFormation portion now looks like this:

        userData: !Base64
          'Fn::Join':
            - ''
            - - 'Fn::Sub': |-
                  [settings.kubernetes]
                  cluster-name = "${MyStackName}"
                  api-server = "${Cluster.Endpoint}"
                  cluster-certificate = "${Cluster.CertificateAuthorityData}"
                  [settings.bootstrap-containers.mybootstrap-container]
                  source = "${AWS::Account}.dkr.ecr.${AWS::Region}.amazonaws.com/my-awscli-alpine:latest"
                  mode = "once"
                  essential = true
                  user-data = "
              - 'Fn::Base64':
                  'Fn::Sub': |
                    #!/bin/sh
                    INSTANCE_ID=$(/usr/bin/ec2metadata --instance-id)
                    /usr/bin/aws --region ${AWS::Region} elbv2 register-targets \
                            --target-group-arn ${MyALBTargetGroup1} \
                            --targets Id=$INSTANCE_ID
              - '"'

Getting the Fn::Sub in the right place with the doubley nested Fn::Base64 was interesting.

The documentation in the comments at the top of this file was very helpful:

https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/api/bootstrap-containers/src/main.rs

Also note that the default version of awscli for alpine installs

aws-cli/1.18.177 Python/3.8.10 Linux/4.19.121-linuxkit botocore/1.19.47

And if you need AWS CLI v2, you need to install something like what is here: https://stackoverflow.com/a/65969295/999943

And then you would get:

aws-cli/2.2.4 Python/3.8.8 Linux/4.19.121-linuxkit exe/x86_64.alpine.3 prompt/off

Hope that helps someone in the future.