tag node root volumes properly

[errordeveloper]

see https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-instance-tag-root-volume/ for a potential work around to this

[JeremJR]

As it took me a bit of time to figure out how to do that with eksctl, here is an example

First create an IAM policy to allow tag creation

Cust-Ec2Tags-Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Then in your config file update your nodeGroups section with :

nodeGroups:
  - name: mynodegroup
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::<aws account id>:policy/Cust-Ec2Tags-Policy
[...]
    preBootstrapCommands:
      - |
        cat <<EOF > /usr/local/sbin/tag_ebs_volume.sh
        #!/bin/bash
        AWS_AVAIL_ZONE=\$(curl -S http://169.254.169.254/latest/meta-data/placement/availability-zone)
        AWS_REGION="\`echo \"\$AWS_AVAIL_ZONE\" | sed 's/[a-z]$//'\`"
        AWS_INSTANCE_ID=\$(curl -S http://169.254.169.254/latest/meta-data/instance-id)
        ROOT_VOLUME_IDS=\$(aws ec2 describe-instances --region \$AWS_REGION --instance-id \$AWS_INSTANCE_ID --output text --query Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId)
        while IFS=$'\t' read -r key val
        do
          aws ec2 create-tags --resources \$ROOT_VOLUME_IDS --region \$AWS_REGION --tags Key=\${key},Value="\${val}"
        done < <(aws ec2 describe-instances --region \$AWS_REGION --instance-id \${AWS_INSTANCE_ID} --query "Reservations[*].Instances[*].[Tags[*]]" --output text | grep ^prefix_)
        EOF
      - "chmod +x /usr/local/sbin/tag_ebs_volume.sh"
      - "/usr/local/sbin/tag_ebs_volume.sh"

Note that in this example it will propagate only tags with a specific prefix "prefix_"

Hope that may help someone

[log2]

Thank you @JeremJR , this helped a lot! 👍

[MartinEmrich]

Doing it from a script within the new instance won't be good enough for billing tags, as there is a short period where the volume exists, but is not tagged.

According to this https://forums.aws.amazon.com/thread.jspa?threadID=122354&start=25&tstart=0, There is a way to tag the volumes within an Auto Scaling Group via Launch Templates.

Apparently eksctl already uses launch templates, maybe all we need is a way to populate them with the tags.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[bitva77]

Just to confirm: It's the year 2021 and we have to manually add tags to EBS volumes mounted on to EC2 instances launched from EKS manually?

Like using that bash script above.

NOT COMPLAINING. just verifying :)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[bitva77]

oh I ain't letting this die just out of spite now

[vodwood]

The fact that this feature is still open and not prioritized is fascinating.

[liangxibing]

Will anyone take a look at this? We have some EBS volume enforce tagging policy.
So we can't use user data to add tags later but only when invoking EC2 RunInstances API.

[mcburn13]

Just want to piggyback on this; how is it in 2021 AWS doesn't allow a OU level SCP for tagging actually work because of major services that need to negate it? EBS volumes is one, and Fargate is another as from console it creates a CloudFormation stack that we require tags on , and it doesn't tag it. Come on AWS, stop creating 5000 new services per week and work on the EXISTING ONES. thanks in advance!

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[mike-stewart]

Can this be re-opened?

[Skarlso]

Apologies, the reason this wasn't prioritised is that, apparently, there was a workaround, however bad, and there were other things with higher priority awaiting fixes and features all around. It's difficult to balance these things properly. Eventually, we'll get to things though, it just takes time.

Other than that, we always accept contributions. :) It's possible that this can be fixed via the launch template but it still requires time and effort and testing and verifying that the proposed solution actually works, even after the volume was rotated out.

That said, we'll take a look at this and see if the launch template is a viable solution or not.

[Skarlso]

Actually, we already tag volumes with a specific tag through launch templates:

		}, gfnec2.LaunchTemplate_TagSpecification{
			ResourceType: gfnt.NewString("volume"),
			Tags:         cfnTags,

where cfnTags are

	cfnTags := []cloudformation.Tag{
		{
			Key:   gfnt.NewString("Name"),
			Value: gfnt.NewString(generateNodeName(ng, meta)),
		},
	}

I'm guessing you would like to insert custom tags in here?

[Skarlso]

Actually, actually reading more into this. All tags, set through Tags spec in the nodegroup are applied to the LaunchTemplate.

	for k, v := range ng.Tags {
		cfnTags = append(cfnTags, cloudformation.Tag{
			Key:   gfnt.NewString(k),
			Value: gfnt.NewString(v),
		})
	}

Which is then applied as the above. ng.Tags is coming from:

	// Applied to the Autoscaling Group and to the EC2 instances (unmanaged),
	// Applied to the EKS Nodegroup resource and to the EC2 instances (managed)
	// +optional
	Tags map[string]string `json:"tags,omitempty"`

Configuration:

managedNodeGroups:
  - name: managed-ng-1
    minSize: 2
    desiredCapacity: 2
    maxSize: 4
    labels:
      role: worker
    tags:
      whatever: billing

The whatever: billing will be applied to

	launchTemplateTagSpecs = append(launchTemplateTagSpecs,
		gfnec2.LaunchTemplate_TagSpecification{
			ResourceType: gfnt.NewString("instance"),
			Tags:         cfnTags,
		}, gfnec2.LaunchTemplate_TagSpecification{
			ResourceType: gfnt.NewString("volume"),
			Tags:         cfnTags,
		}, gfnec2.LaunchTemplate_TagSpecification{
			ResourceType: gfnt.NewString("network-interface"),
			Tags:         cfnTags,
		})

The instance, the volume and the network-interface. Please verify if this is satisfactory.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[MartinEmrich]

@Skarlso IDK about the original author of this issue, but for me, the tags are propagated to managed nodes and their volume, but not to the network interfaces.

[Skarlso]

@MartinEmrich Yep, tried it. I can see that it doesn't propagate. We'll take a look.

[Skarlso]

I can confirm that we are giving it the right value:

            {
              "ResourceType": "network-interface",
              "Tags": [
                {
                  "Key": "Name",
                  "Value": "test-tags-cluster-2-managed-ng-1-Node"
                },
                {
                  "Key": "whatever",
                  "Value": "billing"
                },
                {
                  "Key": "alpha.eksctl.io/nodegroup-name",
                  "Value": "managed-ng-1"
                },
                {
                  "Key": "alpha.eksctl.io/nodegroup-type",
                  "Value": "managed"
                }
              ]
            }

This might be an AWS issue. But I'll investigate further.

[Skarlso]

Oh freck.

The type of resource to tag. Currently, the resource types that support tagging on creation are instance and volume. To tag a resource after it has been created, see CreateTags.

Well.... :/

[Skarlso]

So this gets into the area now about tagging things after creation and out of the scope of the cloudformation stack which is that... Everything we do after creation will have the effect that it's temporary. eksctl will not monitor tags and won't reapply them. And since these resources are controlled by the ASG, or the autoscaler, new instances will not have this tag, since the launchtemplate doesn't support them.

Thus, if an instance is rotated out, that's it. The tags are lost. And we can't do anything about that I'm afraid until there is AWS support for it through the LaunchTemplate.

[Skarlso]

I raised this issue internally, let's see if there is any future regarding tagging ENIs at least.

[pcolmer]

I came across this issue because I'm trying to solve the challenge of parsing the Cost & Usage Report and figuring out which volume went with which EC2 instance.

Doing it from a script within the new instance won't be good enough for billing tags, as there is a short period where the volume exists, but is not tagged.

According to this https://forums.aws.amazon.com/thread.jspa?threadID=122354&start=25&tstart=0, There is a way to tag the volumes within an Auto Scaling Group via Launch Templates.

Apparently eksctl already uses launch templates, maybe all we need is a way to populate them with the tags.

Unfortunately that forum link doesn't work any more so I don't know what it said. Looking at other blog posts and the like, I can see that tags and their corresponding values can be set in the launch template.

But because they are fixed values, I don't see any way of setting a dynamic value like the EC2 instance ID.

Is there a solution to this?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.