feat: Allow tagging on per-subnet basis

[raxod502-plaid]

Description

New variables public_subnet_tags_per_subnet, private_subnet_tags_per_subnet, etc. Optional, no effect if not specified. If specified, they are additional tags to apply to each respective public and private subnet.

Extend the ability to allow naming and tagging route tables, EIPs, and NAT Gateways on per-subnet basis as well, add variables for that.

Motivation and Context

Previously, it was only possible to customize arbitrary tags on a per-AZ basis, not a per-subnet basis. You could customize the Name tag on a per-subnet basis but not any other tag.

The VPCs in our environment have several different types of subnets. We would like to tag them differently, e.g. for cost attribution and reference in external Terraform code. Currently this is not possible: subnets can only be disambiguated by VPC, AZ, and visibility.

Breaking Changes

No breaking changes

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have executed pre-commit run -a on my pull request

This is currently running in production

[raxod502-plaid]

Testing in our infrastructure with:

Generating expected Terraform plan:

[raxod502-plaid]

This is now live in our infrastructure. I also added the ability to customize route table tagging.

[raxod502-plaid]

I've completed the rollout of the new tagging system throughout our infrastructure, using the code from this pull request in production. No problems uncovered.

[kahirokunn]

Awesome!
LGTM!

[jseiser]

I think this behavior is what we are looking for.

We wanted to be able to have 2 sets of private subnets.

1 for our EKS cluster's worker nodes, and 1 for some other random servers that still survive post EKS migration. So we could just target the server private subnets when deploying those specific EC2 Instances

[davepattie]

Ahh, I see now that tags per az is no use when you have more than one private subnet per az

[hpschry]

I'd love to see this merged, this would address our use case as well: We'll need individual tags per sub-net, e.g. to control the assignment of load balancers to specific sub-nets in an EKS scenario, which is controlled by sub-net tags.

[reoring]

Once this Pull Request is merged, I will use it right away

[KevinBrooke]

We have been using this branch in production for 3 weeks without any issues.

[davepattie]

@antonbabenko would you mind taking a look at this PR please, it's a really helpful and thorough addition, thanks

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[kahirokunn]

keep

[jseiser]

@raxod502-plaid

Any chance you can fix the conflicts, id really love to get this one merged in.

[raxod502-plaid]

There's not much point in fixing merge conflicts until we get confirmation from a maintainer that they are interested in merging the PR.

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[kahirokunn]

i need this PR.

[rarescosma]

We're hitting the same needs for our EKS setup at the moment: need to tag secondary subnets differently to tell karpenter to slowly start moving the nodes towards these subnets.

What are the plans for moving this PR forward?

[andrewmiskell]

I'm running into some of the same issues in our environment as well, needing to tag specific subnets differently due to different uses.

[netomin]

Hi, hitting the same needs here, it would be great to have this PR to move forward ;)

[kahirokunn]

I think so!

[raxod502-plaid]

Not stale. @bryantbiggs would you be able to disable stale-bot on this PR to save people some time?

[manuelmazzuola]

Hi, hitting the same needs here, please move this PR forward!!!!

[Smana]

tf-controller plan output:

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

To apply this plan, please merge this pull request.

[jandersen-plaid]

@bryantbiggs what needs to be done here to get this PR merged? If the conflicts are resolved will that be enough?

[PLeS207]

Hello team, any updates on when this PR will be merged?

[edas-smith]

This is something that would be really good to have, at the moment this is a bit of a pain

[prashanthabitat]

@raxod502-plaid can you maybe change reviewers?

[kahirokunn]

Let's go ahead!

[raxod502-plaid]

can you maybe change reviewers?

I have no control over how and if the repository maintainers choose to review this PR.

[manuelmazzuola]

@bryantbiggs @antonbabenko can you help us?

[bryantbiggs]

I personally do not see the use of this - it seems like an anti-pattern to me. What is the scenario where one subnet needs a particular tag, or set of tags, that is different from its neighboring subnet?

Also, this PR is doing more than what its stating - I'm not sure what the route table changes are intended to do or why they are valid here

[tunix]

I personally do not see the use of this - it seems like an anti-pattern to me. What is the scenario where one subnet needs a particular tag, or set of tags, that is different from its neighboring subnet?

Hi, in my case, I need to setup multiple EKS clusters inside a VPC. Each cluster look for certain tags to exist on subnets so that they can allocate IP's from the correct subnet.

[bryantbiggs]

Each cluster look for certain tags to exist on subnets so that they can allocate IP's from the correct subnet.

This sounds peculiar - why do you care which IPs a cluster uses? Regardless, you can pass the subnet IDs to the resource that is creating your nodes

[MarkSRobinson]

I use per-subnet tags to look up subnets to connect to ASGs.
Roughly my setup is

• Create VPC in one TF state repo
• Create ASG in another TF state repo

[tunix]

Each cluster look for certain tags to exist on subnets so that they can allocate IP's from the correct subnet.

This sounds peculiar - why do you care which IPs a cluster uses? Regardless, you can pass the subnet IDs to the resource that is creating your nodes

EKS requires 3 different tag sets:

"kubernetes.io/cluster/my-cluster" = "owned"
"kubernetes.io/role/elb"           = "1"
"kubernetes.io/role/internal-elb"  = "1"

So it's able to discover which subnets it works in as well as which subnet is intended for private/public ELB IP's. @MarkSRobinson's case looks alike mine.

If you pass the subnet IP's to EKS directly to create these tags, VPC code removes them.

[bryantbiggs]

Each cluster look for certain tags to exist on subnets so that they can allocate IP's from the correct subnet.

This sounds peculiar - why do you care which IPs a cluster uses? Regardless, you can pass the subnet IDs to the resource that is creating your nodes

EKS requires 3 different tag sets:

"kubernetes.io/cluster/my-cluster" = "owned"
"kubernetes.io/role/elb"           = "1"
"kubernetes.io/role/internal-elb"  = "1"

So it's able to discover which subnets it works in as well as which subnet is intended for private/public ELB IP's. @MarkSRobinson's case looks alike mine.

If you pass the subnet IP's to EKS directly to create these tags, VPC code removes them.

EKS does not require those tags - the AWS load balancer controller can use those tags to identify which subnets it should create public and/or private load balancers within, but again you can also provide the subnet IDs to the AWS LBC controller as well. Also, these are broadly applied across the public subnets or the private subnets, which this module already supports https://github.com/terraform-aws-modules/terraform-aws-eks/blob/098c6a86ca716dae74bd98974accc29f66178c43/examples/eks_managed_node_group/main.tf#L418-L424

"kubernetes.io/cluster/my-cluster" = "owned" is not applied to a subnet, but it is applied to the security group, which is handled either by EKS or by the EKS module https://github.com/terraform-aws-modules/terraform-aws-eks/blob/098c6a86ca716dae74bd98974accc29f66178c43/node_groups.tf#L220

[bryantbiggs]

• Create ASG in another TF state repo

Common pattern - this is what data sources are created to solve:

https://github.com/aws-ia/ecs-blueprints/blob/d6e737b062ae4b365e898523fc845c8f31e0db6c/terraform/ec2-examples/backend-service/main.tf#L104-L109

[raxod502-plaid]

I personally do not see the use of this - it seems like an anti-pattern to me. What is the scenario where one subnet needs a particular tag, or set of tags, that is different from its neighboring subnet?

Why would it be assumed that all subnets within a VPC will be identical? There are many obvious use cases where that is not true, for example if different subnets are used for different workload types, or have different security properties, and so on.

Also, this PR is doing more than what its stating

Not sure where that claim comes from, the PR desc pretty clearly says: "Extend the ability to allow naming and tagging route tables, EIPs, and NAT Gateways on per-subnet basis as well, add variables for that."

I'm not sure what the route table changes are intended to do or why they are valid here

Can you clarify? Seems pretty clear to me that the route table changes are intended to allow you to add tags to your route tables. What is your query about whether they are "valid"?

you can also provide the subnet IDs to the AWS LBC controller as well [...] this is what data sources are created to solve

This is what tags are created to solve. Sure, you can shoehorn every possible way you might want to filter subnets in a data source by stuffing them all into the Name tag - or you can manually list out subnet IDs in config files - but it is way better to use tags the way they are intended to be used.

For example, we have tags that can be used to identify the subset of subnets that run Kubernetes workloads, the subset that are used for data-plane versus specialized control-plane redundancy, tags that can be used to identify subnets used for different subsets of Kubernetes clusters appropriate to different use cases, etc etc.

It's really unclear to me why it would not be recommended to use tags to manage your resources, in any context.

[bryantbiggs]

I personally do not see the use of this - it seems like an anti-pattern to me. What is the scenario where one subnet needs a particular tag, or set of tags, that is different from its neighboring subnet?

Why would it be assumed that all subnets within a VPC will be identical? There are many obvious use cases where that is not true, for example if different subnets are used for different workload types, or have different security properties, and so on.

Also, this PR is doing more than what its stating

Not sure where that claim comes from, the PR desc pretty clearly says: "Extend the ability to allow naming and tagging route tables, EIPs, and NAT Gateways on per-subnet basis as well, add variables for that."

I'm not sure what the route table changes are intended to do or why they are valid here

Can you clarify? Seems pretty clear to me that the route table changes are intended to allow you to add tags to your route tables. What is your query about whether they are "valid"?

you can also provide the subnet IDs to the AWS LBC controller as well [...] this is what data sources are created to solve

This is what tags are created to solve. Sure, you can shoehorn every possible way you might want to filter subnets in a data source by stuffing them all into the Name tag - or you can manually list out subnet IDs in config files - but it is way better to use tags the way they are intended to be used.

For example, we have tags that can be used to identify the subset of subnets that run Kubernetes workloads, the subset that are used for data-plane versus specialized control-plane redundancy, tags that can be used to identify subnets used for different subsets of Kubernetes clusters appropriate to different use cases, etc etc.

It's really unclear to me why it would not be recommended to use tags to manage your resources, in any context.

Always new this was going to be a contentious one, but oof 😅

If I look at the title of your PR and compare it with the code changes, they are quite a bit different. So the first couple of comments that you replied to, that is what I am referring to. PR body's are not captured in the changelog or release notes, only the title

Why would it be assumed that all subnets within a VPC will be identical? There are many obvious use cases where that is not true, for example if different subnets are used for different workload types, or have different security properties, and so on.

I don't follow this one. This module has always worked off the notion of subnet *groups - where there are different subnet group types (public, private, intra, database, etc.), and you can have n-number of subnets within each group. Their integration and configuration is pretty much consistent across all subnets within a subnet group - so how can one treat subnets differently within the same subnet group *within this module? or better yet, what is the motivation for doing this (or trying to do this)?

This is what tags are created to solve. Sure, you can shoehorn every possible way you might want to filter subnets in a data source by stuffing them all into the Name tag - or you can manually list out subnet IDs in config files - but it is way better to use tags the way they are intended to be used.

This one is hard to debate since its subjective. Tags are great - I don't think anyone will debate that. And we have tags for the subnet *groups (i.e. - public, private, intra, etc.), so thats great too and aligns with the statement above about consistent integration and configuration across those things created/associated within a subnet *group. But here, it sounds like you are wanting to name each individual subnet differently - which sounds like we are treating them as pets, sort of. And that is the use case I am struggling to find validity in supporting. In some instances above, I think folks are trying to take a subnet group and further segment that - so lets say we take the subnets created within the private subnet group and break those up further into n-number of sub-groups within that private subnet group. And I suspect this pattern is due to the fact that we currently do not support a generic - "create as many different subnet groups of your choosing" - its only those pre-defined subnet group types that we support in this module today. This is something @antonbabenko and I have discussed in the past; whether or not we should add a generic "subnets" or "subnet_group" (naming TBD) that would allow users to stamp out as many additional subnet groups as they would like, and we'd just need to figure out how that integrates into the main module we have today (so that it feels like a nautral extension of the main module)

Personally, I do not believe this is the intended use case for tags. If you look around within AWS, you will find many concepts that relate back to a container, and the things that go into that container. We have subnet groups for groups of subnets, we have node groups for groups of nodes (EKS), we even have a generic resource group which is more of a meta object within AWS. You create a new group for a particular use case - but you don't create a generic group and try to treat them differently within that group. Not at least when there is an existing notion of a group that allows you to create this explicit segregation of resources. For example - you wouldn't create a bunch of IAM users and then try to mange which users get which permissions by looking up their tags (metadata) - you would add them into the appropriate, respective group. Likewise for Kubernetes nodes - there are numerous ways of managing a collection of compute resources as a collective amongst the much larger whole (cluster).

So that said - if there were a generic sub-module where users could create various different groups of subnets, does this issue become a moot point?

[bryantbiggs]

But here, it sounds like you are wanting to name each individual subnet differently

*not just name, but more generically - treat them differently via unique tags

[raxod502-plaid]

this pattern is due to the fact that we currently do not support a generic - "create as many different subnet groups of your choosing" - its only those pre-defined subnet group types that we support in this module today

Yes, that is the requirement. If you provided a way to create an arbitrary set of subnet groups then it would not be necessary to parameterize tags within a subnet group.

We aren't treating subnets as pets, we simply have more types of subnets than just "private" and "public" (plus the few other special cases in this module).

So that said - if there were a generic sub-module where users could create various different groups of subnets, does this issue become a moot point?

Yes, exactly.

[andrewmiskell]

I think a generic submodule to stamp out as many private/public sets of subnets I want each with it's own set of tags would make this whole thing moot, at least in my use-case.

In my case, I need to different sets of public subnets that have different tags. One tagged with Subnet_Type "public" and another tagged with Subnet_Type "sending" (our use case is very particular and somewhat edge case due to the custom software we use). Currently I'm using the redshift subnets feature with enable_public_redshift enabled in order to do this. Having a more generic way to create subnets without have to "repurpose" the pre-defined subnet types would be very helpful.

[erpel]

So this "what you're trying to do is covered by groups" argument is the first time I understood the resistance. To me the groups always seemed really type based rather than the main differentiation. This has me rethinking how we use the module in some way.

My use case mainly comes from a situation like this:
We have for example 5 subnets that are "private" in type. That's more than one per AZ - which is problematic in some services where only one subnet per AZ is accepted.

Using tags would be how I'd like to set which subnets of the type "private" should be used for EKS control plane (don't hold me to the exact example, I'm going off memory here).
The subnets are otherwise quite the same, the fact that they are separate subnets comes from some organizational history...

[bryantbiggs]

If I look at the recent issues/PRs, I think most of them can be traced back to - more subnet flexibility. Such as:

• Custom amount of NAT gateways #1065
• Using create_multiple_public_route_tables = true only creates an internet gateway route on one route table. #1087
• "Error in function call" when using neither private subnets nor NAT #944
• Pass already created VPC id  #1082
• Error when creating VPC without any private subnets and has NAT gateways #1068
• This module, doesnt integrate with the Network Firewall Module #978
• Enable all Az specific resources to have a {resource_type}_tags_per_az map #984
• Confusing DNS64 behaviour with public subnets #972
• fix: NAT gw route table associations shouldn't be created when enable_nat_gateway=false #1090
• feat: Custom NAT gateways #1066
• feat: Enable the creation of private NAT gateways #1072

(not including those that have since been closed - this should suffice to show the intent)