default_tags always shows an update #18311
Comments
github-actions
bot
added
the
needs-triage
anGie44
added
thinking
and removed
needs-triage
|
Hi @acdha, thank you raising this issue! do you mind providing additional configuration details regarding the |
anGie44
added
the
waiting-response
Basically if nothing else has changed, Terraform will show no changes and naturally no diff. If any other resource has changed when it triggers the plan display for that resource it will also include the resources listed above. The subnet configuration is here: My invocation of that is somewhat complicated: module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 2.69"
name = local.deployment_id
tags = local.tags
cidr = local.vpc_cidr_allocation
azs = slice(data.aws_availability_zones.available.names, 0, 4)
public_subnets = cidrsubnets(local.public_cidr, 2, 2, 2, 2)
private_subnets = cidrsubnets(local.private_cidr, 2, 2, 2, 2)
enable_nat_gateway = true
single_nat_gateway = false
one_nat_gateway_per_az = true
enable_dns_hostnames = true
nat_eip_tags = { Name = "${local.deployment_id} NAT Gateway" }
igw_tags = { Name = "${local.deployment_id} Public Internet Gateway" }
# We won't use the default network ACL but we want to make sure that it has
# a safe configuration by default just in case new resources are launched in
# it for any reason:
manage_default_network_acl = true
default_network_acl_name = "${local.deployment_id}-default"
# Leaving these as empty lists will result in no rules being applied, leaving
# the AWS default deny-all rules:
default_network_acl_ingress = []
default_network_acl_egress = []
public_dedicated_network_acl = true
private_dedicated_network_acl = true
database_dedicated_network_acl = true
public_inbound_acl_rules = concat(
…
)
public_outbound_acl_rules = concat(
…
)
private_inbound_acl_rules = concat(
…
)
private_outbound_acl_rules = concat(
…
)
enable_s3_endpoint = true
enable_efs_endpoint = true
enable_ses_endpoint = true
efs_endpoint_private_dns_enabled = true
efs_endpoint_security_group_ids = [aws_security_group.vpc_endpoints.id]
ses_endpoint_private_dns_enabled = true
ses_endpoint_security_group_ids = [aws_security_group.vpc_endpoints.id]
ses_endpoint_subnet_ids = data.aws_subnet_ids.private_subnets.ids
}The resulting resources look like this: resource "aws_subnet" "public" {
arn = "arn:aws:ec2:us-east-1:…:subnet/subnet-…"
assign_ipv6_address_on_creation = false
availability_zone = "us-east-1d"
availability_zone_id = "use1-az1"
cidr_block = "…/26"
id = "subnet-…"
map_customer_owned_ip_on_launch = false
map_public_ip_on_launch = true
owner_id = "…"
tags = {
"Name" = "…-public-us-east-1d"
}
tags_all = {
"Environment" = "Development"
"Name" = "…-public-us-east-1d"
"Project" = "…"
}
vpc_id = "vpc-…"
}I notice this a lot because of #14892 — this project also has some IAM policies which are defined using the ARNs of an ECS cluster and that means that those always show an update in-place even though all of the information is available at plan time. # module.collection_access.data.aws_iam_policy_document.ecs_update_service will be read during apply
# (config refers to values not yet known)
<= data "aws_iam_policy_document" "ecs_update_service" {
~ id = "83089849" -> (known after apply)
~ json = jsonencode(
{
- Statement = [
- {
- Action = "ecs:UpdateService"
- Effect = "Allow"
- Resource = "arn:aws:ecs:us-east-1:…:service/…/collection-access"
- Sid = ""
},
]
- Version = "2012-10-17"
}
) -> (known after apply)
- version = "2012-10-17" -> null
~ statement {
- effect = "Allow" -> null
- not_actions = [] -> null
- not_resources = [] -> null
# (2 unchanged attributes hidden)
}
}
# module.collection_access.aws_iam_policy.ecs_update_service will be updated in-place
~ resource "aws_iam_policy" "ecs_update_service" {
id = "arn:aws:iam::630942203890:policy/…-ECS-updatecollection-access-service"
name = "…-ECS-updatecollection-access-service"
~ policy = jsonencode(
{
- Statement = [
- {
- Action = "ecs:UpdateService"
- Effect = "Allow"
- Resource = "arn:aws:ecs:us-east-1:630942203890:service/…/collection-access"
- Sid = ""
},
]
- Version = "2012-10-17"
}
) -> (known after apply)
# (2 unchanged attributes hidden)
} |
ghost
removed
the
waiting-response
|
Hi @acdha 👋 Does the difference disappear if you no longer declare # main.tf
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "3.36.0"
}
}
required_version = ">= 0.14.10"
}
locals {
tags = {
sometagkey = "sometagvalue"
}
}
provider "aws" {
region = "us-east-2"
default_tags {
tags = local.tags
}
}
module "test" {
source = "./mod"
providers = {
aws = aws
}
name = "defaulttagstest"
tags = local.tags
}
# mod/main.tf
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.33.0"
}
}
required_version = ">= 0.14.10"
}
variable "name" {
description = "Name to be used on all the resources as identifier"
type = string
default = ""
}
variable "tags" {
description = "A map of tags to add to all resources"
type = map(string)
default = {}
}
resource "aws_vpc" "this" {
cidr_block = "10.0.0.0/16"
tags = merge(
{
"Name" = format("%s", var.name)
},
var.tags,
)
}
resource "aws_subnet" "this" {
cidr_block = cidrsubnet(aws_vpc.this.cidr_block, 8, 0)
vpc_id = aws_vpc.this.id
tags = merge(
{
"Name" = format("%s", var.name)
},
var.tags,
)
}Applying twice: The plan difference here is expected in this case as provider-level default tags are removed from the underlying resource |
|
I think that's correct. If I remove the tags option, I see diffs to remove the tags from resources which I guess do not yet support the provider default tags? The list notably does not include the |
|
Correct, the most recent version only includes |
|
That makes sense — it's a bit messy during the transition period but since what I already works the easiest thing is just to leave it alone until |
|
I'm inclined to say this can be closed with 3.38.0 since the support for all of the resources types allowed me to make a nice de-boilerplate commit. The only wart I ran into was |
|
The lack of support for volume_tags makes this a pretty painful problem for us. We currently use the same variable passed in most modules to populate tags and volume_tags, so if we want volume_tags to be populated we will have overlapping tags/tags_all which causes this messy output. Without a ton of module rewrites this means we cannot use default_tags at all. Either this bug or the fact that volume_tags doesn't inherit default_tags really should really be addressed. The benefits of default_tags is huge, but it will be very painful for us to adopt if one of those two bugs isn't addressed to ease the transition. We cannot have this constant messy plan output (the json output of 'unknown' values is the real pain point for us, it has even exposed secrets), and we cannot have volume_tags not populated, so we are stuck not using default_tags at all. |
I took that rewrite path which wasn't especially terrible for a small project but it meant that I had to keep passing that variable around. It would have been really nice to have a way to get the default_tags attribute from the provider so you could write that code generically. |
|
Still a major annoyance in aws v3.45.0 - I am surprised this is not marked as a bug? affected resources (so far):
|
|
If I understand the situation correctly, this is caused by having a tag's name match in the default_tag and tag collections. This is something that is easily fixed if you own the module but might be impossible if you are making use of other people's modules. This should be something that shows up when validate is run as an info level message, just to let people know. |
|
Well default-tags should be the first layer, any other tags being set should act as an override to a default-tag of the same name-key, and if the value is the same and set already, should not be displayed as a change (where it currently thinks a change is needed). If the default-tag is being overridden by an explicitly set tag, then yes show an info message of some sort. The problem is enhanced in production environments where TF is claiming a change is to be made, but no changes occur - this is the scary part as the larger your project folder grows, the more potential noise TF shows as "changing" making it harder to see potential mistakes. |
|
Some situation here. |
|
@olenm observed:
on the surface what @olenm suggests feels like the most desirable behavior to me -- the implicit equivalent of and also:
indeed, this is what we're struggling with at the moment. a bunch of reusable inner library modules, composed together by app engineering teams in root modules to build their own unique, opinionated infrastructures. we're in the transition phase of getting everyone on provider >~3.40, using maybe worse, if i understand correctly, this isn't just noise. TF will call the AWS API and perform a CRUD action for every affected resource, despite the fact that nothing has changed. Yes, the modify should end up as a noop, but the behavior wastes network bandwidth, consumes API quota, slows down the apply phase, and increases the chances of something going wrong. this contrived example below illustrates the issue, though IRL the resource-level tags might be managed within a module. the EC2 API is called with a modify each time plan/apply is executed, though nothing has changed. |
|
Is this being looked over? Absolutely annoying issue which is stopping me using default_tags... |
|
Still seeing this with aws provider v3.60.0 and Terraform 0.15.3 for resources:
Even when I configure the provider to ignore all the tag keys, e.g.: locals {
tags = {
foo = "bar"
baz = "bop"
}
}
provider "aws" {
ignore_tags {
keys = ["foo", "baz"]
}
}
resource "aws_ssm_parameter" "this" {
...
tags = local.tags
}the plan still shows this: |
|
Ok there is no changes, but still it's unusable in production. |
|
Just found this bug when attempting to manage aws_iam_user. This is the entirety of the terraform config (for this set of resources, see notes at the bottom): terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
backend "s3" {
# The S3 bucket where state-files are kept
bucket = "terraform-statefiles.example.com"
# DynamoDB tables where the lock is kept
key = "live/global/iam/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-locks"
encrypt = true
}
}
provider "aws" {
region = "us-east-1"
default_tags {
tags = {
created_by = "me"
created_tool = "terraform"
created_date = formatdate("YYYY-MM-DD", timestamp())
team = "..."
owner = "team-..."
}
}
ignore_tags {
keys = ["created_date"]
}
}
### This is the account alias for the whole account
resource "aws_iam_account_alias" "alias" {
account_alias = "...example..."
}
resource "aws_iam_user" "root_iam_user" {
name = "root_iam_user"
}
### IAM group for the admins
resource "aws_iam_group" "root_accounts" {
name = "root_accounts"
}
resource "aws_iam_group_membership" "group_root_accounts_membership" {
name = "group_root_accounts_membership"
users = [
aws_iam_user.root_iam_user.name,
]
group = aws_iam_group.root_accounts.name
}
resource "aws_iam_group_policy_attachment" "admin_access" {
group = aws_iam_group.root_accounts.name
policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}
resource "aws_iam_group_policy_attachment" "billing_access" {
group = aws_iam_group.root_accounts.name
policy_arn = "arn:aws:iam::aws:policy/job-function/Billing"
}
Gives output: And this persists even with Notes about my environment:
Update: Update 2:
So it appears that the first time I setup with default_tags / ignore_tags, it works as intended, and repeated times show the anomalous behavior. |
|
This does not just Show with ECS but Multiple different services , From ECS to lb and rds dbs |
|
I think I found a workaround. I kind of bypassed this issue by creating a new aws "provider" without the default-tags, and tagging only the bugged resources manually using a "locals" definition.
I wanted to create a variable "tags" with a map inside, but you can't create variables within a variable so I defined a "locals" instead:
Tag only the bugged resources. In my case it happened with "aws_s3_bucket". Use the provider "no-default-tags" and merge the tags you want with local.tags.
PD: You can find more examples of how to use the default-tags on this repo: https://github.com/rocketinsights/terraform-blog-default-tags and this video https://www.youtube.com/watch?v=F9eKnx3l8KA
|
|
Great work @onisim-iacob. I've been waiting for the workaround so much! |
|
To my mind, it's not a workaround. It worked like this before |
|
You can try @2rs2ts solution, but in my opinion, it is hard to understand. In my case I had only a couple of resources bugged and this "worked around" the bug. In all cases, you have to edit the TF code unless the AWS provisioner will get an official fix. |
|
I'm willing to fix this by fixing the Terraform AWS provider myself. Could any of the maintainers of this project give me even the slightest hint as to how this problem is happening or what code I can fix? With a little help, I'll try to solve it myself. It seems now is the time to narrow down the problem by having a constructive conversation rather than asking the project maintainer to solve it. |
|
@morremeyer you are stating that
But what can really help? The issue is almost 2 years old and you have plenty of thumbs up Do not use "default_tags" - is not an answer and I can't accept it as a permanent solution It doesn't make any sense, really |
|
@mossad-zika If you urgently need a solution, please talk to the Hashicorp support directly about prioritizing this. |
|
I don't want to add a "+1" here - but I've also found that this seems to affect |
|
i can confirm that updating terraform to version 1.4.6 solves the issue described above. Although before the update i realized that all resources were affected not only vpc and subnet. |
|
@johnsonaj by chance, is it going to be back-ported to |
|
This functionality has been released in v5.0.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you! |
|
@johnsonaj i have pinned my versions to the latest provider version 5.0.1 however i could still see updates for the tags even though there were not any changes to the module . |
|
@neelaruban just checking, since I was about to go make changes to update my provider version and I saw your message... do you see the update on subsequent applies? Maybe it's just a one-time diff and once you apply with the 5.x provider it will stop being a permadiff? I'd like to hear if it's still a problem for you before I go do the work to upgrade my repos. 😅 |
@neelaruban thanks for the update. I have tried a few different configurations and I am unable to produce perpetual diffs. Would it be possible to submit a new issue, with an example of your configuration? It is important to note the following about this update to It is still not possible to have The following configuration is possible: provider "aws" {
region = "us-west-2"
default_tags {
tags = {
created_at = "placeholder time"
}
}
}
resource "aws_s3_bucket" "example" {
bucket = "example-bucket"
tags = {
created_at = timestamp()
}
}However, the follow will still not compute correctly because provider "aws" {
region = "us-west-2"
default_tags {
tags = {
created_at = timestamp()
}
}
}
resource "aws_s3_bucket" "example" {
bucket = "example-bucket"
tags = {
additional_tag = "value
}
} |
I had to apply once and then on subsequent plan/apply runs I have seen no changes. I'm operating under the assumption that the state needed to be updated once to accurately calculate diffs on followup runs? |
Seems so, also had to apply once after update to provider v5.x and then all next runs were without diffs. |
|
@johnsonaj I am not using a dynamic value as you have shown for the default tags at the provider level , to give it a more context this how my provider configurations look at the moment terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "5.0.0"
}
}
}
provider "aws" {
region = "af-south-1"
ignore_tags {
key_prefixes = ["VVVVVV","XXX","VVVVV","CCVVVV"]
}
}
another thing to note is that i am not running terraform apply to my plans , i am still running the plans to see if my changes to the pinning down the version works or not . However looking at the comments from @2rs2ts @TimothyLoyer and @nantiferov proves a point that we have to apply first then the subsequent runs will be without any updates . thanks all |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
and others
Description
I have been looking forward to the default tagging support and tested it on a project yesterday which uses https://github.com/terraform-aws-modules/terraform-aws-vpc/ — this immediately showed some tags on aws_vpc and aws_subnet resources as having been changed, but only if another unrelated change was also present.
Community Note
Terraform CLI and Terraform AWS Provider Version
Affected Resource(s)
Terraform Configuration Files
Debug Output
Expected Behavior
No changes would be displayed
Actual Behavior
If an unrelated resource triggers a diff, all of the subnet and VPC resources will show an update in-place diff showing the tags which are already present. Curiously, in my project it lists 5 tags which are present as having been changed but then display a “1 unchanged element hidden”
Steps to Reproduce
terraform applyReferences
The text was updated successfully, but these errors were encountered: