Provider produced inconsistent final plan - tags_all errors without using default_tags block

[ogdowski]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.3
hashicorp/aws 3.33

hashicorp/aws 3.32 is working without issues

Affected Resource(s)

• aws_vpc

Terraform Configuration Files

We are using terraform-aws-vpc module in 2.77.0 version for our VPC resources

We are not using "default_tags" block in provider added in 3.33 release

Expected Behavior

terraform apply should work without issues

Actual Behavior

terraform apply produces these kind of errors:

Error: Provider produced inconsistent final plan
When expanding the plan for
module.main_vpc.module.vpc.aws_vpc.this[0] to
include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "Name" has appeared.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent final plan
When expanding the plan for
module.main_vpc.module.vpc.aws_vpc.this[0] to
include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "created_at" has appeared.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

...and more for other elements

Steps to Reproduce

1. Prepare module using terraform-aws-vpc module in 2.77.0 version for VPC
2. Set tags in it
3. Do not set "default_tags" block in the provider
4. Run terraform apply

[ewbankkit]

Similar(ish):

• default_tags always shows an update #18311

[mkielar]

We're experiencing this problem in a scenario where the value of resource tags is fetched from a null_data_source like this:

resource "aws_ssm_parameter" "is_core_deployed" {
   ... 
}

data "null_data_source" "tags" {
  inputs = {
    tags = jsonencode(aws_ssm_parameter.is_core_deployed.tags)
  }
  depends_on = [
    aws_ssm_parameter.is_core_deployed
  ]
}

resource "aws_vpc" "this" {
   ...
   tags = jsondecode(data.null_data_source.tags.outputs["tags"])
}

This is a trick that allowed us (with terraform 0.12.x) to have all resources in this template depend on the SSM Parameter which allows us to easily identify in our automation if given environment is still deployed.

[anGie44]

Hi @ogdowski , thank you for raising this issue. To clarify first, when you note that tags are configured, do you mean via both of the variables referenced in the module's vpc resource (ie. var.tags and var.vpc_tags) https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/master/main.tf#L40-L46 , or just one of the 2 variables are configured with additional tag data? In addition, at any point was the provider block configured with an internal "default_tags" block? or it was left unchanged before and after the upgrade to v3.33.0 of the Terraform AWS Provider?

[ogdowski]

Hello @anGie44, thank you for working on this issue. We were adding just tags in the module and we were not using default_tags in provider after updating to 3.33.

Like @mkielar said - we are suspecting that the reason of this might be dynamic tag values fetched from null_data_source, which are known only during apply.

[mkielar]

I've started refactoring this, and now I have this on an AutoScaling Group...
The terraform plan produces this:

      + tags                      = [
          + {
              + "key"                 = "Bastion"
              + "propagate_at_launch" = "true"
              + "value"               = "true"
            },
          + {
              + "key"                 = "environment"
              + "propagate_at_launch" = "true"
              + "value"               = "dev-i1109"
            },
          + {
              + "key"                 = "pci_scope"
              + "propagate_at_launch" = "true"
              + "value"               = "none"
            },
          + {
              + "key"                 = "project"
              + "propagate_at_launch" = "true"
              + "value"               = "my_application"
            },
          + {
              + "key"                 = "terraform"
              + "propagate_at_launch" = "true"
              + "value"               = "true"
            },
          + {
              + "key"                 = "workspace"
              + "propagate_at_launch" = "true"
              + "value"               = "i1109"
            },
          + (known after apply),
        ]

As you can see there's one tag that says known after apply. This tag is an ARN of a SSM Parameter that needs to get deployed and it's ID is only known after apply.

Terraform fails to deploy this ASG with:

Error: Provider produced inconsistent final plan
When expanding the plan for
module.core_infrastructure.module.bastion.aws_autoscaling_group.bastion_asg[0]
to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags:
length changed from 7 to 8.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Versions:

Terraform 0.13.3
- Using hashicorp/random v3.1.0 from the shared cache directory
- Using hashicorp/template v2.2.0 from the shared cache directory
- Using hashicorp/null v3.1.0 from the shared cache directory
- Using hashicorp/aws v3.36.0 from the shared cache directory
- Using cloudflare/cloudflare v2.19.2 from the shared cache directory

[mkielar]

Potentially related:

• Provider produced inconsistent final plan #7987
• Error: aws_autoscaling_group - Provider produced inconsistent final plan #10297
• [aws_autoscaling_group] Provider produced inconsistent final plan (for tags) #14085 (mostly this one)

[anGie44]

Thank you for all the great feedback @ogdowski @mkielar ! We've gotten to the bottom of the issue (way in which we're computing the diff for the computed attribute tags_all) and the fix will be released with the next version of the provider.

[ghost]

This has been released in version 3.38.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[mkielar]

@anGie44 Looks like it's back in 3.38...
We're getting this on SSM Parameter now:

Error: Provider produced inconsistent final plan
When expanding the plan for aws_ssm_parameter.is_core_deployed to include new
values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "created_at" has appeared.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

The SSM Parameter mentioned in the error is deployed like this:

locals {
  tags = {
    terraform   = "true"
    environment = local.environment
    workspace   = terraform.workspace
    project     = var.core_project_name
  }
}

resource "aws_ssm_parameter" "is_core_deployed" {
  name        = "${local.environment}-core-deployment"
  description = "Foo"
  value = jsonencode({
    "description"     = "Do not modify."
    "success"         = false
    "updated_by/cicd" = false
  })
  type = "String"
  tags = merge(local.tags, {
    "pci_scope"  = "none"
    "created_at" = timestamp()
    "updated_at" = timestamp()
  })

  overwrite = true
  lifecycle {
    ignore_changes = [
      tags["created_at"],
      value,
    ]
  }
}

The error only appears when we terraform updates already existing parameter. When we delete the resource first, and then let terraform create it from scratch, it works. Something is seriously wrong.

We're still not using default_tags at all.

[mkielar]

Using 3.37 made it work for us, so the problem is definitely in 3.38. Can you please reopen the ticket @ewbankkit @anGie44, or would you rather have me open a new one? IMO this is exactly the same issue as before, except on a new resource.

[anGie44]

Thank you for raising this issue again @mkielar! I'm able to reproduce and am seeing the this type of output on plan

Terraform will perform the following actions:

  # aws_ssm_parameter.is_core_deployed will be updated in-place
  ~ resource "aws_ssm_parameter" "is_core_deployed" {
        id          = "test-core-deployment"
        name        = "test-core-deployment"
      ~ tags        = {
          - "created_at"  = "2021-05-05T15:57:11Z"
          - "environment" = "test"
          - "pci_scope"   = "none"
          - "project"     = "test"
          - "terraform"   = "true"
          - "updated_at"  = "2021-05-05T15:59:19Z"
          - "workspace"   = "tf-providers"
        } -> (known after apply)
      ~ tags_all    = {
          - "created_at"  = "2021-05-05T15:57:11Z" -> null
          - "environment" = "test" -> null
          - "pci_scope"   = "none" -> null
          - "project"     = "test" -> null
          - "terraform"   = "true" -> null
          - "updated_at"  = "2021-05-05T15:59:19Z" -> null
          - "workspace"   = "tf-providers" -> null
        }
        # (8 unchanged attributes hidden)
    }

In this case, I would of expected the tags_all diff to be similar to the tags one with (known after apply), but this seems to be a consequence of using a computed tag with timestamp().
Applying these changes outputs inconsistent final plan for several of the tags defined in local.tags, did you encounter something similar as well?

[mkielar]

Yes, I'm seeing a flare of errors, one for each tag. Just pasted one of them in the reopening comment for brevity, but there's more than just one.

[anGie44]

Interesting, so I've played around with that config more and it seems to actually be triggered by the lifecycle ignore_changes reference to the tags["created_at"]..if I comment that line out, the apply runs without the previous errors and the diff for tags_all appropriately shows (known after apply) 🤔

[ghost]

This has been released in version 3.39.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!