google_storage_bucket resources with inconsistent final plan against dynamic labels in version 4.40.0

[q-oscarng]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

tested in both v1.2.8 and v1.3.2, on linux_amd64

Affected Resource(s)

• google_storage_bucket

Terraform Configuration Files

terraform {
    required_version = "~>1.0"
    backend "local" {
        path = "/my/local/machine/path/terraform.tfstate"
    }
    required_providers {
        null = {
            version = "~>3.0"
        }
        random = {
            version = "~>3.0"
        }
        google = {
            // using an older version (e.g. 4.39.0) would work before
            version = "4.40.0"
        }
    }
}

resource "random_string" "this" {
    length = 3
    upper   = false
    numeric = false
    special = false
}

resource "null_resource" "this" {}

resource "google_storage_bucket" "bucket" {
  project      =  "my-google-project"
  name          = "onion-super-bucket"
  location      = "EU"
  force_destroy = true
  labels = {
    // either one of these is enough for it to fail
    "one" = "${null_resource.this.id}"
    "two" = "${random_string.this.result}"
  }
}

Debug Output

╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for google_storage_bucket.bucket to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an invalid new value for .labels: was null, but now cty.MapVal(map[string]cty.Value{"one":cty.StringVal("4696786782519982561"), "two":cty.StringVal("fud")}).
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

Panic Output

Expected Behavior

terraform apply will succeed in one-go

Actual Behavior

terraform apply failed with the error above. Re-running terraform apply however would work.

Steps to Reproduce

1. create a module and copy the code into a new file called main.tf.
2. Change the backend path and update the project name inside google_storage_bucket
3. terraform apply

Important Factoids

References

• b/275608065

[edwardmedia]

@q-oscarng can you share your debug logs for both apply? Are you aware of the name of bucket is global?

[q-oscarng]

Hi, the debug log is here: https://gist.github.com/q-oscarng/057a318d0e4dda2addf1dddafe15712d
I noticed that the first terraform apply's plan does not recognize changes on labels, while re-applying it does

As for the name of bucket I create, I believe it is globally unique in a single namespace shared across all cloud storage users, as suggested in Google Cloud Doc. Setting it to something generic (e.g. example-bucket) won't work in terraform.

[edwardmedia]

It looks like below func causes this issue. By removing this line, the issue goes away, but other issues could show up. @rileykarson how do you want to handle this?

https://github.com/hashicorp/terraform-provider-google/blob/main/google/resource_storage_bucket.go#L85

[prauc]

hi all,

as I am the initial contributor of this suppress func:

Seems for me like a terraform sdk bug itself, not a provider issue.

var and / or local references work fine, while "only" resource outputs seem to fail:

variable "foo" { 
    type = string
    default = "bar"  
}

locals {
    foo = "bar"
}

resource "google_storage_bucket" "bucket" {
  project      =  "project"
  name          = "bucket"
  location      = "EU"
  force_destroy = true
  labels = {
    "three" = "foo"
    "two"   = "${var.foo}"
    "one"   = "${local.foo}"
  }
}

[sosimon]

@prauc I'm not sure that's true, we ran into this bug with only locals and vars.

[bharathkkb]

@edwardmedia @rileykarson we seem to be seeing this in CI for the GCS module too. Any updates?

[anthonydurot]

@edwardmedia @rileykarson Hi, is there any updates on this please ?

Thank you very much ;)

[fmillevi]

Do you have any new regarding when the bug fix will be available please ?

Is there any workarround we might use ?

[rileykarson]

No updates at the moment, sorry! We're not actively addressing this atm, so removing stale assignments.

[mdesbordes]

Hello, are there any updates regarding this issue ? Thanks !

[Ameausoone]

We did an awful workaround, if someone needs this :

resource "google_storage_bucket" "bucket" {
  project                     = var.gcp_project_id
  name                        = var.bucket_name
  [...]

  lifecycle {
    ignore_changes = [
      # Labels are managed through gcloud directly due to a bug, see below.
      labels,
    ]
  }
}

# This is a workarround done because there's a bug on google provider since v4.44 which cause labels to not be created.
module "gcloud" {
  source                = "terraform-google-modules/gcloud/google"
  version               = "3.1.2"
  additional_components = ["gsutil"]

  # Here we use the module-defined "module_depends_on" instead of the Terraform buit-in because of this bug : https://github.com/terraform-google-modules/terraform-google-gcloud/issues/82
  module_depends_on = [
    google_storage_bucket.bucket
  ]
  platform = "linux"
  create_cmd_entrypoint  = "gsutil"
  destroy_cmd_entrypoint = "gsutil"
  create_cmd_body        = "label ch -l env:${var.env} ${google_storage_bucket.bucket.url}"
  destroy_cmd_body       = "label ch -l env  ${google_storage_bucket.bucket.url}"
}

[kunzese]

Can someone please try the provider in version 4.66.0 or newer and check if the issue is fixed?

https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.66.0

[q-oscarng]

I have tested the error example above and can confirm that it is working! Thanks for the fix @kunzese.

[BapRx]

The fix works for a resource creation but there's another related issue when trying to change one of these labels:

Reading required version from terraform file
Reading required version from constraint: ~> 1.5.0
Matched version: 1.5.1
Installing terraform at /home/baptiste/bin
Switched terraform to version "1.5.1" 

Initializing the backend...

Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...

Initializing provider plugins...
- Reusing previous version of hashicorp/google from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Using previously-installed hashicorp/google v4.70.0
- Using previously-installed hashicorp/random v3.5.1

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
module.gcs_buckets.random_id.bucket_suffix: Refreshing state... [id=kmc]
module.gcs_buckets.google_storage_bucket.buckets["import-configs"]: Refreshing state... [id=import-configs]
module.gcs_buckets.google_storage_bucket_iam_binding.viewers["import-configs"]: Refreshing state... [id=b/import-configs/roles/storage.objectViewer]
module.gcs_buckets.google_storage_bucket_iam_binding.admins["import-configs"]: Refreshing state... [id=b/import-configs/roles/storage.objectAdmin]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.gcs_buckets.google_storage_bucket.buckets["import-configs"] will be updated in-place
  ~ resource "google_storage_bucket" "buckets" {
        id                          = "import-configs"
      ~ labels                      = {
          - "created-at" = "20230626082130" -> null
          - "customer"   = "acme" -> null
          - "env"        = "preprod" -> null
          - "made-by"    = "terraform" -> null
          - "name"       = "import-configs" -> null
        }
        name                        = "import-configs"
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "created-at" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "customer" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "env" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "made-by" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "name" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform plans null values instead of the dynamically generated ones. The labels are static except for the timestamp:

merge({ created-at = formatdate("YYYYMMDDhhmmss", timestamp()) }, var.labels)

[kunzese]

@BapRx can you please provide a minimal Terraform setup to reproduce your issue? I tried it with my setup and the 4.70.0 version of this provider and i am able to add, update and remove labels without issues.

[BapRx]

Yes I faced the issue with the provider v4.70.0, here's a slightly simplified version of the code:

terraform {
  required_version = "~> 1.5.0" # Also tested with 1.4.x

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.0"
    }
  }
}

provider "google" {
  project = "project_id"
}

module "gcs_buckets" {
  source        = "terraform-google-modules/cloud-storage/google"
  version       = "~> 4.0"
  project_id    = "project_id"
  names         = ["import-configs"]
  location      = "EU"
  storage_class = "STANDARD"
  labels = merge({ created-at = formatdate("YYYYMMDDhhmmss", timestamp()) }, {
    customer = "acme"
    env      = "preprod"
    service  = "import-configs"
    made-by  = "terraform"
    }
  )
  bucket_policy_only = { ("import-configs") = false }
}

[BapRx]

With the plan, etc (adding the dynamic label to an existing map of static labels):

❯ terraform show | grep labels -A6
    labels                      = {
        "customer" = "acme"
        "env"      = "preprod"
        "made-by"  = "terraform"
        "name"     = "import-configs"
    }
❯ cat bucket.tf
module "gcs_buckets" {
  source           = "terraform-google-modules/cloud-storage/google"
  version          = "~> 4.0"
  project_id       = var.project_id
  names            = [var.bucket_name]
  set_admin_roles  = true
  admins           = var.members_readwrite
  viewers          = var.members_read
  set_viewer_roles = true
  location         = var.location
  storage_class    = var.storage_class
  labels           = merge({ created-at = formatdate("YYYYMMDDhhmmss", timestamp()) }, var.labels)

  bucket_policy_only = { (var.bucket_name) = false }
  lifecycle_rules    = var.lifecycle_rules
}
❯ terraform plan -var-file tfvars/preprod.tfvars
module.gcs_buckets.random_id.bucket_suffix: Refreshing state... [id=kmc]
module.gcs_buckets.google_storage_bucket.buckets["import-configs"]: Refreshing state... [id=import-configs]
module.gcs_buckets.google_storage_bucket_iam_binding.viewers["import-configs"]: Refreshing state... [id=b/import-configs/roles/storage.objectViewer]
module.gcs_buckets.google_storage_bucket_iam_binding.admins["import-configs"]: Refreshing state... [id=b/import-configs/roles/storage.objectAdmin]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.gcs_buckets.google_storage_bucket.buckets["import-configs"] will be updated in-place
  ~ resource "google_storage_bucket" "buckets" {
        id                          = "import-configs"
      ~ labels                      = {
          - "customer" = "acme" -> null
          - "env"      = "preprod" -> null
          - "made-by"  = "terraform" -> null
          - "name"     = "import-configs" -> null
        }
        name                        = "import-configs"
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
❯ tfa -var-file tfvars/preprod.tfvars
module.gcs_buckets.random_id.bucket_suffix: Refreshing state... [id=kmc]
module.gcs_buckets.google_storage_bucket.buckets["import-configs"]: Refreshing state... [id=import-configs]
module.gcs_buckets.google_storage_bucket_iam_binding.admins["import-configs"]: Refreshing state... [id=b/import-configs/roles/storage.objectAdmin]
module.gcs_buckets.google_storage_bucket_iam_binding.viewers["import-configs"]: Refreshing state... [id=b/import-configs/roles/storage.objectViewer]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.gcs_buckets.google_storage_bucket.buckets["import-configs"] will be updated in-place
  ~ resource "google_storage_bucket" "buckets" {
        id                          = "import-configs"
      ~ labels                      = {
          - "customer" = "acme" -> null
          - "env"      = "preprod" -> null
          - "made-by"  = "terraform" -> null
          - "name"     = "import-configs" -> null
        }
        name                        = "import-configs"
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "created-at" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "customer" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "env" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "made-by" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.gcs_buckets.google_storage_bucket.buckets["import-configs"] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google" produced an
│ invalid new value for .labels: new element "name" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

[kunzese]

I think it is not the merge nor the formatdate, it is the use of timestamp() itself. Even in the docs they advice against using timestamp() on a resource:

The result of this function will change every second, so using this function directly with resource attributes will cause a diff to be detected on every Terraform run. We do not recommend using this function in resource attributes, but in rare cases it can be used in conjunction with the ignore_changes lifecycle meta-argument to take the timestamp only on initial creation of the resource. For more stable time handling, see the Time Provider.

Other issue i found:

hashicorp/terraform#22461

But i do not know why Terraform fails so hard in this case.

[BapRx]

Thank you for looking into this! I'll switch to the safer plantimestamp() which doesn't cause any issue so far.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.