Provider errors out on GCP SA key creation with "new value: Root resource was present, but now absent."

[skorobogatydmitry]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.0.11
on linux_amd64

Google Terraform provider: v4.43.0

Affected Resource(s)

• google_service_account_key

Terraform Configuration Files

Overall module content is as follows:

resource "random_string" "sa_name" {
  length  = 23 #
  lower   = true
  number  = true
  special = false
  upper   = false
  lifecycle {
    ignore_changes = [
      special,
      override_special
    ]
  }
}

resource "google_service_account" "sa" {
  account_id   = random_string.sa_name.result
  display_name = "<NAME>"
}

resource "google_service_account_key" "key" {
  service_account_id = google_service_account.sa.id
}

variable "params" {
  type = object({
    REDACTED
  })
}

locals {
  // GCP health check services: https://cloud.google.com/load-balancing/docs/health-checks#firewall_rules
  health_check_addrs = ["35.191.0.0/16", "209.85.152.0/22", "209.85.204.0/22", "130.211.0.0/22"]
}

terraform {
  required_version = "~> 1.0.11"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.43.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "4.43.0"
    }
    local = {
      source  = "hashicorp/local"
      version = "2.2.2"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.1.0"
    }
  }
  backend "gcs" {}
}

provider "google" {
  project     = var.params.gcp_project
  credentials = var.params.gcp_credentials
}

provider "google-beta" {
  project     = var.params.gcp_project
  credentials = var.params.gcp_credentials
}

output "service_account" {
  sensitive   = true
  value = {
    key   = google_service_account_key.key.private_key
    id    = google_service_account.sa.id
    email = google_service_account.sa.email
  }
}

I joint multiple files to a single and generalized it a bit.

Debug Output

https://pastebin.com/G23a7B07

This log is extracted from out logging system (TF_LOG=JSON -s output), some sensitive data is redacted, plus I removed flooding lines such Found data source type. Lines are from the message filed of JSON logs.

Panic Output

google_service_account.sa: Creation complete after 0s [id=projects/XXX/serviceAccounts/YYY@XXX.iam.gserviceaccount.com]
google_service_account_key.key: Creating...
Error: Provider produced inconsistent result after apply
When applying changes to google_service_account_key.key, provider
"provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected
new value: Root resource was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Expected Behavior

is SA and key to be created successfully.

Actual Behavior

terraform apply errors out with this error

Steps to Reproduce

It occurs sporadically in automation, so there it no good reproducer.
I checked GCP logs, doesn't report any errors accessing SA or its key. There are 2 operations: SA creation, Key creation.

References

• Provider errors out on GCP SA key creation with "new value: Root resource was present, but now absent." #12535 - same issue, but without debug logs

[edwardmedia]

@skorobogatydmitry thanks for sharing the log. From the log, I am not sure what happened. I noticed below error. Do you see if the service account was successfully created? I wonder if you could try by adding sleep between google_service_account and google_service_account_key?

Retry 404s for service account creation - googleapi: Error 404: Unknown service account, notFound

[skorobogatydmitry]

Yes, service account was successfully created. I added retries and the 2nd apply operation showed only 1 resource to create - the key.
Also, it showed SA as an existing resource:

google_service_account.backuper: Refreshing state... [id=projects/REDACTED/serviceAccounts/REDACTED@REDACTED.iam.gserviceaccount.com]

I will try to add sleep 1 between the two, but it's just workaround, right ?

[skorobogatydmitry]

Sleep 1 between SA and the key didn't help to overcome the issue.

[edwardmedia]

@skorobogatydmitry how much time did you add in sleep? In some resources, api returns done while they are still not available for use. Some need short time while others might need more time. This is the behavior at apis. Not much we can do at the provider level. Can you try adding enough time to see if that fixes the problem?

[edwardmedia]

@skorobogatydmitry is this still an issue?

[edwardmedia]

@skorobogatydmitry closing this assuming it is no longer an issue. Feel free to reopen if it is still an issue

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.