GCP Storage Permissions returns error on destroy

[lwmobeent]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.23

• provider.google v3.11.0

Affected Resource(s)

• google_storage_bucket

Terraform Configuration Files

resource "google_storage_bucket" "lacework_bucket" {
  count = var.existing_bucket_name != "" && var.audit_log ? 0 : 1

  name = "${var.prefix}-${var.project_id}-lacework-bucket"
}

resource "google_storage_bucket_iam_binding" "bucket_object_viewer" {
  bucket = var.existing_bucket_name != "" && var.audit_log ? var.existing_bucket_name : google_storage_bucket.lacework_bucket[0].name
  role = "roles/storage.objectViewer"
  members = ["serviceAccount:${google_service_account.service_account.email}"]
}

resource "google_storage_bucket_iam_binding" "legacy_bucket_owner" {
  count = var.existing_bucket_name != "" && var.audit_log ? 0 : 1

  bucket = google_storage_bucket.lacework_bucket[count.index].name
  role = "roles/storage.legacyBucketOwner"
  members = ["projectEditor:${var.project_id}", "projectOwner:${var.project_id}"]
}

resource "google_storage_bucket_iam_binding" "legacy_bucket_reader" {
  count = var.existing_bucket_name != "" && var.audit_log ? 0 : 1

  bucket = google_storage_bucket.lacework_bucket[count.index].name
  role = "roles/storage.legacyBucketReader"
  members = ["projectViewer:${var.project_id}"]
}

Debug Output

Error: Error reading Resource "storage bucket \"b/demo-test-project-mobeen-lacework-bucket\"" for IAM binding with role "roles/storage.legacyBucketReader": Error applying IAM policy for storage bucket "b/demo-test-project-mobeen-lacework-bucket": Error setting IAM policy for storage bucket "b/demo-test-project-mobeen-lacework-bucket": googleapi: Error 412: Precondition Failed, conditionNotMet

GCP Stackdriver Log

The only issue in the logs, this does not have any authenticationInfo, a similar log for the 'get' is always successful,

2020-03-10 11:52:30.220 PDT
Cloud Storage
get
demo-test-project-mobeen-lacework-bucket
PERMISSION_DENIED
{
 insertId: "1fc12cld3zu6"  
 logName: "projects/test-project-mobeen/logs/cloudaudit.googleapis.com%2Fdata_access"  
 protoPayload: {
  @type: "type.googleapis.com/google.cloud.audit.AuditLog"   
  authenticationInfo: {…}   
  authorizationInfo: [1]   
  methodName: "storage.buckets.get"   
  requestMetadata: {
   callerIp: "2002:a05:6834:22c1:b029:58:f789:f678"    
   callerSuppliedUserAgent: "Google-API-Java-Client Google-HTTP-Java-Client/1.26.0-SNAPSHOT (gzip)"    
   destinationAttributes: {…} 

Expected Behavior

Remove all of the bindings created on the bucket

Actual Behavior

Every time while destroying the binding policies, there is an error on at least one of them where it gives a 412 error as shown above, looking at the gcp logs there is only one issue that is also stated above. It works fine if you run terraform destroy a second time, but first time it always fails. It also sometimes fail on terraform apply but that is rare

[venkykuberan]

@lwmobeent I used your config to create and destroy the resources. Every time it worked fine and i don't see the errors you are facing. BTW, what role does your terraform service account have ?, mine have Storage Admin role. I tried with older versions of terraform and provider as well.

I am suspecting some concurrency issue, however i am not able to repro it. Please try setting the Storage Admin role for your terraform service account and also let me know what role you are currently setup for terraform ?

Terraform v0.12.23
+ provider.google v3.11.0

google_storage_bucket_iam_binding.legacy_bucket_reader[0]: Destroying... [id=b/test-venky-external-org-lacework-bucket/roles/storage.legacyBucketReader]
google_storage_bucket_iam_binding.legacy_bucket_owner[0]: Destroying... [id=b/test-venky-external-org-lacework-bucket/roles/storage.legacyBucketOwner]
google_storage_bucket_iam_binding.bucket_object_viewer: Destroying... [id=b/test-venky-external-org-lacework-bucket/roles/storage.objectViewer]
google_storage_bucket_iam_binding.legacy_bucket_reader[0]: Still destroying... [id=b/test-venky-external-org-lacework-bucket/roles/storage.legacyBucketReader, 10s elapsed]
google_storage_bucket_iam_binding.legacy_bucket_owner[0]: Still destroying... [id=b/test-venky-external-org-lacework-bucket/roles/storage.legacyBucketOwner, 10s elapsed]
google_storage_bucket_iam_binding.bucket_object_viewer: Still destroying... [id=b/test-venky-external-org-lacework-bucket/roles/storage.objectViewer, 10s elapsed]
google_storage_bucket_iam_binding.legacy_bucket_reader[0]: Destruction complete after 12s
google_storage_bucket_iam_binding.bucket_object_viewer: Destruction complete after 12s
google_storage_bucket_iam_binding.legacy_bucket_owner[0]: Destruction complete after 12s
google_storage_bucket.lacework_bucket[0]: Destroying... [id=test-venky-external-org-lacework-bucket]
google_storage_bucket.lacework_bucket[0]: Destruction complete after 1s

Destroy complete! Resources: 4 destroyed.

[lwmobeent]

The service account I am using with terraform has the storage admin role on the project level, also the complete template has other things as well so it might be conflicting with those if it's a concurrency issue? I can upload the complete template to google drive folder and share with you?

[venkykuberan]

Please share your config and debug log

[lwmobeent]

Please find the config file zipped in the folder below
https://drive.google.com/drive/folders/1S6OQiKDtSjmo7zfVoWQjAVV1Te5uvtnq?usp=sharing

[venkykuberan]

@lwmobeent I am able to repro the behavior with your whole template, I see its conflicting with google_storage_notification resource during the destroy. You can avoid the error by adding the iam_member resources (example given below) of storage as dependents to google_storage_notification resource so terraform can manage the sequence during destroy. I don't think that's the best solution, we may have some bug on the provider which requires to add some mutex functions. However this alternate fix will get you going while we dig into the issue further and work on it.

resource "google_storage_notification" "lacework_notification" {
  count = var.existing_bucket_name != "" && var.audit_log ? 0 : 1

  bucket = google_storage_bucket.lacework_bucket[count.index].name
  payload_format = "JSON_API_V1"
  topic = google_pubsub_topic.lacework_topic[count.index].name
  event_types = ["OBJECT_FINALIZE"]

  depends_on = [google_pubsub_topic_iam_member.topic_publisher,google_storage_bucket_iam_binding.legacy_bucket_owner, google_storage_bucket_iam_binding.legacy_bucket_reader, google_storage_bucket_iam_member.bucket_object_viewer] 
}

P.S. - I am keeping this issue open

[c2thorn]

Hey @lwmobeent, the fix mentioned by @venkykuberan is the way to go. google_storage_notification has a side effect of modifying the IAM policy. When Terraform goes to destroy that along with the IAM resources at once, you can end up in a race condition error. Currently, there isn't a way to control the destroy ordering other than depends_on.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!