Weird crash in google_storage_object_acl

[marcb1]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

$ terraform version
Terraform v0.12.31
+ provider.aws v3.63.0
+ provider.google v3.88.0

Affected Resource(s)

google_storage_object_acl

Terraform Configuration Files

We have the following terraform code and we recently just started seeing a crash on terraform plan. This was working before (planning/applying), so I'm not sure why it suddenly started crashing. We are on the latest version of the terraform provider.

resource "google_storage_bucket_object" "static-sitemap" {
  name   = "static-sitemap.xml"
  source = "./static-sitemap.xml"
  bucket = "v-sitemaps"
}

resource "google_storage_object_acl" "static-sitemap-acl" {
  bucket = "v-sitemaps"
  object = google_storage_bucket_object.static-sitemap.output_name

  role_entity = [
    "READER:allUsers",
  ]
}

This is the output from the crash:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

google_storage_bucket_object.static-sitemap: Refreshing state... [id=v-sitemaps-static-sitemap.xml]
google_storage_object_acl.static-sitemap-acl: Refreshing state... [id=static-sitemap.xml-acl]

------------------------------------------------------------------------

Warning: Resource targeting is in effect

You are creating a plan with the -target option, which means that the result
of this plan may not represent all of the changes requested by the current
configuration.
                
The -target option is not for routine use, and is provided only for
exceptional situations such as recovering from errors or mistakes, or when
Terraform specifically suggests to use it as part of an error message.


Error: rpc error: code = Unavailable desc = transport is closing

panic: runtime error: invalid memory address or nil pointer dereference
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x27aaf4e]
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: 
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: goroutine 95 [running]:
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: github.com/hashicorp/terraform-provider-google/google.resourceStorageObjectAclDiff(0x35aa800, 0xc001409f80, 0xc001436480, 0x3050500, 0xc000f2c300, 0x35dec08, 0xc001418c00)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-google/google/resource_storage_object_acl.go:83 +0x2ee
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0xc000916ae0, 0x35aa800, 0xc001409f80, 0xc00141ca10, 0xc0009fa630, 0x31e38f8, 0x3050500, 0xc000f2c300, 0x44fa300, 0x29d5880, ...)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/schema.go:540 +0xb29
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0xc000919c00, 0x35aa800, 0xc001409f80, 0xc00141ca10, 0xc0009fa630, 0x3050500, 0xc000f2c300, 0x0, 0x0, 0x0)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource.go:506 +0xa5
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).PlanResourceChange(0xc0005ab410, 0x35aa800, 0xc001409f80, 0xc001410e60, 0xc001409f80, 0x2f96a40, 0xc0009f7800)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/grpc_provider.go:693 +0x7c5
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).PlanResourceChange(0xc000b8e440, 0x35aa8a8, 0xc001409f80, 0xc00141c930, 0xc000b8e440, 0xc0009f7830, 0xc00059eba0)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:315 +0xb5
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x2f96a40, 0xc000b8e440, 0x35aa8a8, 0xc0009f7830, 0xc001426840, 0x0, 0x35aa8a8, 0xc0009f7830, 0xc000f00c00, 0x17f)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:362 +0x214
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000501340, 0x35e8c58, 0xc000594780, 0xc000c005a0, 0xc000528a50, 0x44b59a8, 0x0, 0x0, 0x0)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.40.0/server.go:1297 +0x52b
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc000501340, 0x35e8c58, 0xc000594780, 0xc000c005a0, 0x0)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.40.0/server.go:1626 +0xd0c
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0004a4600, 0xc000501340, 0x35e8c58, 0xc000594780, 0xc000c005a0)
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.40.0/server.go:941 +0xab
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2021-10-18T12:23:41.899-0400 [DEBUG] plugin.terraform-provider-google_v3.88.0_x5:       /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.40.0/server.go:939 +0x1fd
2021/10/18 12:23:41 [ERROR] <root>: eval: *terraform.EvalDiff, err: rpc error: code = Unavailable desc = transport is closing
2021/10/18 12:23:41 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2021/10/18 12:23:41 [TRACE] [walkPlan] Exiting eval tree: google_storage_object_acl.static-sitemap-acl
2021-10-18T12:23:41.903-0400 [DEBUG] plugin: plugin process exited: path=/root/at/repos/Infrastructure/terraform/7133/default/google-projects/v-prod/.terraform/plugins/linux_amd64/terraform-provider-google_v3.88.0_x5 pid=168088 error="exit status 2"
2021/10/18 12:23:41 [TRACE] vertex "google_storage_object_acl.static-sitemap-acl": visit complete
2021/10/18 12:23:41 [TRACE] vertex "google_storage_object_acl.static-sitemap-acl": dynamic subgraph encountered errors
2021/10/18 12:23:41 [TRACE] vertex "google_storage_object_acl.static-sitemap-acl": visit complete
2021-10-18T12:23:41.903-0400 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021/10/18 12:23:41 [TRACE] dag/walk: upstream of "provider.google (close)" errored, so skipping
2021/10/18 12:23:41 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2021/10/18 12:23:41 [TRACE] dag/walk: upstream of "root" errored, so skipping
2021/10/18 12:23:41 [INFO] backend/local: plan operation completed
2021-10-18T12:23:42.103-0400 [DEBUG] plugin: plugin exited



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain 
sensitive information that must be redacted before it is safe to share 
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

The Only strange thing I'm seeing in the logs is

2021-10-18T12:15:44.415-0400 [INFO]  plugin.terraform-provider-google_v3.88.0_x5: 2021/10/18 12:15:44 [INFO] Instantiating Google Storage client for path https://storage.googleapis.com/storage/v1/: timestamp=2021-10-18T12:15:44.414-0400

I'm not sure if the URI is missing the bucket name.

Anyways, I have the terraform debug log here: https://gist.github.com/marcb1/977596ca35349e3e56ffc02eba29dea3#file-terraform-crash-debug-log

[marcb1]

Upon further digging (found this: https://stackoverflow.com/questions/55635097/how-do-i-find-the-owner-of-a-google-cloud-storage-object). The object does not have an owner entity set:

$ gsutil ls -L gs://v-sitemaps/static-sitemap.xml
gs://v-sitemaps/static-sitemap.xml:
    Creation time:          Tue, 23 Mar 2021 01:49:05 GMT
    Update time:            Tue, 23 Mar 2021 01:49:06 GMT
    Storage class:          STANDARD
    Content-Length:         5919
    Content-Type:           text/xml; charset=utf-8
    Hash (crc32c):          GzBErQ==
    Hash (md5):             WFzE7uCbb9tb/luUC7PDLQ==
    ETag:                   CK3MvJmmxe8CEAQ=
    Generation:             1616464145491501
    Metageneration:         4
    ACL:                    [
  {
    "entity": "allUsers",
    "role": "READER"
  }
]
TOTAL: 1 objects, 5919 bytes (5.78 KiB)
$ gsutil acl get gs://v-sitemaps/static-sitemap.xml
[
  {
    "entity": "allUsers",
    "role": "READER"
  }
]

I think the fix should be to check for nil when parsing the OWNER here and set it to an empty string:

terraform-provider-google/google/resource_storage_object_acl.go

Line 83 in e2e533f

objectOwner := sObject.Owner.Entity

[maroux]

Running into the same thing, except I do have an OWNER role on the object in question. So odd

[maroux]

│ Error: Plugin did not respond
│ 
│   with ...google_storage_object_acl.app_config_acl,
│   on ... line 7, in resource "google_storage_object_acl" "acl":
│    7: resource "google_storage_object_acl" "acl" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).PlanResourceChange call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-google_v4.6.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x32e73ee]

goroutine 168 [running]:
github.com/hashicorp/terraform-provider-google/google.resourceStorageObjectAclDiff(0x3fca580, 0xc000c445c0, 0xc000c44f80, 0x3a64b20, 0xc000041100, 0x3ff5e90, 0xc000232c80)
	github.com/hashicorp/terraform-provider-google/google/resource_storage_object_acl.go:83 +0x2ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0xc0004997a0, 0x3fca580, 0xc000c445c0, 0xc000e34310, 0xc000d7de30, 0x3c1dcf8, 0x3a64b20, 0xc000041100, 0x4ebb300, 0x3506e20, ...)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/schema.go:540 +0xb29
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0xc000b02620, 0x3fca580, 0xc000c445c0, 0xc000e34310, 0xc000d7de30, 0x3a64b20, 0xc000041100, 0x0, 0x0, 0x0)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource.go:506 +0xa5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).PlanResourceChange(0xc000132a98, 0x3fca580, 0xc000c445c0, 0xc000e08280, 0xc000c445c0, 0x39d5940, 0xc000d7c600)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/grpc_provider.go:693 +0x7c5
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).PlanResourceChange(0xc001051240, 0x3fca628, 0xc000c445c0, 0xc000e34230, 0xc001051240, 0xc000d7c600, 0xc0006c4ba0)
	github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:315 +0xb5
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x39d5940, 0xc001051240, 0x3fca628, 0xc000d7c600, 0xc001146600, 0x0, 0x3fca628, 0xc000d7c600, 0xc000e40000, 0x35c)
	github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:362 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000421880, 0x4000d78, 0xc000103380, 0xc000c86240, 0xc00070c990, 0x4e76848, 0x0, 0x0, 0x0)
	google.golang.org/grpc@v1.40.0/server.go:1297 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000421880, 0x4000d78, 0xc000103380, 0xc000c86240, 0x0)
	google.golang.org/grpc@v1.40.0/server.go:1626 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000123060, 0xc000421880, 0x4000d78, 0xc000103380, 0xc000c86240)
	google.golang.org/grpc@v1.40.0/server.go:941 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.40.0/server.go:939 +0x1fd

Error: The terraform-provider-google_v4.6.0_x5 plugin crashed!

[maroux]

Upon digging further, it appears that https://cloud.google.com/storage/docs/json_api/v1/objects/get API call didn't include owner in response which is whats used by https://pkg.go.dev/google.golang.org/api/storage/v1#Object to populate the Owner property. I think it's because of this little tidbit: If iamConfiguration.uniformBucketLevelAccess.enabled is set to true, this field does not apply. in the docs. The bucket in question does have a uniform bucket level access enabled, and that's why I'm guessing the storage API does not include owner property even with projection=full. I'm not sure why owner is not set by the storage API - the bucket is question does NOT have uniform bucket level access enabled. Ultimately, the answer is to consider this field as optional here.

Aside - I'm not sure what prompted this change - my bucket has always had the uniform bucket level access as enabled and I don't see any change in the provider or google api client library, so that just leaves the storage API :/

[maroux]

Ok, I have a new theory - this happened with exactly 3 objects in my org. All three of which were last edited by former employees. So it seems:

1. Create a bucket with uniform policy = false
2. Create an object within that bucket using terraform
3. Edit that object from an user account
4. Delete that user account
5. Run terraform plan - crash!

[maroux]

Ok, confirmed my theory above. The patch is simple enough, I'm just not sure how to write test for it, thinking ..

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.