google_bigquery_dataset_access produces produced an unexpected new value errors when assigning a service account

[ghost]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Affected Resource(s)

• google_bigquery_dataset_access

Terraform Configuration Files

resource "google_bigquery_dataset" "vpc_log_internal" {
  dataset_id    = "vpc_log_internal"
  friendly_name = "VPC Logging Database"
  description   = "Dataset sink for VPC logs"
  location      = "US"
  project = var.project_id

  default_table_expiration_ms = var.dataset_expiration_ms

  labels = { "terraform" : "true" }

}

resource "google_bigquery_dataset_access" "dns_query_sink" {
  dataset_id = google_bigquery_dataset.vpc_log_internal.dataset_id
  project = var.project_id
  role = "roles/bigquery.dataEditor"
  #role = "WRITER"
  iam_member = google_logging_project_sink.resource_dns_query_bigquery.writer_identity
}

resource "google_logging_project_sink" "resource_dns_query_bigquery" {
  name = "resource_dns_query_bigquery"
  project = var.project_id

  # Can export to pubsub, cloud storage, or bigquery
  destination = "bigquery.googleapis.com/${google_bigquery_dataset.vpc_log_internal.id}"

  # Log all WARN or higher severity messages relating to instances
  filter = "resource.type=\"dns_query\""

  # Use a unique writer (creates a unique service account used for writing)
  unique_writer_identity = true
}

Debug Output

https://gist.github.com/asaba-gl/1f1065d5dc924ad08dc2f5adabbf3dc7

Panic Output

Expected Behavior

The permission is assigned and terraform reports success.

Actual Behavior

The change was applied. The service account is assigned the permissions as verified in the GCP console, but terraform reports an error.

Steps to Reproduce

1. terraform apply

Important Factoids

The full writer_identity which is prefixed by serviceaccount: is accepted by the provider and the API, but the resources does not parse it correctly and searches for "" (empty string) according to the logs.

References

Past problems with this resource:

• Cannot apply google_bigquery_dataset_access #6176

[venkykuberan]

Looks like API is accepting iam_member as access member type however its sending the response under user_by_email access member type. Provider doesn't find the required member type in the response and failing to set the state

sample response

"access": [
     {
       "role": "WRITER",
       "specialGroup": "projectWriters"
     },
     {
       "role": "WRITER",
       "userByEmail": "p744037022015-094547@gcp-sa-logging.iam.gserviceaccount.com"
     },
     {
       "role": "OWNER",
       "specialGroup": "projectOwners"
     },
     {
       "role": "OWNER",
       "userByEmail": "terrform-acct@project.iam.gserviceaccount.com"
     },
     {
       "role": "READER",
       "specialGroup": "projectReaders"
     }
   ]

Change is required on the API side i believe @megan07 what are your thoughts on this ?

[megan07]

@asaba-gl and @venkykuberan , it looks like this is a bug in the provider and will require some custom code, I believe. From what I'm seeing it appears that the API accepts an "iamMember" which has a prefix (in this case "serviceAccount"), and it seems like on the backend they parse that to realize it is of type "userByEmail" and sends it back in the response that way. I'm guessing (but will require further testing) that "iamMember" likely is never returned, and depending on the prefix, the associated type will always be returned. The types are listed here, and in the description of each it shows which IAM prefix it's associated with. I've been digging into this all day, trying to find an ideal solution. I haven't found one I like yet, but wanted to update you as to what I've found. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!