unique_writer_identity permadiff in google_logging_project_sink when destination is a Cloud Logging bucket in the same project

[wuluk]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.5.1
on linux_amd64

• provider registry.terraform.io/hashicorp/google v4.74.0
• provider registry.terraform.io/hashicorp/google-beta v4.74.0
• provider registry.terraform.io/hashicorp/random v3.5.1

Affected Resource(s)

• google_logging_project_sink

Terraform Configuration Files

resource "google_logging_project_bucket_config" "retention_logging_bucket" {
  bucket_id      = "Retention"
  project        = data.terraform_remote_state.qaus-connect-projects.outputs.my_project.project_id
  location       = "global"
  description    = "Extended retention bucket"
  retention_days = 400
}

resource "google_logging_project_sink" "logging-sink-cloudnat" {
  name        = "logging-sink-cloudnat"
  project     = data.terraform_remote_state.qaus-connect-projects.outputs.my_project.project_id
  destination = "logging.googleapis.com/projects/${data.terraform_remote_state.qaus-connect-projects.outputs.my_project.project_id}/locations/global/buckets/Retention"
  filter      = "resource.type=\"nat_gateway\" AND jsonPayload.allocation_status=\"OK\""
}

Debug Output

  # google_logging_project_sink.logging-sink-cloudnat must be replaced
-/+ resource "google_logging_project_sink" "logging-sink-cloudnat" {
      - disabled               = false -> null
      ~ id                     = "projects/my_project/sinks/logging-sink-cloudnat" -> (known after apply)
        name                   = "logging-sink-cloudnat"
      ~ unique_writer_identity = true -> false # forces replacement
      + writer_identity        = (known after apply)
        # (3 unchanged attributes hidden)
    }

{"@level":"info","@message":"google_logging_project_sink.logging-sink-cloudnat: Plan to replace","@module":"terraform.ui","@timestamp":"2023-07-24T09:28:24.831867Z","change":{"resource":{"addr":"google_logging_project_sink.logging-sink-cloudnat","module":"","resource":"google_logging_project_sink.logging-sink-cloudnat","implied_provider":"google","resource_type":"google_logging_project_sink","resource_name":"logging-sink-cloudnat","resource_key":null},"action":"replace","reason":"cannot_update"},"type":"planned_change"}

Panic Output

Expected Behavior

Terraform shouldn't attempt to set an unique_writer_identity to 'true' when a google_logging_project_sink when destination is a Cloud Logging bucket in the same project. When the destination is in the same project, a default "serviceAccount:cloud-logs@system.gserviceaccount.com" should be used. When creating a sink via a GCP console, it is not possible to enable unique_writer_identity

 terraform state show google_logging_project_sink.logging-sink-cloudnat
# google_logging_project_sink.logging-sink-cloudnat:
resource "google_logging_project_sink" "logging-sink-cloudnat" {
    destination            = "logging.googleapis.com/projects/my_project/locations/global/buckets/Retention"
    disabled               = false
    filter                 = "resource.type=\"nat_gateway\" AND jsonPayload.allocation_status=\"OK\""
    id                     = "projects/my_project/sinks/logging-sink-cloudnat"
    name                   = "logging-sink-cloudnat"
    project                = "my_project"
    unique_writer_identity = true
}

gcloud logging sinks describe logging-sink-cloudnat --project=my_project
createTime: '2023-07-24T09:13:25.715510151Z'
destination: logging.googleapis.com/projects/my_project/locations/global/buckets/Retention
filter: resource.type="nat_gateway" AND jsonPayload.allocation_status="OK"
name: logging-sink-cloudnat
updateTime: '2023-07-24T09:13:25.715510151Z'

Compared to sink that actually requires unique_writer_identity:

gcloud logging sinks describe vpc-flow-logs --project=my_project
createTime: '2022-04-28T10:38:47.789649030Z'
destination: bigquery.googleapis.com/projects/my_remote_project/datasets/vpcflowlogsqa_dataset
filter: resource.type="gce_subnetwork" AND log_id("compute.googleapis.com/vpc_flows")
name: vpc-flow-logs
updateTime: '2022-04-28T10:38:47.789649030Z'
writerIdentity: serviceAccount:p964033355324-900639@gcp-sa-logging.iam.gserviceaccount.com

Actual Behavior

Terraform succesfully creates a working sink. However, on 2nd and subsequent runs it attempts to change "unique_writer_identity = true -> false # forces replacement" which fails and results in permadiff.

Steps to Reproduce

1. run terraform apply to create a bucket and a sink
2. run 'terraform apply' 2nd time to observe the issue

Important Factoids

References

• b/293780716

[pengq-google]

The unique_writer_identity flag is used to distinguish legacy behavior from new behavior and will be deprecated when all migration completes. New create/update requests will always set this flag to true. So Terraform setting this field to be true is an expected behavior.

The gcloud logging sinks describe command does not show the writerIdentity field for log bucket sinks whose destination is the same project.

[pengq-google]

I am not sure why
However, on 2nd and subsequent runs it attempts to change "unique_writer_identity = true -> false # forces replacement" which fails and results in permadiff.

happened. I haven't succeeded to reproduce this error yet.

[awoisoak]

Terraform succesfully creates a working sink. However, on 2nd and subsequent runs it attempts to change "unique_writer_identity = true -> false # forces replacement" which fails and results in permadiff.

I was able to replicate that force replacement behavior with Google provider 4.58.0.
After upgrading to 4.84.0 google_logging_project_sink doesn't replace itself anymore when terraform attempts to change unique_writer_identity
I think this behavior change was introduced here (which was released in 4.82.0)

[tSte]

I am not sure if this is related to this issue or it should be filed as separate one, but when I create two logging sinks with unique_writer_identity = true, both sinks will have the same service account name.

[pengq-google]

Cloud logging (along with other GCP services) changed the behavior of configuring service accounts.
Before the Cloud Logging config service creates a distinct service account and associates it with the sink just created.
With the most up-to-date recommendation we changed to create one service account per project, if the user does not provide a user-defined service account.