Adding required column to big query table fails with 'Cannot add required fields to an existing schema.'

[Ulf-H]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.0.8
registry.terraform.io/hashicorp/google v3.89.0

Affected Resource(s)

• google_bigquery_table

Terraform Configuration Files

Before change

resource "google_bigquery_table" "test_table" {
  dataset_id          = google_bigquery_dataset.data_normalized.dataset_id
  table_id            = "testTable"
  labels              = { "service" = "dataland" }
  project             = google_project.analytics_project.project_id
  deletion_protection = false

  encryption_configuration {
    kms_key_name = google_kms_crypto_key.bigquery_crypto_key.self_link
  }
  schema = <<-EOT
[
  {
    "name": "buildDeployId",
    "type": "STRING",
    "mode": "REQUIRED"
  },
  {
    "name": "project",
    "type": "STRING",
    "mode": "NULLABLE"
  }
]
EOT
}

After change

resource "google_bigquery_table" "test_table" {
  dataset_id          = google_bigquery_dataset.data_normalized.dataset_id
  table_id            = "testTable"
  labels              = { "service" = "dataland" }
  project             = google_project.analytics_project.project_id
  deletion_protection = false

  encryption_configuration {
    kms_key_name = google_kms_crypto_key.bigquery_crypto_key.self_link
  }
  schema = <<-EOT
[
  {
    "name": "buildDeployId",
    "type": "STRING",
    "mode": "REQUIRED"
  },
  {
    "name": "project",
    "type": "STRING",
    "mode": "NULLABLE"
  },
  {
    "name": "origin",
    "type": "STRING",
    "mode": "REQUIRED"
  }
]
EOT
}

Debug Output

Panic Output

Expected Behavior

The table is destroyed and a new table with the additional required column is created.

Actual Behavior

The provider tries to update the table and add the required column. This results in an error:

Error: googleapi: Error 400: Provided Schema does not match Table ulfh-analytics-25900:normalized.testTable. Cannot add required fields to an existing schema. (field: origin), invalid

Steps to Reproduce

1. terraform apply
2. Add a required column
3. terraform apply

Important Factoids

References

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.