-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
In-place upgrade fails for BigQuery ExternalTables #19177
Comments
Hi @rshamunov! Looking in your code I noticed that you are not using the correct format as terraform registry describes. Try to use something like this(watch the
|
Hello @ggtisc. resource "google_bigquery_table" "test-hived-table" {
dataset_id = "example_dataset"
project = "sh-unmanaged-tests"
table_id = "test-hived-table-tf"
deletion_protection = false
labels = {
l1 = "hello_again"
}
external_data_configuration {
autodetect = false
source_format = "CSV"
source_uris = [
"gs://sh-unmanaged-tests-bucket/example/*"
]
max_bad_records = 0
ignore_unknown_values = false
csv_options {
field_delimiter = ","
skip_leading_rows = 1
quote = "\""
}
hive_partitioning_options {
mode = "CUSTOM"
source_uri_prefix = "gs://sh-unmanaged-tests-bucket/example/{day:DATE}"
require_partition_filter = false
}
schema = <<EOF
[
{
"name": "Id",
"type": "INTEGER",
"mode": "REQUIRED"
},
{
"name": "Name",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "Weight",
"type": "INTEGER",
"mode": "REQUIRED"
}
]
EOF
}
}
provider "google" {
default_labels = {
l1 = "hello_again"
}
} Both |
Are you trying to use an |
@ggtisc yes. I create a bucket |
@rshamunov I saw it but it seems more like a naming error.
Example:
|
@ggtisc, Could you explain what's the "terraform registry format" exactly in this case? Do you want me to replace actual bucket name to terraform variable? Why it should be different? source_uris: source_uri_prefix: Did you try to create the table, change table label and apply the configuration again? |
yes @rshamunov and after testing many times everything is okay without errors. Terraform registry is the official documentation site of the terraform resources. Did you open the link to check the correct format? You need to open the link and realize the correct format that you need to follow to avoid this kind of errors. Also I gave you an example of how these resources work correctly. |
I've applied configuration almost exactly as you provided (changed bucket name and project name): resource "google_bigquery_dataset" "bq_ds_19177" {
dataset_id = "bq_ds_19177"
friendly_name = "BQ DS 19177"
description = "something"
location = "EU"
default_table_expiration_ms = 3600000
project = "sh-unmanaged-tests"
labels = {
env = "something"
}
}
resource "google_storage_bucket" "bucket_19177" {
name = "bucket-19177-777"
location = "US"
}
resource "google_bigquery_table" "bq_table_19177" {
dataset_id = google_bigquery_dataset.bq_ds_19177.dataset_id
project = "sh-unmanaged-tests"
table_id = "bq-table-19177"
deletion_protection = false
labels = {
l1 = "hello_again"
}
external_data_configuration {
autodetect = false
source_format = "CSV"
source_uris = [
"gs://${google_storage_bucket.bucket_19177.name}/*"
]
max_bad_records = 0
ignore_unknown_values = false
csv_options {
field_delimiter = ","
skip_leading_rows = 1
quote = "\""
}
hive_partitioning_options {
mode = "CUSTOM"
source_uri_prefix = "gs://${google_storage_bucket.bucket_19177.name}/{key1:STRING}"
require_partition_filter = false
}
schema = <<EOF
[
{
"name": "Id",
"type": "INTEGER",
"mode": "REQUIRED"
},
{
"name": "Name",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "Weight",
"type": "INTEGER",
"mode": "REQUIRED"
}
]
EOF
}
} As expected it applied successfully. I changed a label l1 ("hello" to "hello_again") and applied it again. It failed with the same error.
|
Confirmed issue! When |
Thanks a lot for the script. Was able to repro. I saw that |
Hi @rshamunov, here is the guidance from our API team:
Could you try specifying |
I tried to define a schema outside of external_data_configuration, in
Moreover, documentation for google_bigquery_dataset suggests to use
|
Thanks for the detailed follow-up. I believe the permadiff issue is the same as in #12465. The provider-side fix is in discussion but unfortunately won't be available soon. Meanwhile The other issue has some workarounds that may unblock you for the time being. For the resource documentation, I will check with our API team to see if that's no longer accurate. |
Community Note
Terraform Version & Provider Version(s)
Terraform v1.9.4
on linux_amd64
Affected Resource(s)
google_bigquery_table
Terraform Configuration
Debug Output
https://gist.github.com/rshamunov/8814cff7f2c6733061d92ca45ab36979#file-output-log
Expected Behavior
terrafrom apply completes in-place update of an external table
Actual Behavior
Error: googleapi: Error 400: When defining a table with an ExternalDataConfiguration, a schema must be present on either the Table or the ExternalDataConfiguration. If the schema is present on both, the schemas must be the same., invalid
Steps to reproduce
script to reproduce
Important Factoids
No response
References
No response
b/362528330
The text was updated successfully, but these errors were encountered: