Terraform BigQuery Table Hive partitioning support

[ffung]

Fixes: hashicorp/terraform-provider-google#5664
As of March 2, range partioning / hive partitioning is GA, see https://cloud.google.com/bigquery/docs/release-notes.
Note: Doesn't support require_partition_filter attribute as this isn't available from the used BigQuery SDK.

Release Note Template for Downstream PRs (will be copied)

bigquery: Added support for `google_bigquery_table` `hive_partitioning_options`

bigquery: Added `google_bigquery_table` `range_partitioning` to GA

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@slevenick, please review this PR or find an appropriate assignee.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 318 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 152 insertions(+), 5 deletions(-))

[modular-magician]

Oops! It looks like you're using an unknown release-note type in your changelog entries:

• REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

[ffung]

± env |grep GOOG
GOOGLE_REGION=us-central1
GOOGLE_ZONE=us-central1-a

± make testacc TEST=./google TESTARGS='-run=TestAccBigQueryTable_Hive'

==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google -v -run=TestAccBigQueryTable_Hive -timeout 240m -ldflags="-X=github.com/terraform-providers/terraform-provider-google/version.ProviderVersion=acc"
=== RUN   TestAccBigQueryTable_HivePartitioning
=== PAUSE TestAccBigQueryTable_HivePartitioning
=== CONT  TestAccBigQueryTable_HivePartitioning
--- PASS: TestAccBigQueryTable_HivePartitioning (8.96s)
PASS
ok  	github.com/terraform-providers/terraform-provider-google/google	(cached)

[slevenick]

Thanks for the contribution!

Sorry it took so long to review, but this all looks great. Just a few notes on docs/tests. When you push a new commit it should successfully complete the checks as I standardized the release note comment

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

[slevenick]

Minor issues and this will likely require a rebase due to an unrelated PR that changed test structure.

If you can do the rebase I'll get this merged in, looks good!

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 327 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 3 files changed, 161 insertions(+), 9 deletions(-))

[ffung]

@slevenick I think I resolved all the issues, is there anything I still can do for this PR?

[slevenick]

@slevenick I think I resolved all the issues, is there anything I still can do for this PR?

Ah, sorry lost track of this. Looks like there are conflicts now with master, could you rebase on master and I can run tests and get this merged?

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 327 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 3 files changed, 161 insertions(+), 9 deletions(-))

[ffung]

@slevenick done.I cannot see why the build fails, but let me know if I can help address the issue.

[slevenick]

@slevenick done.I cannot see why the build fails, but let me know if I can help address the issue.

Looks like the files aren't formatted correctly:

gofmt needs running on the following files:
./google/resource_bigquery_table_test.go
./google/resource_bigquery_table.go```

You can generate the downstream files, run `gofmt` on them and then copy them back into Magic Modules to fix this

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

[ffung]

@slevenick ran gofmt, build still fails though

[slevenick]

@slevenick ran gofmt, build still fails though

Looks like the problem is now:

google/resource_bigquery_table_test.go:69:30: not enough arguments in call to testBucketName
	have ()
	want (*testing.T)

You can run make test in the generated provider to make sure everything passes

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

[ffung]

@slevenick ah sorry that was sloppy, tests are running succesfully now locally, but the builds still seems to fail.

[slevenick]

Sorry the process is a little rough around the edges. Looks like we enforce that gofmt must be run with a specific flag 😞

Step #22 - "tpgb-test": google-beta/resource_bigquery_table.go:336: File is not `gofmt`-ed with `-s` (gofmt)
Step #22 - "tpgb-test": 			"range_partitioning": &schema.Schema{```

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 320 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 3 files changed, 152 insertions(+), 2 deletions(-))

[ffung]

@slevenick that should fix it;-)

[slevenick]

Thanks for the addition!

[LoekL]

FYI I tried using this today, no dice. The table does get created:

But there's no sign of the partitions in the table schema:

If I however run:

bq mkdef \
  --autodetect \
  --ignore_unknown_values \
  --source_format=NEWLINE_DELIMITED_JSON \
  --hive_partitioning_mode=CUSTOM \
  --hive_partitioning_source_uri_prefix=gs://anhistous-metonymic-7578834-analytics-staging-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING} \
  gs://anhistous-metonymic-7578834-analytics-staging-batched/data_type=jsonl/event_schema=playfab/\* \
  AnalyticsEnvironment:STRING,PlayFabEnvironment:STRING,SourceType:STRING,Source:STRING,EventNamespace:STRING,TitleId:STRING,GroupBatchId:STRING,BatchId:STRING,EventId:STRING,EventName:STRING,EntityType:STRING,EntityId:STRING,Timestamp:TIMESTAMP,ReceivedTimestamp:TIMESTAMP,BatchedTimestamp:TIMESTAMP,BatchJobName:STRING,EventAttributes:STRING \
  > /Users/loek/Desktop/events_playfab_staging_hive_partitioned_batched

Followed by:

bq mk --table --location=US --external_table_definition=/Users/loek/Desktop/events_playfab_staging_hive_partitioned_batched external.events_playfab_hive_partitioned_batched

It does work:

&

The only difference within the BQ UI I can see is that when going via the bq CLI, I don't see Compression = GZIP in the table details. Am using Terraform v0.12.18 & provider.google v3.26.0.

[slevenick]

@LoekL can you create a new issue on https://github.com/terraform-providers/terraform-provider-google with steps to reproduce this problem so that we can look into this?

[ffung]

yes @LoekL please raise an issue, I cannot reproduce your problem. If I apply this, hive partitioning works just fine.

resource "google_bigquery_dataset" "test" {
  dataset_id                  = "foo"
  friendly_name               = "test"
  description                 = "This is a test description"
  location                    = "europe-west3"
  default_table_expiration_ms = 3600000

  labels = {
    env = "default"
  }
}

resource "google_bigquery_table" "test" {
  dataset_id = google_bigquery_dataset.test.dataset_id
  table_id   = "bar"

  external_data_configuration {
    source_format = "PARQUET"
    autodetect = true
    source_uris= ["gs://project-2resdfdsafsdf43/test/*"]

    hive_partitioning_options {
      mode = "CUSTOM"
      source_uri_prefix = "gs://project-2resdfdsafsdf43/test/{part:STRING}/{dt:DATE}"
    }
  }
}

[LoekL]

@slevenick @ffung - see: hashicorp/terraform-provider-google#6693