Add BigQuery push subscriptions for Google Pub/Sub

[alexanderjlee]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Google recently added "BigQuery push" subscription delivery types for Pub/Sub.
https://cloud.google.com/pubsub/docs/bigquery

New or Affected Resource(s)

google_pubsub_subscription

Potential Terraform Configuration

resource "google_pubsub_topic" "example" {
  name = "example-topic"
}

resource "google_pubsub_subscription" "example" {
  name  = "example-subscription"
  topic = google_pubsub_topic.example.name

  ack_deadline_seconds = 20

  labels = {
    foo = "bar"
  }

  push_bigquery {
    bigquery_table = "project.dataset.table"

    attributes = {
      use_topic_schema = bool
      write_metadata = bool
      drop_unknown_fields = bool
    }
  }
}

References

• N/A

[DrFaust92]

looking into this

[voycey]

I can see this is merged however with the latest release (that I can see it is included in) I am still getting:

 Error: Unsupported block type
│
│   on modules/pubsub/pubsub.tf line 28, in resource "google_pubsub_subscription" "data-sub-bq":
│   28:   bigquery_config {
│
│ Blocks of type "bigquery_config" are not expected here.

Terraform config is:

resource "google_pubsub_subscription" "data-sub-bq" {
  provider = google-beta
  name  = "data-${var.stage}-sub-bq"
  topic = google_pubsub_topic.data.name

  ack_deadline_seconds = 20

  bigquery_config {
    table = "${var.gcp_project_id}:${var.bq_dataset}.${var.bq_table}"
  }

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.data_dead_letter.id
    max_delivery_attempts = 10
  }
}

Provider:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.28.0"
    }
    google = {
      source = "hashicorp/google"
    }

    google-beta = {
      source  = "hashicorp/google-beta"
      version = ">= 4.31.0"
    }

[mr-pascal]

@voycey
I think the latest release 4.31.0 was done right before this was merged. So I guess we have to wait for 4.32.0 till this feature is released 😞

This PR was merged on 1st August at 17:07 CEST, while the 4.31.0 release was at 1st August at 16:37 CEST, so really just like 30 minutes 😞

[voycey]

I checked the commits and could see the modified code in there but maybe there wasnt the magic module release in time :(

[mr-pascal]

@voycey
Seems like 4.32.0 is released, containing the changes we waited for :)

Also you can find the docs now in the Terraform Registry documentation -> https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription#example-usage---pubsub-subscription-push-bq

[voycey]

Great thanks 🙏

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.