Support pubsub schema updates

[tkinz27]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

PubSub recently added support for pubsub schema revisions -- Release Notice

Right now changing the a pubsub schema is a destructive action (even adding a new field which is a backwards compatible change typically). With the new feature, you can make those changes without having to recreate the pubsub topic.

New or Affected Resource(s)

• google_pubsub_schema_revision or maybe built into google_pubsub_schema

Potential Terraform Configuration

Separate Resource

resource "google_pubsub_schema" "schema" {
  name = "my-schema"
}

resource "google_pubsub_schema_revision" "revision1" {
   schema = google_pubsub_schema.schema.name
   definition = var.pubsub_schema
}

resource "google_pubsub_schema_revision" "revision2" {
  schema = google_pubsub_schema.schema.name
  definition = var.pubsub_schema_rev2

   depends_on = [google_pubsub_schema_revision.revision1] # maybe a way to ensure the order is set up?
}

resource "google_pubsub_topic" "topic" {
   name = "mytopic"
  schema_settings = {
    ....
  }
}

Built into google_pubsub_schema

resource "google_pubsub_schema" "schema" {
  name = "my-schema"

  schema_revision {
      definition = var.pubsub_schema
  } 
  schema_revision {
     definition = var.pubsub_schema_rev2
  }
}

resource "google_pubsub_topic" "topic" {
   name = "mytopic"
  schema_settings = {
     ....     
  }
}

I do not really have a preference, I just want to be able to evolve schemas without destroying my topic.

References

• Updating Pub/Sub topic schema causes issues #12287 -- I think schema revisions is what this user was looking for

[rodmatos]

@rileykarson: would pull requests to get this feature upstream be welcome? At my workplace we are making use of this provider to manage schemas and would love to have the possibility of evolving them.

[kamalaboulhosn]

I will be making the updates needed for this.

[sebastienrufiange]

Hi !

I also need this issue resolved not only to support newest pub/sub schema evolution feature but also essentially because the terraform import command ignores the definition field, thus detecting unplanned changes. So, it is currently unusable for my use case.

Steps I used to reproduce:

1. Import a pubsub schema locally

terraform import -var-file=$HOME/.../src/environments/production.tfvars module.my_module.google_pubsub_schema.records_schema projects/prod-project-xxx/schemas/records-schema

2. output the imported schema
   terraform state show module.my_module.google_pubsub_schema.records_schema

resource "google_pubsub_schema" "records_schema" {
id = "projects/prod-project-xxx/schemas/records-schema"
name = "records-schema"
project = "prod-project-xxx"
type = "AVRO"

timeouts {}
}

KO 1: The definition field is ignored.
(I tried different things: loading the definition from file; or hardcoding the value to no avail -- the definition field is always skipped and not imported.)

To validate I check the pub sub schema in GCloud
gcloud pubsub schemas describe records-schema --project=prod-project-xxxx --format="value(definition)"

{
"type" : "record",
"name" : "record",
....
"fields" : [
{
"name" : "url",
"type" : "string",
},
...

In the end I cannot import the full pub/sub schema ; this issue would allow me to add a revision which may also fix it.
I can also open a bug issue if relevant.

Thanks!

[rileykarson]

Ah- I see there's a community PR, let's let that play out.

[aleccool213]

Hi! I'm really glad this is being worked on, I need this for my current workflow.

I see that the PR has been merged. How do I opt-in to this new behavior? I am still getting full-replacement by Terraform when I change the schema.

Edit:

Oh, likely need a new release of terraform-provider-google eh, Ill wait!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.