Bigtable: Update local gc_rules on read

[kevinsi4508]

Currently when GCPolicy remote resource is updated not on TF (Pantheon, gcloud CLI, etc.), the gc_rules string are not updated on read. This PR fixes the issue by parsing the GC policy into JSON string that TF understands and set it in the local gc_rules. This hopefully will resolve the drift issue we have seen in KCC.

The original PR: #6415 was created by hoangpham95.

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

bigtable: added drift detection on `gc_rules` for `google_bigtable_gc_policy`

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @slevenick, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

---

Your assigned reviewer will help review your code by:

• Ensuring it's backwards compatible, covers common error cases, etc.
• Summarizing the change into a user-facing changelog note.
• Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

---

[kevinsi4508]

@hoangpham95 Please take a look.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 196 insertions(+), 14 deletions(-))
Terraform Beta: Diff ( 2 files changed, 196 insertions(+), 14 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2169
Passed tests 1931
Skipped tests: 235
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccComputeInstance_soleTenantNodeAffinities|TestAccComputeForwardingRule_update|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample

[modular-magician]

Tests passed during RECORDING mode:
TestAccComputeForwardingRule_update[Debug log]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[Debug log]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 202 insertions(+), 17 deletions(-))
Terraform Beta: Diff ( 2 files changed, 202 insertions(+), 17 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2170
Passed tests 1930
Skipped tests: 236
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccComputeInstance_soleTenantNodeAffinities|TestAccCGCSnippet_eventarcWorkflowsExample|TestAccFirebaserulesRelease_BasicRelease|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample

[modular-magician]

Tests passed during RECORDING mode:
TestAccCGCSnippet_eventarcWorkflowsExample[Debug log]
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[Debug log]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 204 insertions(+), 17 deletions(-))
Terraform Beta: Diff ( 2 files changed, 204 insertions(+), 17 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2170
Passed tests 1931
Skipped tests: 236
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccComputeInstance_soleTenantNodeAffinities|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccFirebaserulesRelease_BasicRelease

[modular-magician]

Tests passed during RECORDING mode:
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[Debug log]
TestAccFirebaserulesRelease_BasicRelease[Debug log]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[kevinsi4508]

FYI. I also tested this PR locally in KCC and I have verified that this fixes the drift issue stated in b/230626815.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 210 insertions(+), 17 deletions(-))
Terraform Beta: Diff ( 2 files changed, 210 insertions(+), 17 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2171
Passed tests 1932
Skipped tests: 237
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccComputeInstance_soleTenantNodeAffinities|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample

[modular-magician]

Tests passed during RECORDING mode:
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[Debug log]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[kevinsi4508]

Hi Sam. PTAL.

[slevenick]

I'm still not seeing this track changes quite right. Are you sure that changes are being detected and resolved on the gc_rules field itself?

I have a config of:

  resource "google_bigtable_gc_policy" "policy" {
    instance_name = google_bigtable_instance.instance.id
    table         = google_bigtable_table.table.name
    column_family = "some-column"
    provider = google-beta

    gc_rules = "{\"rules\":[{\"max_age\":\"10h\"},{\"max_version\":3}], \"mode\":\"intersection\"}"
  }

I manually go update the maxNumVersions via the console to be 4, the API starts returning:

         "gcRule": {
           "intersection": {
             "rules": [
               {
                 "maxAge": "36000s"
               },
               {
                 "maxNumVersions": 4
               }
             ]
           }
         }

And Terraform only sees a diff on the ID field which it ignores:

  # google_bigtable_gc_policy.policy has changed
  ~ resource "google_bigtable_gc_policy" "policy" {
      ~ id            = "(age() > 10h && versions() > 3)" -> "(age() > 10h && versions() > 4)"
        # (5 unchanged attributes hidden)
    }

[kevinsi4508]

See the comment I posted earlier.

With this PR:

1. tf apply
2. Modify gc_rules in both config and state files. For example: "10h" to "12h"
3. tf plan showed diff (will apply max_age = "10h" -> "12h")

We are seeing diff because we are reading the remote gc policy back now.

I believe we have to modify the gc_rules in both the config and the state file, then TF can detect the change. I don't fully understand this; I suspect that this has to do with GC policy doesn't support import.

[kevinsi4508]

Sam, I did you said earlier (modify the resource resource), I did get the correct diff in gc_rules (not just the id).

Terraform will perform the following actions:

  # google_bigtable_gc_policy.policy will be updated in-place
  ~ resource "google_bigtable_gc_policy" "policy" {
      ~ gc_rules      = jsonencode(
          ~ {
              ~ rules = [
                  ~ {
                      ~ max_age = "12h" -> "10h"
                    },
                    {
                        mode  = "intersection"
                        rules = [
                            {
                                max_age = "2h"
                            },
                            {
                                max_version = 2
                            },
                        ]
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        id            = "(age() > 12h || (age() > 2h && versions() > 2))"
        # (4 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.