Fix permadiff that reorders stateful_external_ip blocks on google_compute_instance_group_manager and google_compute_region_instance_group_manager resources

[SarahFrench]

Relates to (but doesn't close!) this issue : hashicorp/terraform-provider-google#13430

This PR updates code related to stateful_external_ip blocks in the google_compute_instance_group_manager & google_compute_region_instance_group_manager resources to address a permadiff that proposes re-ordering those blocks. The root of this problem is that the API returns those blocks in a sorted order that doesn't match the user's config. The state will contain data returned from the API, so there's always a plan trying to return to the config-defined order.

Prior to this PR, the flattener function for stateful_external_ip blocks performs a conversion from a map (how the data from the API is presented by the compute Go client library) to an array (how the data is stored in state).

This PR updates the flattener function to also re-order data returned by the API so it matches the user's Terraform configuration. This is achieved by passing the schema.ResourceData data for the resource into the flattener, to act as a source of information about the order of those fields in the user's config.

Release Note Template for Downstream PRs (will be copied)

compute: fixed a permadiff that reordered `stateful_external_ip` and `stateful_internal_ip` blocks on `google_compute_instance_group_manager` and `google_compute_region_instance_group_manager` resources

[SarahFrench]

@rileykarson - I requested you as reviewer as you provided initial feedback on the issue this PR relates to about how the solution would be to re-order information from the API to match the user's config.

I feel a bit weird about this logic being in the flattener, but on the other hand the flattener was already performing a data transformation. If you think this reordering should be performed in a function separate to the flattener (or should be pulled into a separate function that's used in there, etc) please let me know! Having said that, we're lucky that whatever we land on would be removed when the 'true' fix for this issue, changing the field to a Set, is added in a future major release.

Also, should I extend this logic to also affect the stateful_internal_ip field?

[SarahFrench]

Odd error in the generate diffs build:

error: RPC failed; curl 16 Error in the HTTP2 framing layer

I'll try again

/gcbrun

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 369 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 4 files changed, 369 insertions(+), 6 deletions(-))

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 370 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 4 files changed, 370 insertions(+), 6 deletions(-))

[modular-magician]

Tests analytics

Total tests: 3259
Passed tests 2926
Skipped tests: 332
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccRegionInstanceGroupManager_APISideListRecordering

Get to know how VCR tests work

[modular-magician]

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccRegionInstanceGroupManager_APISideListRecordering[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

---

$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 388 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 4 files changed, 388 insertions(+), 7 deletions(-))

[modular-magician]

Tests analytics

Total tests: 3262
Passed tests 2930
Skipped tests: 332
Affected tests: 0

$\textcolor{green}{\textsf{All tests passed in REPLAYING mode.}}$
View the build log

[SarahFrench]

@rileykarson could you please take a look at this when you have a chance? Thanks!

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 429 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 4 files changed, 429 insertions(+), 7 deletions(-))

[rileykarson]

Small note on deadcode in the handwritten test, but otherwise LGTM. Going through some snippets in the parent comments:

The root of this problem is that the API returns those blocks in a sorted order that doesn't match the user's config.

Just to make sure I'm not missing something, it returns them in an order that doesn't match the user's config, but not a sorted order, i.e. the ordering on the API side does not matter, right? That's my understanding based on hashicorp/terraform-provider-google#13430 (comment)

I feel a bit weird about this logic being in the flattener...

Seems right to me. I'd only really consider breaking out of a flattener if we were accessing multiple same-level fields. We could use a decoder to handle both the external and internal versions in the same place, although flatteners are fine since their individual handlers are still field-scoped. (wrt. child fields in when I'd use a flattener vs an alternative, a parent can access its children in its flattener)

Also, should I extend this logic to also affect the stateful_internal_ip field?

I think so, unless there's a reason this issue couldn't happen there!

[modular-magician]

Tests analytics

Total tests: 860
Passed tests 795
Skipped tests: 65
Affected tests: 0

Click here to see the affected service packages

• compute

$\textcolor{green}{\textsf{All tests passed in REPLAYING mode.}}$
View the build log

[SarahFrench]

Thanks for the review!

The root of this problem is that the API returns those blocks in a sorted order that doesn't match the user's config.

Just to make sure I'm not missing something, it returns them in an order that doesn't match the user's config, but not a sorted order, i.e. the ordering on the API side does not matter, right? That's my understanding based on hashicorp/terraform-provider-google#13430 (comment)

Yeah you're right that the API side ordering doesn't appear to matter. It's not clear if the reordering is due to the API explicitly choosing to do that or because the Compute Go client library storing the data in a map results in losing the old ordering. Below's an example of the map where the keys correspond to interface_name values in the TF configuration and the keys appear alphabetically ordered, which means slices generated from it are alphabetical and may not match a user's TF configuration:

map[string]google.golang.org/api/compute/v1.StatefulPolicyPreservedStateNetworkIp {
   "nic0": {AutoDelete: "NEVER", ForceSendFields: []string len: 0, cap: 0, nil, NullFields: []string len: 0, cap: 0, nil},
   "nic1": {AutoDelete: "NEVER", ForceSendFields: []string len: 0, cap: 0, nil, NullFields: []string len: 0, cap: 0, nil},
}

---

Also, should I extend this logic to also affect the stateful_internal_ip field?

I think so, unless there's a reason this issue couldn't happen there!

I'll update this PR to pull the sorting code into a reusable function and I'll update the flattenStatefulPolicyStatefulInternalIps func to also use it.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 373 insertions(+), 21 deletions(-))
Terraform Beta: Diff ( 4 files changed, 373 insertions(+), 21 deletions(-))

[modular-magician]

Tests analytics

Total tests: 860
Passed tests 794
Skipped tests: 65
Affected tests: 1

Click here to see the affected service packages

• compute

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccRegionInstanceGroupManager_APISideListRecordering

Get to know how VCR tests work

[modular-magician]

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccRegionInstanceGroupManager_APISideListRecordering[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

---

$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test

[SarahFrench]

Re-requesting review as I've made big-ish changes since the last approval and don't want to abuse the fact the approval isn't dismissed by new commits