Possible bug when using defined tags and extended_metadata

[stephenpearson]

Terraform Version

Terraform v0.11.8

OCI Provider Version

terraform-provider-oci 2.2.4

Description:

When building a compute instance which includes defined_tags and also extended_metadata then occasionally we see strange behaviour where Terraform tries to reapply the defined_tags even though they are already set on the instance. This leads to an error.

I've narrowed this down to a specific set of circumstances. All the following conditions must be true.

• The extended_metadata must be set to something, anything it seems.
• The defined_tags must be applied with the wrong case. E.g. if we have a namespace called Test and a defined tag called foo then this would be triggered if the tag were referred to as Test.FOO in the Terraform definition.
• The resource is removed from the state and reimported

Other actions are capable of triggering this error although I'm not sure what they are. But removing and importing the resource definitely triggers it.

Example terraform:

Assume we already have a tag namespace called stepears and a defined tag in that namespace called foo (all lower case).

resource "oci_core_instance" "server" {
  compartment_id      = "ocid1.compartment.oc1..aaaaaaaaif2z7ejltfg37zli4imszthhlnbve5dlbaapluczp5njak3dhzva"
  availability_domain = "KhAa:US-ASHBURN-AD-1"
  shape               = "VM.Standard1.1"

  display_name = "bugtest"

  source_details {
    source_type = "image"
    source_id   = "ocid1.image.oc1.iad.aaaaaaaa2tq67tvbeavcmioghquci6p3pvqwbneq3vfy7fe7m7geiga4cnxa"
  }

  create_vnic_details {
    subnet_id        = "ocid1.subnet.oc1.iad.aaaaaaaa4xrp6r2hnr6ww3q44xxsmezbcau4pvtd3iu2olitgd26ts6zuxnq"
    hostname_label   = "bugtest"
    assign_public_ip = true
  }

  metadata {
    ssh_authorized_keys = "${file("/Users/stephen/.ssh/id_rsa.pub")}"
  }

  defined_tags = {
   # >>> NOTICE INCORRECT CASE IN TAG NAME BELOW <<<
    stepears.Foo = "foo"
  }

  # >>> THIS NEEDS TO EXIST AS WELL TO TRIGGER THE ERROR <<<
  extended_metadata {
    thing = "foo"
  }

  lifecycle = {
    ignore_changes = ["extended_metadata"]
  }
}

Then run the following commands:

$ terraform apply
$ terraform state show oci_core_instance.server
# Make a note of the OCID
$ terraform state rm oci_core_instance.server
$ terraform import oci_core_instance.server <OCID>
$ terraform apply

Expected behaviour: Terraform should detect that the instance is in the desired state and do nothing.
Actual behaviour:

Terraform will perform the following actions:

  ~ oci_core_instance.server
      defined_tags.%:            "1" => "1"
      defined_tags.stepears.Foo: "" => "foo"


Plan: 0 to add, 1 to change, 0 to destroy.
...
Error: Error applying plan:

1 error(s) occurred:

* oci_core_instance.server: 1 error(s) occurred:

* oci_core_instance.server: Service error:InvalidParameter. definedTags contain duplicate key
. http status code: 400. Opc request id: ad19b472e227c8c3243f3e683db49205/F9D50C49CA6615E496BC200CAAC19B67/AF74B1DEC5C49B68EF13B540E99EFDF3

The error is because OCI does not allow duplicate defined tags on an instance, but Terraform has somehow failed to detect that the tag is already there.

Workaround

The workaround is very simple. Just make sure that the defined tag in the HCL has exactly the same case as the one defined in OCI.

I'm not 100% sure whether this is a bug or a feature, but I'm logging this as an issue because the behaviour was certainly confusing to me especially the fact that the presence or absense of extended_metadata appears to switch the problem on and off.

Terraform Plan

plan.gz

[alexng-canuck]

Investigating this.

[alexng-canuck]

Hello @stephenpearson

This appears to be a bug in Terraform that manifests only when you have 3 factors:

• the ignore_changes setting is set
• the field being ignored would otherwise have triggered a new resource (i.e. a ForceNew field)
• you are specifying other fields that are case-insensitive, such as defined_tags (our provider handles this with diff suppression functions)

When these 3 things occur, Terraform is somehow omitting our provider's diff suppression functions and causing unnecessary diffs for you. We'll need to root cause this further and give that feedback to HashiCorp.

In the meantime, our latest version of the provider (v3.1.0) does not exhibit this behavior because the extended_metadata field is now updatable (i.e. no longer ForceNew). Can you give that version a try?

To upgrade from v2 to v3.1.0; follow this guide: https://www.terraform.io/docs/providers/oci/guides/version-3-upgrade.html

[stephenpearson]

Hello @alexng-canuck. Yes, I can confirm that version 3.1.0 of the provider fixes this. Nice that we can modify the metadata and extended_metadata too 👍

Thank you! I'll close this issue as it is no longer applicable to the latest version.

[alexng-canuck]

Just to close on the root cause, the Terraform issue that caused this behavior is known to HashiCorp and a common behavior that folks have been seeing with other Terraform providers.

hashicorp/terraform#18209

The issue will always occur when you have all of the following factors:

1. You update a field that is normally suppressed by a DiffSuppressFunction (e.g. a field that is suppressed because it's supposed to be case insensitive)
2. You update another field that forces a new resource.
3. You add the field from (2) in ignore_changes

The details are very technical, but this issue happens because both fields from (1) and (2) are being suppressed in different ways. Terraform inadvertently forgets that the attribute from (1) was originally suppressed when it attempts to suppress (2). You end up with a plan where (1) has changed even though it should have been suppressed.

This is expected to be addressed in v0.12 release of Terraform.