Express Route: Partial update clears authorizations

[Preskton]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.11.10
+ provider.azurerm v1.18.0

Affected Resource(s)

• azurerm_express_route_circuit
• azurerm_express_route_circuit_authorization
• azurerm_express_route_circuit_peering

Terraform Configuration Files

resource "azurerm_resource_group" "rg" {
  name      = "gh2250-repro"
  location  = "${local.region}"
  tags      = "${local.tags}"
}

resource "azurerm_express_route_circuit" "express_route" {
  name                     = "${azurerm_resource_group.rg.name}-exr"
  resource_group_name      = "${azurerm_resource_group.rg.name}"
  location                 = "${azurerm_resource_group.rg.location}"

  service_provider_name    = "${local.express_route_service_provider_name}"
  peering_location         = "${local.express_route_peering_location}"
  bandwidth_in_mbps        = "${local.express_route_bandwidth_in_mbps}"

  sku {
    tier   = "${local.express_route_tier}"
    family = "${local.express_route_family}"
  }

  tags    = "${local.tags}"

  lifecycle {
    prevent_destroy   = true
  }
}

resource "azurerm_express_route_circuit_peering" "primary" {
  peering_type                  = "AzurePrivatePeering"
  resource_group_name           = "${azurerm_resource_group.rg.name}"
  express_route_circuit_name    = "${azurerm_express_route_circuit.express_route.name}"

  peer_asn                      = "${local.express_route_private_peering_asn}"
  primary_peer_address_prefix   = "${cidrsubnet(local.peering_network_prefix, 6, 0)}"
  secondary_peer_address_prefix = "${cidrsubnet(local.peering_network_prefix, 6, 1)}"
  vlan_id                       = "${local.express_route_private_peering_vlan_id}"
}

Steps to Reproduce

[Note: these are approximate steps to repro, as I have not yet reproduced this in a lab due to the important nature of this particular Express Route.]

I had manually provisioned an Express Route, Express Route Peerings, and 3 Express Route Authorizations via the Azure portal. As part of bringing legacy resources into Terraform, I was going through a multi-step process to run terraform imports and incrementally import portions of the above setup into terraform state. Note the authorizations had all been consumed by a third-party.

The process for each resource was:

• Write the terraform configuration block
• Execute terraform plan to validate the deployment and discard the plan
• Execute a terraform import for the target resource to bring it into state control
• Execute terraform plan again to ensure configuration values matched imported state values
• Execute terraform apply to write any outputs to state

The order I did this was:

• azurerm_resource_group
• azurerm_express_route_circuit
• azurerm_express_route_peering (only added private peering)

The import went successfully and upon executing the post-import terraform apply, the only difference that terraform noted was adding tags (an in place update). Upon applying, the peering, the ARM API failed stating that I was using a "stale" configuration -- sorry, I don't have the exact text of the error, will try to repro. The Azure portal stated my circuit was now in a failed state. After running the Express Route circuit refresh via the portal, I noticed that my Express Route Authorizations were no longer listed in the Azure portal nor via az cli.

At this point, I opened a Microsoft support ticket and engaged Azure Rapid Response and the Product Team.

Actual Behavior

• Terraform reported a diff on the MicrosoftPeering even though there was not a MicrosoftPeering configuration set:

Terraform will perform the following actions:

  ~ azurerm_express_route_circuit_peering.primary
      microsoft_peering_config.#: "1" => "0"

• Terraform applied updates to express route (tags)
• Express route authorizations no longer visible in ARM/az cli in providing subscription
• Underlying network still up - consuming subscription of authorizations still has express route authorizations attached

Hypothesis

• ARM endpoint to update ExpressRoute might require peerings/authorizations as part of request or Azure considers it a delete for those blocks.
• Azure product team to add more detail below.

Expected Behavior

Many possibilities:

• TF should have applied the change and not destroyed the child authorizations
• TF should have notified that it would have destroyed the authorizations as part of a plan or apply
• ARM should have rejected the update since the authorizations were already reedemed and not truly deletable

Debug Output

None available, yet

Panic Output

None

References

• Azure ticket issue: 118110521001969 (Microsoft-internal only)

[mbfrahry]

Hey @Preskton, thanks for opening this issue. We've got a fix up for this that should prevent this from happening again.

[mbfrahry]

This has been merged and will make it into the next release 1.29.0

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!