Unable to determine the Resource ID for the Key Vault at URL

[tschechniker]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.9

AzureRM Provider Version

3.30.0

Affected Resource(s)/Data Source(s)

Unable to determine the Resource ID for the Key Vault at URL

Terraform Configuration Files

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_key_vault" "example" {
  name                        = "examplekeyvault"
  location                    = azurerm_resource_group.example.location
  resource_group_name         = azurerm_resource_group.example.name
  enabled_for_disk_encryption = true
  tenant_id                   = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days  = 7
  purge_protection_enabled    = false

  sku_name = "standard"

  access_policy {
    tenant_id = data.azurerm_client_config.current.tenant_id
    object_id = data.azurerm_client_config.current.object_id

    key_permissions = [
      "Get",
    ]

    secret_permissions = [
      "Get",
    ]

    storage_permissions = [
      "Get",
    ]
  }
}

resource "azurerm_key_vault_key" "example" {
  name         = "example-key"
  key_vault_id = azurerm_key_vault.example.id
  key_type     = "RSA-HSM"
  key_size     = 4096

  expiration_date = timeadd(formatdate("YYYY-MM-DD'T'12:00:00Z", timestamp()), "26297h")

  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]
}

Debug Output/Panic Output

Error: Unable to determine the Resource ID for the Key Vault at URL "https://XXXX.vault.azure.net/"

Expected Behaviour

We are using Terratest to setup and destroy azure resources in a pipeline. The creation and the deletion works fine.

Actual Behaviour

The creation works fine but as soon as we run terraform destroy he cant find the resource id

Steps to Reproduce

terraform apply
terraform destroy

Important Factoids

Azure Westeurope

References

No response

[jeffwmiles]

I'm seeing this issue as well - I'm wondering how many other reporters of this issue are using Private Endpoints in addition to public access for their KeyVault (as I am)?

I have enabled debug logging in my pipeline run for Terraform, and I can tell these errors are surfaced by the azurerm_key_vault_secret in my case.

2023-03-21T22:44:32.361Z [ERROR] provider.terraform-provider-azurerm_v3.48.0_x5: Response contains error diagnostic: diagnostic_severity=ERROR diagnostic_summary="Unable to determine the Resource ID for the Key Vault at URL "https://kv-redacted-jutp.vault.azure.net/"" tf_proto_version=5.3 tf_resource_type=azurerm_key_vault_secret tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_detail= tf_provider_addr=provider tf_req_id=ab559199-904c-528e timestamp=2023-03-21T22:44:32.361Z
2023-03-21T22:44:32.363Z [ERROR] vertex "module.client-basesetup.azurerm_key_vault_secret.connection-string-secret (destroy)" error: Unable to determine the Resource ID for the Key Vault at URL "https://kv-redacted-jutp.vault.azure.net/"

I have a few suspicions at this point that I'm trying to narrow down, but the debug logs aren't really giving an indication

• Some other destroy isn't locking the KeyVault when it should be (like an access policy removal?) causing a resource conflict; I'm doubting this however because the azurerm_key_vault_secret resource is provided through the data plane, as identified on keyvault: support for creating items over a nested endpoint #9738
• The removal of azurerm_private_endpoint resources connected to the KeyVault are disrupting access to the data plane for the secret removal (even though in my case public access through a firewall IP is still enabled)

I do know that when I re-run my destroy, AFTER my azurerm_private_endpoint resources have been removed, the remaining destroy steps succeed.

[jeffwmiles]

I've traced this particular issue a little further in my environment, and I think it is due to some unexpected behaviour with the Azure REST API.

When I closely inspect the debug logs of a destroy operation, I noticed two things:

1. During the refreshing state operation (of my KeyVault, and an access policy) the provider uses the Resource List By Resource Group api to return the KeyVault object:

module.default.azurerm_key_vault_access_policy.pipeline_spn: Refreshing state... [id=/subscriptions/subscriptionIDredacted/resourceGroups/rg-default-vnhs/providers/Microsoft.KeyVault/vaults/kv-default-vnhs/objectId/c90f4efb]
2023-03-23T19:20:38.690Z [DEBUG] provider.terraform-provider-azurerm_v3.48.0_x5: AzureRM Request:
GET /subscriptions/subscriptionIDredacted/resourceGroups/rg-default-vnhs/providers/Microsoft.KeyVault/vaults/kv-default-vnhs?api-version=2021-10-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 keyvault/2021-10-01 HashiCorp Terraform/1.3.7 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
X-Ms-Correlation-Request-Id: 72bb7e97
Accept-Encoding: gzip: timestamp=2023-03-23T19:20:38.690Z
2023-03-23T19:20:38.781Z [DEBUG] provider.terraform-provider-azurerm_v3.48.0_x5: AzureRM Response for https://management.azure.com/subscriptions/subscriptionIDredacted/resourceGroups/rg-default-vnhs/providers/Microsoft.KeyVault/vaults/kv-default-vnhs?api-version=2021-10-01:
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Thu, 23 Mar 2023 19:20:37 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-IIS/10.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Aspnet-Version: 4.0.30319
X-Content-Type-Options: nosniff
X-Ms-Client-Request-Id: 74104b8a
X-Ms-Correlation-Request-Id: 72bb7e97
X-Ms-Keyvault-Service-Version: 1.5.692.0
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11978
X-Ms-Request-Id: e25c0369
X-Ms-Routing-Request-Id: EASTUS:20230323T192038Z:670e3ba7

{"id":"/subscriptions/subscriptionIDredacted/resourceGroups/rg-default-vnhs/providers/Microsoft.KeyVault/vaults/kv-default-vnhs", - SNIPPED }

2. When the destroy action takes place for an azurerm_key_vault_secret resource, the provider uses the Resource List api to return the KeyVault object, with a filter on its resourceType and Name:

2023-03-23T19:20:58.144Z [DEBUG] module.aks-setup.azurerm_key_vault_secret.cluster_admin_passwd: applying the planned Delete change
2023-03-23T19:20:58.144Z [DEBUG] provider.terraform-provider-azurerm_v3.48.0_x5: AzureRM Request:
GET /subscriptions/subscriptionIDredacted/resources?%24filter=resourceType+eq+%27Microsoft.KeyVault%2Fvaults%27+and+name+eq+%27kv-default-vnhs%27&%24top=5&api-version=2020-06-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 resources/2020-06-01 HashiCorp Terraform/1.3.7 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
X-Ms-Correlation-Request-Id: 444a209e
Accept-Encoding: gzip: timestamp=2023-03-23T19:20:58.144Z
2023-03-23T19:20:59.072Z [DEBUG] provider.terraform-provider-azurerm_v3.48.0_x5: AzureRM Response for https://management.azure.com/subscriptions/subscriptionIDredacted/resources?%24filter=resourceType+eq+%27Microsoft.KeyVault%2Fvaults%27+and+name+eq+%27kv-default-vnhs%27&%24top=5&api-version=2020-06-01:
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Thu, 23 Mar 2023 19:20:58 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 444a209e
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11999
X-Ms-Request-Id: 472b5153
X-Ms-Routing-Request-Id: SOUTHCENTRALUS:20230323T192059Z:472b5153

{"value":[]}: timestamp=2023-03-23T19:20:59.072Z

Surprisingly, # 1 returns the object properties of the KeyVault, and # 2 returns a blank value, even though it is the same KeyVault! This is where the original error is coming from; the Secret resource has the ID reference, but during the destroy operation the lookup for the KeyVault doesn't return any values.

I have confirmed this both in Postman, and in the Resource Explorer. The latter for example, when I navigate through Subscriptions -> Subid -> Providers -> Microsoft.KeyVault I do not see my "kv-default-vnhs" at all, even though I can browse and manage it in the Azure Portal.

But if I navigate through the resource group, I do see it!

Even more strange, within the Resource List api (not scoped through a resource group) it returns KeyVaults that no longer exist (such as the "kv-*-vlnb" in the first image; not even in a Soft-Delete stage.

I cannot understand how this can be - a resource can exist and function, but only partially discover-able by the API, hidden if you are looking at the subscription scope instead of resource group? I will likely open a support ticket with Azure about this, but if there are any internal Hashicorp or Microsoft resources who see this, help would certainly be appreciated.

Edit: 20 minutes after typing this, while trying to gather screenshots for an Azure support ticket, my Postman queries magically showed the KeyVault and had removed the actual non-existent ones. My hunch is some kind of delay in replication of objects across regions, which is terrible for a CI/CD pipeline of a Terraform module that is doing an apply and destroy 😟

Edit 2: it appears this terraform provider issue is directly related to similar behaviour found in Azure CLI: Azure/azure-cli#23428

[immae1]

Same on my side; on a terraform destroy I get Error: Unable to determine the resource ID for the key vault at URL if the key vault was previously created in a terraform apply run. If I run terraform destroy again, the key vault is removed and destroy is successful. Right now this is a blocking issue for me as I rely on terraform apply plus destroy in my pipelines so that I can be sure that my development is always able to build and destroy the whole stack. FYI, I have already bumped to the versions azurerm: 3.48.0 and terraform: 1.4.1.

@jeffwmiles are you still facing this issue?
Greetz

[immae1]

@tschechniker anything new about this topic? Do you have a workaround?

[danieltroyer]

I've got this issue blocking me today. Azure Key Vaults, created on previous days so less likely to be propagation lag, refresh during terraform plan properly, but they fail with terraform destroy throwing error Error: Unable to determine the Resource ID for the Key Vault at URL ..... Even the plan stage of destroy shows the correct resource IDs for the secrets and vaults, but the apply stage fails with the error.

Adding to the fun, the state file shows the full and correct resource IDs but won't use them. I workaround would be an override command to allow the use of IDs from state rather than the API values that the user above reports are missing.

[w4rgrum]

I have the same issue with azurerm v3.78.0. Manually testing with curl showed propagation lag on the resources List api on my end (~15-20 mins), which causes the failure when the KV is new and a destroy is called (typical use case seems to be TF module tests).
Which is basically what was described in this previous comment: #19322 (comment)
The destroy probably need to use a different API to get this fixed.

[phoehnel]

It is interesting that the behavior seems to be random. We have pipelines with Terratest that regularly build and destroy KeyVaults and some destroys work fine, some exit with "unable to determine the Resource ID for the Key Vault at URL".

Our pipeline logs unfortunately only go back 6 months, but in this period I could not find any pipeline with this error until yesterday. Since yesterday every pipeline failed with it, even using the same provider version and tf-files that once worked.

[quanthereal]

I'm running Terraform v1.6.3 and azurerm v3.89.0 with similar setup.

Unfortunately I still run into this issue in my tests module trying to destroy a key vault and a key vault key even though it seems to be fixed with azurerm v3.83.

[dkirrane]

I'm hitting this issue with Terraform 1.6.5. It does seem to happen randomly.

    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.96.0"
    }

[Uqqasha]

Experiencing this same issue with Terraform 1.7.5 and azurerm 3.91.0, although it happens randomly and not on every terraform destroy run.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.