v3.0.1: azurerm_key_vault: Cannot read the ressource with KeyVault: keyvault.BaseClient#GetCertificateContacts

[uncycler]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v1.1.7
azurerm v3.0.1

Affected Resource(s)

• azurerm_key_vault

Expected Behaviour

No error

Actual Behaviour

Cannot read the ressource with `KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

Steps to Reproduce

1. Create a keyvault with v2:

resource "azurerm_key_vault" "mykeyvault" {
  name                = "mykeyvault"
  sku_name            = "standard"
  location            = "myloc"
  resource_group_name = "myrg"
  tenant_id           = "mytenant"
  purge_protection_enabled = true
}

2. with v3, do a terraform plan

Important Factoids

Since v3, azurerm is trying to fetch "contact" from KeyVault and for some reason, the access is timing out. I've tried adding ManagedContacts permission but no luck.

Using same credentials, I can access the API without isuse.

az rest --method GET --uri "https://mykeyvault.vault.azure.net/certificates/contacts?api-version=7.1" --resource "https://vault.azure.net"
Not Found({"error":{"code":"ContactsNotFound","message":"Contacts not found"}})

• #0000

[noiano]

I have a similar issue. Simply referencing a certificate in a key vault causes the plan to fail

`data "azurerm_key_vault" "keyVault" {
  name                = "XXX"
  resource_group_name = "YYY"
}

# Get ID for latest certificate version in KV
data "azurerm_key_vault_secret" "certificate" {
  name         = "ZZZ"
  key_vault_id = data.azurerm_key_vault.keyVault.id
}

The plan command fails (after a long while) with the following error:

╷
│ Error: making Read request on Azure KeyVault Secret <name redacted>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: authorizer was not an auth.CachedAuthorizer for https://vault.azure.net
│ 
│   with data.azurerm_key_vault_secret.certificate,
│   on subscription.tf line 13, in data "azurerm_key_vault_secret" "certificate":
│   13: data "azurerm_key_vault_secret" "certificate" {
│ 
╵
Releasing state lock. This may take a few moments...

[kovaliovsg]

I have these both issues as well. Had to roll back to provider 2.99.0

[cmendible]

Just hit this one...

Terraform 1.1.7
azurerm 3.0.1

[nbjohnson]

I just ran into a similar issue with azurerm_key_vault_secret data source

Error: making Read request on Azure KeyVault Secret test-secret: autorest/Client#Do: 
Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token

Error: keyvault.BaseClient#GetKey: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

I see this for both v3.0.0 and v3.0.1

[manicminer]

Hi, thanks for reporting this. I'm looking at each report in this issue in turn and trying to reproduce across the authentication methods we support, however it would help greatly if anyone getting a timeout or "context deadline exceeded" error, were able to post a debug log (via gist) whilst this is happening.

You can do this by setting the TF_LOG environment variable to debug, for example with bash-compatibles:

TF_LOG=debug terraform apply

[SimonGasparKentico]

I got another error
Error: retrieving "contact" for KeyVault: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: authorizer was not an auth.CachedAuthorizer for https://vault.azure.net
I'm not sure if this is related to this problem or if the problem is on my side :)

We are using:
Az CLI 2.33.0
Terraform v1.1.7
Azurerm v3.0.1

[djryanj]

@manicminer I'm preparing a gist but does the azurerm provider team have a different public key than the terraform one? I'm going to redact the file but would still prefer to encrypt it.

[manicminer]

@SimonGasparKentico, @noiano I believe I may have found an issue with Azure CLI authentication and Key Vault.

[manicminer]

@djryanj If you wish to encrypt your log, please use the public key for the "terraform" Keybase user. Thanks!

[djryanj]

@manicminer here you are: https://gist.github.com/djryanj/2a0f0b13e2eb9891af850a0b4fdacd69

[manicminer]

@djryanj Many thanks, I'm having some trouble accessing that privkey at the moment - in the interest of expedience would you be able to re-encrypt with my personal key?

Nevermind, I was able to decrypt the log! It looks like you're getting the authorizer was not an auth.CachedAuthorizer for https://vault.azure.net error, can you confirm if you're using Azure CLI or MSI authentication? We should have a fix for this shortly.

[manicminer]

@uncycler, @kovaliovsg, @cmendible, @nbjohnson Would you be able to share a debug log for when you're seeing the "context deadline exceeded" error for a Key Vault related resource? Thanks!

[djryanj]

@manicminer glad you found the key, I promptly broke my terraform config after hitting this bug (developing something else, this was non-blocking for the moment) and also (like an idiot) deleted the debug log so couldn't reproduce quickly enough.

I am using Azure CLI authentication.

[cmendible]

Sorry no logs since I reverted back.

I'm also using Azure CLI authentication

[eoly]

@manicminer

Here are my TF_DEBUG logs. I have the context deadline exceeded issue too.

Using Azure CLI auth.

https://gist.github.com/eoly/702d9e6da370cd946372bce3677db830

[derekwilliamsliquidx]

I am hitting this error as well

Terraform version 1.1.6
Azure CLI Version 2.26.1
AzureRM Provider Version = 3.0.1

Reverting back to 2.99 corrected the issue.

[manicminer]

@eoly Thanks for the log, that's super helpful 👍

Anyone else, please kindly refrain from posting +1 comments and instead upvote the issue. Debug logs are always happily received!

[noiano]

I'm actually using a service principal to authenticate all actions against my subscription ...

I'd also like to add that, using azurerm 3.0.2, the terraform plan command which was previously stuck and ended up in error now goes through.

It's only much slower.

[nbjohnson]

Even after upgrading to v3.0.2, I am still getting a similar, but not exactly the same error. I am attempting to do a data source lookup for a key vault secret. This works on versions < 3.0.0, but is now just broken. If there is a breaking change that I am missing that could be affecting this let me know, or is this a bug related to the other errors users are reporting?

:51:05.293Z [ERROR] vertex "module.storage_account[\"<storage account name>\"].azurerm_key_vault_key.this" error: keyvault.BaseClient#GetKey: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
:51:05.294Z [ERROR] vertex "module.storage_account[\"<storage account name>\"].azurerm_key_vault_key.this" error: keyvault.BaseClient#GetKey: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
:51:05.294Z [ERROR] vertex "module.storage_account.azurerm_key_vault_key.this (expand)" error: keyvault.BaseClient#GetKey: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
:53:35.590Z [ERROR] vertex "data.azurerm_key_vault_secret.lookups[\"<secret local var name>\"]" error: making Read request on Azure KeyVault Secret <secret name>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token for <vault url>
:53:35.590Z [ERROR] vertex "data.azurerm_key_vault_secret.lookups" error: making Read request on Azure KeyVault Secret <secret name>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token for <vault url>
:53:35.591Z [ERROR] vertex "data.azurerm_key_vault_secret.lookups (expand)" error: making Read request on Azure KeyVault Secret <secret name>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token for <vault url>

For initial Azure auth I am using CLI

[djryanj]

@nbjohnson - try doing an az logout and az login. I didn't see a context deadline exceeded like in your error but did get something related to failing to get an MSAL token this morning, and this fixed it. Note my error may not have been the same as yours, but it's worth a try.

[nbjohnson]

@djryanj Thanks for the suggestion, but unfortunately that didn't fix my issue. Still keep getting failed to acquire MSAL token for at least key_vault_key creation and key_vault_secret data resource

[manicminer]

@nbjohnson Thanks for the feedback. I've been unable to reproduce that error, which is related to a failure invoking az-cli to acquire an access token. I would suggest upgrading to the latest version of Azure CLI, and deleting your ~/.azure directory before signing in again. I will look at ways we can improve the error reporting here.

[mark-strasser-nnl]

With USGovernment environment, the example key vault fails to get created or read from state, even with the latest versions of Terraform or azurerm.

│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
│
│   with azurerm_key_vault.example,
│   on main.tf line 15, in resource "azurerm_key_vault" "example":
│   15: resource "azurerm_key_vault" "example" {

See relevant gist.

Terraform versions:

$ terraform version
Terraform v1.1.8
on windows_amd64
+ provider registry.terraform.io/hashicorp/azurerm v3.1.0

[bgrainger]

I believe this issue may be related to use_msal = true (which is now on by default in azurerm >= 3.0).

Works: v2.99 with use_msal= false

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "2.99"
    }
  }
}

provider "azurerm" {
  features {}

  use_msal = false
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "this" {
  name     = "V299ADAL"
  location = "westus2"
}

resource "azurerm_key_vault" "this" {
  name                = "v299adal-kv"
  location            = resource.azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  tenant_id           = data.azurerm_client_config.current.tenant_id
  sku_name            = "standard"
}

Output:

azurerm_key_vault.this: Still creating... [2m30s elapsed]
azurerm_key_vault.this: Creation complete after 2m34s [id=/subscriptions/***/resourceGroups/V299ADAL/providers/Microsoft.KeyVault/vaults/v299adal-kv]

Fails: v2.99 with use_msal = true

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "2.99"
    }
  }
}

provider "azurerm" {
  features {}

  use_msal = true
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "this" {
  name     = "V299MSAL"
  location = "westus2"
}

resource "azurerm_key_vault" "this" {
  name                = "v299msal-kv"
  location            = resource.azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  tenant_id           = data.azurerm_client_config.current.tenant_id
  sku_name            = "standard"
}

Output:

azurerm_key_vault.this: Still creating... [7m30s elapsed]
╷
│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
│
│   with azurerm_key_vault.this,
│   on main.tf line 23, in resource "azurerm_key_vault" "this":
│   23: resource "azurerm_key_vault" "this" {
│
╵

FWIW, I'm not getting a failure to create this key_vault with azurerm 3.1.0. However, I wanted to paste my findings above since it seemed to be the exact same error message from the OP, and in case it provides a clue to where the problem may be.

[manicminer]

There is a related issue #16291 which affects USGovernment users when managing Key Vaults (or any resource that uses the Key Vault API). This will be fixed in next week's release.

Is anyone here not using the USGovernment cloud and experiencing a context deadline exceeded error when trying to manage or reference a Key Vault?

[xinity]

@manicminer i do and i'm pretty sure i'm not the only one.
I'm using "plain" Azure Subscription not a USGovernment

[mkemmerz]

Any update on this? Same error with AzureRm 3.1 on our side.

[tombuildsstuff]

Per @manicminer's comment above this has been fixed in more recent versions of the AzureRM Provider, we're currently on version v3.5.0 - if you're still encountering this issue when using v3.5.0 please let us know, but I'm going to close this out as fixed at the moment since we believe this has since been fixed.

Thanks!

[mkemmerz]

@tombuildsstuff thanks for the reply. We are having this issue on the global Azure cloud so maybe this issue doesn't really fit for me because it is only for the USGov Cloud. Using AzureRM 3.5. doesn't fix my issue. As we are using a Private Endpoint for the connection it seems more related to #9738

[kevinwedwards]

We too are having this issue on AzureRM 3.7. global azure cloud (non USGov).

[elongstreet88]

Same, seems like our general performance of terraform/azurerm provider have drastically reduced lately and we are seeing the context deadline issues as well (none gov as well).

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.