azurerm_kusto_cluster_customer_managed_key - property not working.

[liuwuliuyun]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.3

AzureRM Provider Version

3.21.1

Affected Resource(s)/Data Source(s)

azurerm_kusto_cluster_customer_managed_key

Terraform Configuration Files

provider "azurerm" {
  features {
    # key_vault {
    #   purge_soft_delete_on_destroy       = true
    #   purge_soft_deleted_keys_on_destroy = true
    # }
  }
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "test" {
  name     = "yunliutest122"
  location = "east us"
}

resource "azurerm_user_assigned_identity" "test" {
  name                = "acctest"
  resource_group_name = azurerm_resource_group.test.name
  location            = azurerm_resource_group.test.location
}

resource "azurerm_kusto_cluster" "test" {
  name                = "acctestkc"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name

  sku {
    name     = "Dev(No SLA)_Standard_D11_v2"
    capacity = 1
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.test.id]
  }
}

resource "azurerm_key_vault" "test" {
  name                     = "acctestkvyunliu2235"
  location                 = azurerm_resource_group.test.location
  resource_group_name      = azurerm_resource_group.test.name
  tenant_id                = data.azurerm_client_config.current.tenant_id
  sku_name                 = "standard"
  purge_protection_enabled = true
}

resource "azurerm_key_vault_access_policy" "cluster" {
  key_vault_id = azurerm_key_vault.test.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = azurerm_user_assigned_identity.test.principal_id

  key_permissions = ["Get", "UnwrapKey", "WrapKey"]
}

resource "azurerm_key_vault_access_policy" "client" {
  key_vault_id = azurerm_key_vault.test.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = data.azurerm_client_config.current.object_id

  key_permissions = [
    "Create",
    "Delete",
    "Get",
    "List",
    "Purge",
    "Recover",
  ]
}

resource "azurerm_key_vault_key" "test" {
  name         = "test"
  key_vault_id = azurerm_key_vault.test.id
  key_type     = "RSA"
  key_size     = 2048
  key_opts     = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]

  depends_on = [
    azurerm_key_vault_access_policy.client,
    azurerm_key_vault_access_policy.cluster,
  ]
}

resource "azurerm_kusto_cluster_customer_managed_key" "test" {
  cluster_id    = azurerm_kusto_cluster.test.id
  key_vault_id  = azurerm_key_vault.test.id
  key_name      = azurerm_key_vault_key.test.name
  key_version   = azurerm_key_vault_key.test.version
  user_identity = azurerm_user_assigned_identity.test.id
}

Debug Output/Panic Output

Plan: 1 to add, 0 to change, 0 to destroy.
azurerm_kusto_cluster_customer_managed_key.test: Creating...
azurerm_kusto_cluster_customer_managed_key.test: Still creating... [10s elapsed]
azurerm_kusto_cluster_customer_managed_key.test: Still creating... [20s elapsed]
azurerm_kusto_cluster_customer_managed_key.test: Still creating... [30s elapsed]
╷
│ Error: waiting for completion of Kusto Cluster Update "acctestkc" (Resource Group "yunliutest122"): Code="Failed" Message="Internal Server Error"
│
│   with azurerm_kusto_cluster_customer_managed_key.test,
│   on test.tf line 84, in resource "azurerm_kusto_cluster_customer_managed_key" "test":
│   84: resource "azurerm_kusto_cluster_customer_managed_key" "test" {
│
╵

Expected Behaviour

No response

Actual Behaviour

No response

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[liuwuliuyun]

This is also related to accTestFailure TestAccKustoClusterCustomerManagedKey_userIdentity

[liuwuliuyun]

Currently, I am able to determain that in a PATCH call to service, the server returns 202. But after the PATCH call, the server returns an "Internal Server Error" without any other message for the wait for completion async call.

[liuwuliuyun]

Example REST call response caught in REST API call.
PATCH https://management.azure.com/subscriptions/xxxxxxxxx/resourceGroups/acctestRG-220921084043399667/providers/Microsoft.Kusto/clusters/acctestkcmwqgg?api-version=2022-02-01 HTTP/1.1

HTTP/1.1 202 Accepted

GET https://management.azure.com/subscriptions/xxxxxxxxx/providers/Microsoft.Kusto/locations/West%20Europe/operationResults/xxxxx?api-version=2022-02-01 HTTP/1.1

{"id":"/subscriptions/xxxxxxxxx/providers/Microsoft.Kusto/locations/West Europe/operationresults/xxxxx","name":"xxxx","status":"Failed","startTime":"2022-09-21T00:53:31.6614027Z","endTime":"2022-09-21T00:53:51.1147303Z","percentComplete":1.0,"properties":{"operationKind":"ServiceConfigurationAlter","provisioningState":"Failed","operationState":"Failed"},"error":{"code":"Failed","message":"Internal Server Error"}}

[Lucasjuv]

Hi guys I found the same error on my end, the infrastructure is almost the same the only difference is that the cluster is inside a VNET. I also found a workaround using SystemAssigned, UserAssigned identity type on the cluster and giving access to the system assigned identity to the keyvault and it worked just fine.
I hope this can help the investigation on this issue. From the looks of it it seems like the problem is on the cluster implementation when using user assigned identities to access the keyvault.

[Lucasjuv]

My workaround unfortunately create a cyclic dependency with the access policy and cluster system assigned identity and the destruction can't happen normally.

[Lucasjuv]

Today I have tested through the portal the data explorer customer managed key indeed does not work with user assigned identities. There must be something wrong in the cluster configuration.

[liuwuliuyun]

This could be caused by the API underlying, will test this again once upgraded API is used.

[liuwuliuyun]

This is a bug caused by underlying API and Microsoft Service Team is working on the fix.

[codersinthestorm]

Does anyone have the link to an existing microsoft ticket/bug? Or a forum/ discussion thread?

[liuwuliuyun]

Hi @codersinthestorm, here is the ICM ticket I raised MS ICM (Internal Only) Kusto team said it was an issue with the manifest and will roll back. But the problem still exsits till now.

[codersinthestorm]

Thanks a lot! We're also hitting this issue, and weirdly enough its with a System-Assigned identity for the cluster- which has worked fine for us in other similar clusters.

[codersinthestorm]

Just to update here if anyone is still interested in the ticket, the product team deployed a new build apparently, and that has fixed the issue for us. You might wanna try it again once.

[rcskosir]

@liuwuliuyun Thanks for taking the time to submit this issue. Have you been able to test again to see if you are still having this issue, or if it was resolved by the new build @codersinthestorm mentioned?

[liuwuliuyun]

@rcskosir the issue could be resolved by setting keyvault policy permission with "GetRotationPolicy", "SetRotationPolicy". For future readers with similar issue try this link

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.