az keyvault list does not show newly created key vault

[mmckechney]

This is autogenerated. Please review and update as needed.

Describe the bug

az keyvault list does not show newly created key vault

Command Name
az keyvault list

Errors:
empty list when the key vault should be listed

[]

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

1. Create a new key vault via CLI:
   az keyvault create -g {rg} -n {vault name}
2. List key vaults in the resource group:
   az keyvault list -g {rg}

This gives an empty result: []

However, show does work and give the details of the key vault:
az keyvault show -g {rg} -n {vault name}

Expected Behavior

Key vault information should be shown

Environment Summary

Windows-10-10.0.22000-SP0
Python 3.10.5
Installer: MSI

azure-cli 2.38.0 *

Extensions:
aks-preview 0.5.91
containerapp 0.3.7
storage-preview 0.8.3

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

[mmckechney]

Please note, I upgraded to azure-cli 2.39 and the same problem occurred:

azure-cli                         2.39.0

core                              2.39.0
telemetry                          1.0.6 *

Extensions:
aks-preview                       0.5.92

Dependencies:
msal                            1.18.0b1
azure-mgmt-resource             21.1.0b1

[yonzhan]

@evelyn-ys for awareness

[evelyn-ys]

This could be server delay. Need Keyvault service or ARM service check.

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[navba-MSFT]

@mmckechney Thanks for reaching out to us and reporting this issue. We are looking into this issue and we will provide an update.

[navba-MSFT]

@mmckechney I was able to run the below command to create a KV and was able to list it. See below

az account set --subscription "XXXXXXXXXXXXXXXXX"

az group create --name "MyRGkv" --location "EastUS"

az keyvault create --name "Mykv" --resource-group "MYRGkv" --location "EastUS"

C:>az keyvault list -g MYRGkv
[
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXX/resourceGroups/MYRGkv/providers/Microsoft.KeyVault/vaults/Mykv",
"location": "EastUS",
"name": "Mykv",
"properties": {
"accessPolicies": [
{
"applicationId": null,
"objectId": "c4c82ef4-cc5e-456f-ba01-17126727d9e6",
"permissions": {
"certificates": [
"all"
],
"keys": [
"all"
],

I have tried with both AzCLI 2.38.0 and using 2.39.0 version and both worked.

If the above commands doesn't help, Please run the list command again with --debug switch and share the output here. Awaiting your reply.

[mmckechney]

@navba-MSFT - this works for me today. I wonder if there was some latency in Azure yesterday unrelated to the CLI. In any case, I will close this issue. Thanks.

[tylercarper]

In our org, we are hitting this quite frequently. We have a pipeline for putting up dev clusters that creates keyvaults in one step and later verifies existence with az keyvault list.

We create the keyvault (into an also newly created resourceGroup) with:
az keyvault create --name $keyVaultName --resource-group $resourceGroupName --location $configLocationName

Later we run and see exactly the behavior that @mmckechney describes:
az keyvault list --subscription $azureSubscriptionId --resource-group $resourceGroupName
returns: []

Occasionally, az keyvault list will update immediately and we see the new kv. but (seemingly) more and more often, we see it take hours (sometimes a full day) to return the keyvault from az keyvault list

In out case, using az keyvault show seems like a more reliable way to test for the key vaults existence.

[tylercarper]

@navba-MSFT what is the proper protocol to re-open this ticket? I thought maybe i'd ask instead of creating a dupe...

Thanks!

[rkisliak]

@navba-MSFT

I have the same issue. I create new KeyVaults in a pipeline using:
az keyvault create --name <kv_name> --resource-group <rg_name> --enable-rbac-authorization <bool>
And I am able to see KeyVault in UI (Azure Portal), but CLI commands like:
az keyvault list or az keyvault show --name <kv_name>
shows nothing, but only after 20-30 min (sometimes less, sometimes more) I see the correct output.

@tylercarper perhaps any luck to fix or workaround already? :)

[mmckechney]

Reopening based on experience of @tylercarper and @rkisliak

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @RandalliLama, @schaabs, @jlichwa.

Issue Details

---

This is autogenerated. Please review and update as needed.

Describe the bug

az keyvault list does not show newly created key vault

Command Name
az keyvault list

Errors:
empty list when the key vault should be listed

[]

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

1. Create a new key vault via CLI:
   az keyvault create -g {rg} -n {vault name}
2. List key vaults in the resource group:
   az keyvault list -g {rg}

This gives an empty result: []

However, show does work and give the details of the key vault:
az keyvault show -g {rg} -n {vault name}

Expected Behavior

Key vault information should be shown

Environment Summary

Windows-10-10.0.22000-SP0
Python 3.10.5
Installer: MSI

azure-cli 2.38.0 *

Extensions:
aks-preview 0.5.91
containerapp 0.3.7
storage-preview 0.8.3

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

Author:	mmckechney
Assignees:	-
Labels:	KeyVault, Service Attention, question, needs-team-attention, Auto-Assign
Milestone:	Backlog

[navba-MSFT]

Adding Service team to provide pointers on this.

@RandalliLama, @schaabs, @jlichwa Could you please look into this once you get a chance ?

[rkisliak]

Adding Service team to provide pointers on this.

@RandalliLama, @schaabs, @jlichwa Could you please look into this once you get a chance ?

Any luck to check this? :)

[MarioCakeDev]

I can confirm. This is happening in our organization aswell.

We are using Azure CLI Version 2.41.0.

We have Azure Devops Pipelines and try to deploy KeyVaults and set policies on them directly afterwards.
This only happens on the Azure CLI but not if the Rest API is called directly.

Two of my colleagues had this error multiple times on their machines aswell. We could not determine the reason. An DNS Cache Clear didn't help either.

Once they executed az keyvault list without a --query parameter it worked for them. So we changed our Pipelines to execute it without the query. It did not solve the problem.

It even happens randomly while the script is running and already set policies successfully.

We have a script which accepts multiple policies at once and we loop over them and set policies on the keyvault. On the first policies it worked but stopped working afterwards.

Once it stops working it won't start working again. We introduced retries in the script with delay between the retries. Up to a total of 10 retries with 2 seconds delay in between.

When the Pipeline Job is rerun it works again (probably because another agent is used). This error is somehow bound to the machine for some duration of time.

[jlichwa]

It can take some time until all key vaults list are propagated across regions. The only way to get Key Vault is to retrieve it directly or if list is needed you will need retry for time till vaults list is propagated.

[MarioCakeDev]

By "retrieve it directly" do you mean az keyvault show? When the KeyVault is not found with az keyvault list it won't show if az keyvault show is used.

We have a script like that (simplified):

param (
    $KeyVault,
    $SecretPersmissions
)

foreach ($secretPermission in $SecretPermissions)
{
    for($try = 0; $try -le 10; $try++){
        # Including some error handling which checks if the policy was set correctly (removed for brevity)
        
        az keyvault set-policy `
            --name $KeyVault `
            --secret-permissions $secretPermission.Permissions `
            --object-id $secretPermission.ObjectId

        # Sleep 2 seconds on error, break from inner loop if successful
    }
}

Consider we have 10 Secret Permissions. The first three work perfektly fine but then we get the error "The Vault '$KeyVault' not found within subscription" and it does not go away within the 10 retries. And we can't wait half an hour for our pipeline to complete this one step. But this does not happen always. Only sometimes! We don't know why. When rerunning the pipeline step it usually just works.

If the az keyvault set-policy is replaced with the corresponding REST Api Call, it works without a problem.

Why do we need to wait until its propagated if we could already set the key vault policies successfully?

Also: We have the same problem on KeyVaults which were created way before.

[jlichwa]

@MarioCakeDev Key Vault show is directly calling the vault created. The list is using cached list of vaults in ARM which takes a while to propagate across regions. We can't wait since we don't control of ARM service propagation of resources.

In terms az keyvault set-policy that should not be related to listing vaults in subscription. @yonzhan can you check how set-policy is implemented it should call same rest api? It is likely different issue.

On the other note you should like plan to move from access policies to RBAC (we are moving away from old legacy access policy model).

[MarioCakeDev]

I could narrow down the problem to the https://management.azure.com/subscriptions/{subscriptionId}/resources Endpoint.

If I try to set the KeyVault Policy with the Azure CLI only based on the KeyVault name, I think this endpoint is called to find the KeyVault. This endpoint does not return the requested KeyVault. I don't know why it does not return the KeyVault.

If I add the resource group to the command everything works again.

I have wondered why the KeyVault can be found when I open the portal so I have looked into the network requests and it seems like the portal is using the https://management.azure.com/providers/Microsoft.ResourceGraph/resources endpoint.

[jlichwa]

@MarioCakeDev yes Portal is using that endpoint same as list which is trying to find vault (which can take some time to propagate0, but if you call key vault by name directly, it is calling Key Vault directly.

If you use show with resourcegroup as parameter I see in debug that az cli is directly calling vault.

I'm a bit lost how from listing the vault you are moving to Access Policy - how is that connected. Are you trying to create Access Policy after Vault is created - any reason to not do it in the same time?

[jlichwa]

@MarioCakeDev to summarize based on questions above. The endpoint (https://management.azure.com/subscriptions/{subscriptionId}/resources) is used to list resources across regions, which is using ARM caching across regions, and it takes some time to propagate.
This endpoint should not be used for general operation on key vault.

With resource group and default subscription it can construct ARM resource it and call it directly and that is why it works.

[MarioCakeDev]

@jlichwa When I call az keyvault set-policy --name {kvname} --secret-permissions get --objectId {objId} --debug the command itself is calling https://management.azure.com/subscriptions/{subscriptionId}/resources to find the key vault.
I wanted to make sure the KeyVault does exist before setting the policies. So I called az keyvault list by myself.
But this command is using the same endpoint itself. (I tested az keyvault show --name {kvname} --debug, but it uses the endpoint aswell)
Before I knew that I tried getting the KeyVault using that endpoint itself. This is how I discovered that this endpoint does not work for us.
So I inspected what the portal is doing because the portal showed me the keyvault evne though the endpoint didn't and got the Microsoft.ResourceGraph endpoint.

I could use the az keyvault set-policy including the resource group in our usecase, yes. But that would introduce breaking changes in our code.
So I tried finding another way at first especially because the command is offering not using the resource group.

Are you trying to create Access Policy after Vault is created - any reason to not do it in the same time?

Because the KeyVault is one of the first resources which is created. After that the resources set their required access policies on the KeyVault themselfs. And at the current stage of our pipeline we cannot invert the dependencies.

[jlichwa]

@MarioCakeDev there is no workaround in that case. You need to know resource uri to call key vault. The find operation it is ARM across all resources in Azure and with that it is caching the list in every region, so there is a delay. There is nothing CLI team can do here for you.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!