Intermittent Error: Unable to locate Storage Account "<<storage_account_name>>"!

[smushtaq]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

1.35.0
1.36.0
1.37.0
1.38.0
1.39.0

Description

I am getting this error intermittently when using azurerm provider 1.35.0, 1.36.0, 1.37.0, 1.38.0, 1.39.0. This issue doesnot occur on azurerm provider 1.34.0 and on earlier versions.

Anyone else had seen this issue. Below is the log for terraform plan command

2020-01-03T14:29:08.9014786Z ##[section]Starting: Terraform plan
2020-01-03T14:29:08.9018892Z ==============================================================================
2020-01-03T14:29:08.9018984Z Task         : Terraform
2020-01-03T14:29:08.9019106Z Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
2020-01-03T14:29:08.9019365Z Version      : 0.0.142
2020-01-03T14:29:08.9019475Z Author       : Microsoft Corporation
2020-01-03T14:29:08.9019548Z Help         : [Learn more about this task](https://aka.ms/AA5j5pf)
2020-01-03T14:29:08.9019668Z ==============================================================================
2020-01-03T14:29:09.0425642Z [command]/opt/hostedtoolcache/terraform/0.12.17/x64/terraform providers
2020-01-03T14:29:09.5124944Z .
2020-01-03T14:29:09.5126416Z └── provider.azurerm =1.39.0
2020-01-03T14:29:09.5126663Z 
2020-01-03T14:29:09.5157691Z [command]/opt/hostedtoolcache/terraform/0.12.17/x64/terraform plan -var-file=envs/dev/vars.auto.tfvars -out=tfplan -no-color -input=false
2020-01-03T14:29:11.1176401Z Refreshing Terraform state in-memory prior to plan...
2020-01-03T14:29:11.1179110Z The refreshed state will be used to calculate this plan, but will not be
2020-01-03T14:29:11.1179729Z persisted to local or remote state storage.
2020-01-03T14:29:11.1180065Z 
2020-01-03T14:29:13.9250714Z data.azurerm_resource_group.power: Refreshing state...
2020-01-03T14:29:13.9614330Z azurerm_key_vault.xxxxxxxxxx: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/Microsoft.KeyVault/vaults/mykv]
2020-01-03T14:29:13.9639469Z azurerm_storage_account.power_storage_account: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/Microsoft.Storage/storageAccounts/mysa]
2020-01-03T14:29:13.9644189Z azurerm_application_insights.power_app_insights: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/microsoft.insights/components/myai-we]
2020-01-03T14:29:13.9656039Z azurerm_app_service_plan.func_apps_consumption_plan: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/Microsoft.Web/serverfarms/mycsp]
2020-01-03T14:29:14.1513932Z azurerm_key_vault_access_policy.spn_access_policy: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/Microsoft.KeyVault/vaults/myvault/objectId/xxxxx-xxxx-xxxx-xxxxx]
2020-01-03T14:29:16.1417214Z 
2020-01-03T14:29:16.1418746Z Error: Unable to locate Storage Account "mysa"!
2020-01-03T14:29:16.1418814Z 
2020-01-03T14:29:16.1418851Z 
2020-01-03T14:29:16.2364297Z ##[error]Error: The process '/opt/hostedtoolcache/terraform/0.12.17/x64/terraform' failed with exit code 1
2020-01-03T14:29:16.2381103Z ##[section]Finishing: Terraform plan

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

data "azurerm_resource_group" "power" {
  name = var.resource_group_name
}

resource "azurerm_storage_account" "power_storage_account" {
  name                      = "${var.product_code}sa${var.env_code}${var.location_code_we}"
  resource_group_name       = data.azurerm_resource_group.power.name
  location                  = var.location
  account_tier              = var.storage_account_tier
  account_replication_type  = var.storage_account_replication_type
  account_kind              = "StorageV2"
  enable_https_traffic_only = true
  tags                      = var.default_tags  
}

resource "azurerm_application_insights" "power_app_insights" {
  name                = "${var.product_code}-df-${var.env_code}-ai-${var.location_code_we}"
  location            = var.location
  resource_group_name = data.azurerm_resource_group.power.name
  application_type    = "web"
  tags                = var.default_tags
}

Debug Output

Panic Output

Expected Behavior

It should locate the existing storage account

Actual Behavior

Error: Unable to locate Storage Account "<<storage_account_name>>"!

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[tcz001]

Recently we see a lot of issues introduced by this error,

I'm assuming it's a issue caused by Azure Storage account List API handling in terraform-azurerm-provider because of following reasons, will try to collect debug logs in places that reproducing this error to prove the assumption.

Behaviors:

1. It's a known issue reported by community github issue
2. it's causing a relative issue on child container state refresh, because of the error handling to "empty" account list response: github source code
   1. account list API is used in azurerm_storage_container
   2. account getProperties API is used in azurerm_storage_data_lake_gen2_filesystem
3. it's happening Intermittently, probably be a bug of Azure Storage API state inconsistency and misuse of HTTP response statusCode, maybe a network issue of agent (unlikely to be agent issue because there is no error message responded, but simply account == nil)

Short term workarround:

1. if you are using ADLS Gen2 Storage account, use azurerm_storage_data_lake_gen2_filesystem instead of azurerm_storage_container (because of the API issue)
2. if you see a Unable to locate try rerun the job, and check if it still happen
3. if you see a container creation failure due to 409 already exist, try terraform import to get back the state
4. if you see a follow-up issue different from what reported here, please respond to this thread

Potential long term solution:

1. report this issue to MSFT to fix the response status code and errors in response, instead of responding with an empty list
2. change the error handling in this Terraform provider, remove the "assuming removed, removing state" feature and use GetProperties for Storage Account existence checking

[Farhie]

As @tcz001 suggested, I am currently pushing this with MSFT account manager.

I'll put a PR in to move to using the more reliable getProperties API.

[tcz001]

We got some updates after troubleshooting with MSFT,
the List API will response 200 OK with empty list that causes this issue, when Azure Management Storage Provider Limits hit: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#storage-resource-provider-limits
it usually should response with 429, but the List accounts API is misbehaving on this scenario

[tcz001]

As MSFT support team mentions the List API we use is responding a pagination response, which is ignored by the terraform azurerm provider, a similar issue is addressed on other resource: https://github.com/terraform-providers/terraform-provider-azurerm/pull/7036/files

[nerddtvg]

I'm seeing similar but slightly different results. Using versions 2.20.0 and 2.31.1 we're receiving:

Error: Error retrieving Account "{{{ account name }}}" for Container "logs": Error retrieving next page of storage accounts: storage.AccountsClient#listNextResults: Failure sending next results request: StatusCode=429 -- Original Error: context deadline exceeded

We get a 429 ResourceCollectionRequestsThrottled instead of the improper 200, however we also do not receive the limit headers anymore either.

When performing the List operation manually via command line, it there is a nextLink only 4 times. I don't see how we're hitting a 100 request / 5 minutes limit so I truly think this is a problem on MS' side but can't prove it.

[adamrushuk]

I've started seeing this error using azurerm v2.52.0

[smushtaq]

I have now started seeing this error. its not intermittent and started happening everytime. No idea how i can get it fixed.

08T08:31:46.3425503Z Error: Error retrieving Account "*****": Error retrieving next page of storage accounts: storage.AccountsClient#listNextResults: Failure sending next results request: StatusCode=502 -- Original Error: context deadline exceeded

i am using terraform 0.14.5 and 0.14.10 and azurerm 2.41.0 and 2.54.0

[smushtaq]

I have now started seeing this error. its not intermittent and started happening everytime. No idea how i can get it fixed.

08T08:31:46.3425503Z Error: Error retrieving Account "*****": Error retrieving next page of storage accounts: storage.AccountsClient#listNextResults: Failure sending next results request: StatusCode=502 -- Original Error: context deadline exceeded

i am using terraform 0.14.5 and 0.14.10 and azurerm 2.41.0 and 2.54.0

This error has now been resolved after i raised a support ticket with Microsoft. they said

The product group team have identified the possible cause of the issue, the Storage RP returns 404 when ARM requests listing of storage accounts on specific region because the subscription is not registered with this region. They have manually re-registered it. This should mitigate the issue. Please give it a try and let us know if it is working now.

[masonhuemmer]

I am seeing this error on v2.62.1

I may have some background on this issue. Unsure if it is related.

I ran into issues where Terraform was unable to find specific resources where replication was enabled. After speaking with Microsoft support, they told me they were experiencing issues where metadata replication is taking longer to complete. This may result in errors that respond saying a resource may not exist or cannot be found. They told me this is a backend issue and are working to improve performance.

To enable faster replication, they will be migrating the backend services to using Cosmos DB by the end of the year. This should increase the time it takes for replication to complete. So when you run the APIs against your services, it will return with results.

[nerddtvg]

Hey all,

If you enable tracing for these actions, see if you're getting deferred (HTTP 429) messages. Our issue was due to the storage resource provider list quota of 100 operations per 5 minutes. This occurred due to some information security applications constantly polling for resources and eating up the quotas for everyone on the subscription. This is sort of what @tcz001 called out last year.

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#storage-resource-provider-limits

There's really no other way around this for Terraform to work. It has to list storage accounts for the entire subscription in order to find one and access its details like the blob and file endpoint URLs. That information is cached inside the same process, so it should only matter the first time an account is required during an operation and it hasn't been "seen" yet.

My advice is try to run az storage account list or Get-AzStorageAccount and see if you get everything you're expecting. The results can be partial if the 429 deferral happens mid-request. This can also present itself via the portal if you view all of them: https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts

If these functions work, try running them at the same time Terraform is attempting to search and see if that makes a difference.

I hope this helps.

[mlcooper]

I ran into this today, but I waited 10 minutes and then ran a terraform apply again and it was able to find my storage accounts - no errors. I took a trace of when I was getting this error and I did not see any http 429 errors.

[sinbai]

As @tcz001 suggested, I am currently pushing this with MSFT account manager.

I'll put a PR in to move to using the more reliable getProperties API.

@Farhie Have you submitted a PR to use the getProperties API already?

[RoweJHR]

Currently seeing this with a GRS RA account which was created over 2 hours ago. It is visible in the portal and looks fine however use 'az storage account list' finds nothing ...

What can be done to fix this?

[sidhukumar]

I tried using the version azurerm 2.85.0, 2.86.0, and 2.87.0 with all the versions I get an error on creating storage accounts also am deploying the resources using azure DevOps pipeline.

Error: Provider produced inconsistent result after apply

When applying changes to azurerm_storage_account.starter_storage_account,
provider "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced an
unexpected new value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

What can be done to fix this issue?

I have tried re-deploying. tried deploying it to different regions, changed the version, and deployed. every time I get the same error mentioned above.
check #14344 it has the code and output.

[nerddtvg]

Currently seeing this with a GRS RA account which was created over 2 hours ago. It is visible in the portal and looks fine however use 'az storage account list' finds nothing ...

What can be done to fix this?

@RoweJHR, if az storage account list doesn't show it, then Terraform probably can't find it either. That's not a provider issue.

[RoweJHR]

@nerddtvg indeed. I did some digging and this problem goes away if you select GRS. The GRS RA resource does not appear in resource manager either but is in the portal, once you change to GRS the account is visible from both.

I've raised a call with MS regarding it. Might be useful for reference if others bump into this in the future in this context. Thanks.

[github-actions]

This functionality has been released in v2.88.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.