Unable to force-unlock state with azurerm

[CoRfr]

The error looks like that:

terraform force-unlock -force <id>
Failed to unlock state: failed to delete lock info from metadata: storage: service returned error: StatusCode=412, ErrorCode=LeaseIdMissing, ErrorMessage=There is currently a lease on the blob and no lease ID was specified in the request.
RequestId:6c8ebe08-001e-00d8-71c9-789fb8000000
Time:2017-12-19T12:58:50.4189788Z, RequestInitiated=Tue, 19 Dec 2017 12:58:49 GMT, RequestId=6c8ebe08-001e-00d8-71c9-789fb8000000, API Version=2016-05-31, QueryParameterName=, QueryParameterValue=
Lock Info:
  ID:        <id>
  Path:      terraform-state/dev.terraform.tfstate
  Operation: OperationTypeApply
  Who:       @jenkins
  Version:   0.11.1
  Created:   2017-12-19 12:53:08.462280312 +0000 UTC
  Info:

Terraform Version

0.11.1

Affected Resource(s)

Terraform State ?

Expected Behavior

terraform should unlock the state, and it should be possible to run terraform apply again after.

Actual Behavior

terraform raises an exception and the lock is still present.

Steps to Reproduce

1. terraform apply
2. Ctrl+C
3. terraform apply (prints an error with the unlock id)
4. `terraform force-unlock -force

References

Looks like this user had the same issue: https://groups.google.com/forum/#!topic/terraform-tool/we21XjC58pI

Original issue was opened on hashicorp/terraform-provider-azurerm#640

[r7vme]

Also affected by this issue, any workaround?

[djmitche]

I was able to unlock this by going into the Azure portal and "breaking" the lease on the affected blob. It's not exactly quick :/

[r7vme]

Thanks @djmitche , just did that 👍

[HighwayofLife]

Failed to unlock state - Workaround

When working with Terraform remote states in Azure, sometimes the blob lease can become locked during a long running process and not released. This also occurs when you cancel an operation midway through a terraform run.

Error: Error locking state: Error acquiring the state lock: 2 error(s) occurred:

* storage: service returned error: StatusCode=409, ErrorCode=LeaseAlreadyPresent, ErrorMessage=There is already a lease present.
RequestId:6eefbd6c-001e-0007-5d3e-a2128d000000
Time:2018-02-10T07:11:52.7714549Z, RequestInitiated=Sat, 10 Feb 2018 07:11:52 GMT,
RequestId=6eefbd6c-001e-0007-5d3e-a2128d000000, API Version=2016-05-31, QueryParameterName=, QueryParameterValue=
* blob metadata "terraformlockid" was empty

In this case, something went wrong during a previous terraform run and we don't even have a lock ID to work with. When there is already a lease present, typically it would print out a lock ID and you may us it to "force unlock" the Terraform state.

The command to do this within Terraform is:

terraform force-unlock <lock-id-guid>

To get the lock ID, just use a random number, and it will print out the lock ID, and run terraform force-unlock <lock-id> again with that lock ID.

terraform force-unlock 6eefbd6c-001e-0007-5d3e-a2128d000000
Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may be still be in use. Only 'yes' will be accepted to confirm.

  Enter a value: yes

Failed to unlock state: failed to retrieve lock info: blob metadata "terraformlockid" was empty

But it didn't work

Force unlocking the Terraform state within an azure backend doesn't usually work, so you will have to unlock/break the blob lease manually.
You can break the blob lease through the Azure portal, however, here are the steps for breaking the blob lease through the Azure CLI.

You will need:

1. An Azure login or service principal account (SPN).
2. The access key and account name for the storage account.
3. The container name and blob name containing the lease.

Step 1:

Login with your azure contributor account or service principal

az login --service-principal -u <client-id> -p <client-secret> --tenant <tenant-id>

Step 2:

container key = blob-name

az storage blob lease break -b <blob-name> -c <container-name> \
    --account-name <storage-account-name> --account-key <access-key>

Hazzah! Unlocked!!

You can see a list of blobs and if any of them have a lease/lock associated with them by running:

az storage blob list -c <container-name> --account-name <storage-account-name> --account-key <access-key>

[pavantata]

Started seeing the same above error today with AzureRM backend. Will appreciate if you can help me towards a solution as it is a showstopper for me implementing this at my workplace.

Got the latest 0.11.7 version of terraform.

I am new to Go language but can still contribute to fix the defect if you can guide.

[bfleming-ciena]

I'm seeing the same error, but it's coming from my terraform_remote_state object. Does TF lock that state file in order to read in the data????

blob metadata "terraformlockid" was empty

[bfleming-ciena]

Also, it would be helpful if the timeout option worked. As I mentioned, I noticed that when reading data sources from azure using terraform_remote_state, it locks the file. If I could have a time-out where it would re-try the acquire lock, it would solve my particular case where the problem occurs when using the same data source between multiple projects and running apply on them at the same time.

[rdonkin]

@stonefury @tombuildsstuff - my comment here on the closed #1871 may be relevant here, as it seems Terraform does indeed take locks when reading remote state as a data source.

[amuniz]

FTR: This also happens with Google Cloud. The workaround is to remove the state bucket from the storage. The bucket name to delete can be obtained from the local terraform state file (at ./terraform/terraform.tfstate), this file needs to be removed too to workaround the issue.

[TWCurry]

Just thought I'd add - still an issue for AzureRm at the moment. Only workaround was to break lease, thanks @djmitche. Would be useful for a proper fix, as this could break automation solutions

[pcj]

Any progress on this? This is kind of a big deal.

[ghost]

The issue is still present (using latest azurerm + terraform 0.11.8). Is there any work being done by now to fix it?

[tombuildsstuff]

@JackTheRipper this has been fixed and is available in the Terraform 0.12 Beta

[OffColour]

I've still got this issue in Terraform 12.5.

The odd thing is that calling force-unlock give the same message as trying to get the lock.

terraform12 force-unlock -force d355e46c-3982-f1bd-7ec3-7e14e04d5e76
Failed to load state: failed to lock azure state: storage: service returned error: StatusCode=409, ErrorCode=LeaseAlreadyPresent, ErrorMessage=There is already a lease present.
RequestId:36e4fde1-501e-002e-6d05-424366000000
Time:2019-07-24T09:55:01.8995957Z, RequestInitiated=Wed, 24 Jul 2019 09:55:01 GMT, RequestId=36e4fde1-501e-002e-6d05-424366000000, API Version=2016-05-31, QueryParameterName=, QueryParameterValue=
Lock Info:
ID: d355e46c-3982-f1bd-7ec3-7e14e04d5e76
Path: dev/dev.terraform.tfstateenv:commercial-dev-03
Operation: OperationTypeApply
Who: xxxxxxxxxxxxxxxxxxxxxxxxxxx
Version: 0.12.5
Created: 2019-07-24 08:04:03.5895146 +0000 UTC
Info:

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.