State file is not saved locally when uploading to remote backend fails

[roman-vynar]

Terraform does not save a state file locally when uploading to the remote backend fails. In my case I was creating 104 resources on AWS and after 1 hour of working and completion, suddenly, terraform failed to upload the resulting state file to S3 with 403 error to the bucket (with no reason, I didn't have the access problems) but important here, it left me without a state file at all - no local copy unfortunately. So basically, I have got a ton of orphaned AWS resources.

Terraform Version

v0.9.3, v0.9.4

Affected Resource(s)

Uploading a state file to the remote backend S3

Expected Behavior

From https://www.terraform.io/docs/backends/state.html#state-storage

In the case of an error persisting the state to the backend, Terraform will write
the state locally. This is to prevent data loss. If this happens the end user must 
manually push the state to the remote backend once the error is resolved.

Actual Behavior

Data loss, you are left with the orphaned resources :(

Steps to Reproduce

Here my-bucket should be a real bucket.

$ cat 1.tf
terraform {
  backend "s3" {
    bucket  = "my-bucket"
    key     = "roman/roman.tfstate"
    region  = "us-west-2"
    encrypt = true
  }
}

resource "random_id" "random_password" {
  byte_length = 1

  provisioner "local-exec" {
    command = "sleep 30"
  }
}
$ terraform init -backend=true -get=true
$ terraform plan
...

+ random_id.mysql_password
    b64:         "<computed>"
    b64_std:     "<computed>"
    b64_url:     "<computed>"
    byte_length: "1"
    dec:         "<computed>"
    hex:         "<computed>"


Plan: 1 to add, 0 to change, 0 to destroy.

Now run the below command and while it is sleeping on provisioning you will have 30s to quickly edit your /etc/hosts and add a line like this 1.2.3.4 my-bucket.s3-us-west-2.amazonaws.com to repoint DNS to something fake and have the upload to remote backend to fail:

$ terraform apply
random_id.mysql_password: Creating...
  b64:         "" => "<computed>"
  b64_std:     "" => "<computed>"
  b64_url:     "" => "<computed>"
  byte_length: "" => "1"
  dec:         "" => "<computed>"
  hex:         "" => "<computed>"
random_id.mysql_password: Provisioning with 'local-exec'...
random_id.mysql_password (local-exec): Executing: /bin/sh -c "sleep 30"
random_id.mysql_password: Still creating... (10s elapsed)
random_id.mysql_password: Creation complete (ID: Sg)
Failed to save state: Failed to upload state: RequestError: send request failed
caused by: Put https://my-bucket.s3-us-west-2.amazonaws.com/roman/roman.tfstate: x509: certificate is valid for *.domain.com, not my-bucket.s3-us-west-2.amazonaws.com

No matter what the error is, it could be 403, 404 or in my case certificate problems. Terraform only saves a current state file to terraform.tfstate.backup locally which is a state file before resource creation, but after they are created and upload fails - nothing is saved neither to the current folder or to .terraform/ and you get a data loss.

Thanks, please let me know if you need any further information.

[abrefort]

I can confirm that all Terraform 0.9.x versions are impacted.
This is especially problematic when using AssumeRole credentials as those expire after an hour.

Terraform 0.8 allowed us to just run a second apply with new credentials to resume operations as the local and updated version of the state was synced to the remote storage at the start of the second run.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.