aws-provider: "The input token is no longer current." when operating on multiple WAF objects #10335

keymon · 2016-11-24T10:21:28Z

AWS WAF implements a token based system to ensure that no concurrent changes are applied on the WAF rules. The API expects the API client to take care of synchronisation.

If two or more threads/workers request a token at the same time, they might get the same token:

Each create, update, or delete request must use a unique change token. If your application submits a GetChangeToken request and then submits a second GetChangeToken request before submitting a create, update, or delete request, the second GetChangeToken request returns the same value as the first GetChangeToken request.

But only one can actually do one change, the token will be immediately marked as invalid, the other operations will fail with The input token is no longer current.

This is happening with terraform when handling multiple AWS WAF resources.

Terraform Version

0.7.10

Affected Resource(s)

aws-provider.aws_waf_*

Terraform Configuration Files

provider "aws" {
    region = "eu-west-1"
}
resource "aws_waf_ipset" "hector_waf_ipset_test" {
  count = 4
  name = "${format("hector_waf_ipset_test_%d", count.index)}"

  ip_set_descriptors {
    type  = "IPV4"
    value = "${format("193.0.7.%d/32", count.index)}"
  }
}

Debug Output

If you run the above code, it will create only one resource each time. The same for destroy:

keymon@keytocho:/tmp/test-tf2$ terraform apply
aws_waf_ipset.hector_waf_ipset_test.0: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2805378577.type:  "" => "IPV4"
  ip_set_descriptors.2805378577.value: "" => "192.0.7.0/32"
  name:                                "" => "hector_waf_ipset_test_0"
aws_waf_ipset.hector_waf_ipset_test.2: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.3723951473.type:  "" => "IPV4"
  ip_set_descriptors.3723951473.value: "" => "192.0.7.2/32"
  name:                                "" => "hector_waf_ipset_test_2"
aws_waf_ipset.hector_waf_ipset_test.1: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2589362081.type:  "" => "IPV4"
  ip_set_descriptors.2589362081.value: "" => "192.0.7.1/32"
  name:                                "" => "hector_waf_ipset_test_1"
aws_waf_ipset.hector_waf_ipset_test.3: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.3767981249.type:  "" => "IPV4"
  ip_set_descriptors.3767981249.value: "" => "192.0.7.3/32"
  name:                                "" => "hector_waf_ipset_test_3"
aws_waf_ipset.hector_waf_ipset_test.3: Creation complete
Error applying plan:

3 error(s) occurred:

* aws_waf_ipset.hector_waf_ipset_test.0: WAFStaleDataException: The input token is no longer current.
	status code: 400, request id: 4f3cc0fd-b22e-11e6-b7bd-1999591af44d
* aws_waf_ipset.hector_waf_ipset_test.1: WAFStaleDataException: The input token is no longer current.
	status code: 400, request id: 4f401c62-b22e-11e6-9046-29c04bbde95a
* aws_waf_ipset.hector_waf_ipset_test.2: WAFStaleDataException: The input token is no longer current.
	status code: 400, request id: 4f3d362e-b22e-11e6-b7bd-1999591af44d

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
$?=1 10:11:07 rb:2.2.1 py:paas-cf aws:gov-paas-dev
keymon@keytocho:/tmp/test-tf2$ terraform apply
aws_waf_ipset.hector_waf_ipset_test.3: Refreshing state... (ID: 68c042a9-5e8a-46a7-9b79-76a24cf6d7a0)
aws_waf_ipset.hector_waf_ipset_test.1: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2589362081.type:  "" => "IPV4"
  ip_set_descriptors.2589362081.value: "" => "192.0.7.1/32"
  name:                                "" => "hector_waf_ipset_test_1"
aws_waf_ipset.hector_waf_ipset_test.0: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2805378577.type:  "" => "IPV4"
  ip_set_descriptors.2805378578.value: "" => "192.0.7.0/32"
  name:                                "" => "hector_waf_ipset_test_0"
aws_waf_ipset.hector_waf_ipset_test.2: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.3723951473.type:  "" => "IPV4"
  ip_set_descriptors.3723951473.value: "" => "192.0.7.2/32"
  name:                                "" => "hector_waf_ipset_test_2"
aws_waf_ipset.hector_waf_ipset_test.2: Creation complete
Error applying plan:

2 error(s) occurred:

* aws_waf_ipset.hector_waf_ipset_test.0: WAFStaleDataException: The input token is no longer current.
	status code: 400, request id: 599956e5-b22e-11e6-9fce-255be3b03960
* aws_waf_ipset.hector_waf_ipset_test.1: WAFStaleDataException: The input token is no longer current.
	status code: 400, request id: 599c160c-b22e-11e6-9046-29c04bbde95a

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
$?=1 10:11:25 rb:2.2.1 py:paas-cf aws:gov-paas-dev
keymon@keytocho:/tmp/test-tf2$ terraform apply
aws_waf_ipset.hector_waf_ipset_test.2: Refreshing state... (ID: e1d5fbfe-5133-40be-876a-dbc95ad11f8a)
aws_waf_ipset.hector_waf_ipset_test.3: Refreshing state... (ID: 68c042a9-5e8a-46a7-9b79-76a24cf6d7a0)
aws_waf_ipset.hector_waf_ipset_test.0: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2805378577.type:  "" => "IPV4"
  ip_set_descriptors.2805378577.value: "" => "192.0.7.0/32"
  name:                                "" => "hector_waf_ipset_test_0"
aws_waf_ipset.hector_waf_ipset_test.1: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2589362081.type:  "" => "IPV4"
  ip_set_descriptors.2589362081.value: "" => "192.0.7.1/32"
  name:                                "" => "hector_waf_ipset_test_1"
aws_waf_ipset.hector_waf_ipset_test.0: Creation complete
Error applying plan:

1 error(s) occurred:

* aws_waf_ipset.hector_waf_ipset_test.1: WAFStaleDataException: The input token is no longer current.
	status code: 400, request id: 5eddc253-b22e-11e6-9046-29c04bbde95a

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
$?=1 10:11:34 rb:2.2.1 py:paas-cf aws:gov-paas-dev
keymon@keytocho:/tmp/test-tf2$ terraform apply
aws_waf_ipset.hector_waf_ipset_test.3: Refreshing state... (ID: 68c042a9-5e8a-46a7-9b79-76a24cf6d7a0)
aws_waf_ipset.hector_waf_ipset_test.2: Refreshing state... (ID: e1d5fbfe-5133-40be-876a-dbc95ad11f8a)
aws_waf_ipset.hector_waf_ipset_test.0: Refreshing state... (ID: aadd06d6-3171-4bd0-8c00-0e530f4a878e)
aws_waf_ipset.hector_waf_ipset_test.1: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2589362081.type:  "" => "IPV4"
  ip_set_descriptors.2589362081.value: "" => "192.0.7.1/32"
  name:                                "" => "hector_waf_ipset_test_1"
aws_waf_ipset.hector_waf_ipset_test.1: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Expected Behavior

Should create/update/delete all the resources sequentially without failing.

Actual Behavior

Only one change at a time is possible. More might be possible if they happen to run sequentially.

Steps to Reproduce

Configure multiple WAF resources that would run in parallel
terraform apply

Workaround

Rerun terraform several times, it will create/update one object each time.
Create dependencies between resources
Use the option parallelism to limit the number of concurrent changes to 1: terraform apply -parallelism=1. But everything will be slower.

References

How WAF token works: http://docs.aws.amazon.com/waf/latest/APIReference/API_GetChangeToken.html

The text was updated successfully, but these errors were encountered:

stack72 · 2016-11-25T14:42:12Z

@optimisticanshul did you see this issue when you were testing?

Paul

ventz · 2016-12-23T06:38:04Z

Yep -- seeing the same thing.

vicfl · 2017-02-16T09:40:59Z

+1 still happening

steve-jansen · 2017-02-24T22:30:14Z

+1 also experiencing this issue with our environments. A 2nd apply seems to fix.

sebolabs · 2017-02-27T12:36:46Z

+1 same here

kaaloo · 2017-03-08T14:25:20Z

+1 on 0.8.8

ventz · 2017-03-08T15:23:30Z

@stack72 - Paul, it seems like this issue got lost -- would it be possible to assign someone on it. It's starting to gain a lot of traction, and the problem is very frustrating.

s-nakka · 2017-04-24T22:14:27Z

+1 on Terraform v0.9.3

radeksimko · 2017-04-25T06:08:40Z

The bugfix was merged and it will be released in 0.9.4. Thanks for the patience. 😅

ghost · 2020-04-13T02:21:20Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

stack72 added bug provider/aws labels Nov 25, 2016

radeksimko mentioned this issue Apr 14, 2017

provider/aws: Use mutex & retry for WAF change operations #13656

Merged

radeksimko closed this as completed in #13656 Apr 14, 2017

radeksimko mentioned this issue May 10, 2017

AWS WAF IPSet resources do not delete existing records correctly #10403

Closed

ghost locked and limited conversation to collaborators Apr 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws-provider: "The input token is no longer current." when operating on multiple WAF objects #10335

aws-provider: "The input token is no longer current." when operating on multiple WAF objects #10335

keymon commented Nov 24, 2016 •

edited

Loading

stack72 commented Nov 25, 2016

ventz commented Dec 23, 2016

vicfl commented Feb 16, 2017

steve-jansen commented Feb 24, 2017

sebolabs commented Feb 27, 2017

kaaloo commented Mar 8, 2017

ventz commented Mar 8, 2017

s-nakka commented Apr 24, 2017

radeksimko commented Apr 25, 2017

ghost commented Apr 13, 2020

aws-provider: "The input token is no longer current." when operating on multiple WAF objects #10335

aws-provider: "The input token is no longer current." when operating on multiple WAF objects #10335

Comments

keymon commented Nov 24, 2016 • edited Loading

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Workaround

References

stack72 commented Nov 25, 2016

ventz commented Dec 23, 2016

vicfl commented Feb 16, 2017

steve-jansen commented Feb 24, 2017

sebolabs commented Feb 27, 2017

kaaloo commented Mar 8, 2017

ventz commented Mar 8, 2017

s-nakka commented Apr 24, 2017

radeksimko commented Apr 25, 2017

ghost commented Apr 13, 2020

keymon commented Nov 24, 2016 •

edited

Loading