Route53 - SignatureDoesNotMatch: Signature expired

[gusmat]

When using terraform to manage a single route53 domain with 800+ records I get the following error when running "terraform plan":

* aws_route53_record.myrecord1-domain-com-CNAME: SignatureDoesNotMatch: Signature expired: 20161011T235745Z is now earlier than 20161011T235755Z (20161012T000255Z - 5 min.)
    status code: 403, request id: 3a583fd9-930f-11e6-bfe6-6b6e9007f0ed
* aws_route53_record.myrecord2-domain-com-CNAME: SignatureDoesNotMatch: Signature expired: 20161011T235743Z is now earlier than 20161011T235855Z (20161012T000355Z - 5 min.)
    status code: 403, request id: 5da285cf-910f-11e6-a018-0d45d0ee0e26

Terraform was able to create all 800+ records but it is not able to run "plan".
I already checked the instance date and it is properly set via NTP.
The error happens at random records and it varies from 1 to 5 occurrences every time I run terraform plan

I also increased the aws provider max_retries to 10 because I saw a some "Rate exceeded" errors but that did not fix the issue.

It takes about 7 minutes for the error to happen.

Terraform Version

• 0.7.4 and 0.7.5

Affected Resource(s)

• aws_route53_record

Debug Output

2016/10/12 00:26:23 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2016/10/12 00:26:23 [DEBUG] plugin: terraform: HTTP/1.1 403 Forbidden
2016/10/12 00:26:23 [DEBUG] plugin: terraform: Connection: close
2016/10/12 00:26:23 [DEBUG] plugin: terraform: Content-Length: 353
2016/10/12 00:26:23 [DEBUG] plugin: terraform: Content-Type: text/xml
2016/10/12 00:26:23 [DEBUG] plugin: terraform: Date: Wed, 12 Oct 2016 00:26:22 GMT
2016/10/12 00:26:23 [DEBUG] plugin: terraform: X-Amzn-Requestid: 816c05a6-9342-11e6-43a6-3156af876a6a
2016/10/12 00:26:23 [DEBUG] plugin: terraform: 
2016/10/12 00:26:23 [DEBUG] plugin: terraform: <?xml version="1.0"?>
2016/10/12 00:26:23 [DEBUG] plugin: terraform: <ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code><Message>Signature expired: 20161012T002110Z is now earlier than 20161012T002123Z (20161012T002623Z - 5 min.)</Message></Error><RequestId>816c05a6-9342-11e6-43a6-3156af876a6a</RequestId></ErrorResponse>
2016/10/12 00:26:23 [DEBUG] plugin: terraform: -----------------------------------------------------
2016/10/12 00:26:23 [ERROR] root: eval: *terraform.EvalRefresh, err: aws_route53_record.myrecord1-mydomain-com-CNAME: SignatureDoesNotMatch: Signature expired: 20161012T002110Z is now earlier than 20161012T002123Z (20161012T002623Z - 5 min.)
    status code: 403, request id: 816c05a6-9342-11e6-43a6-3156af876a6a
2016/10/12 00:26:23 [ERROR] root: eval: *terraform.EvalSequence, err: aws_route53_record.myrecord1-mydomain-com-CNAME: SignatureDoesNotMatch: Signature expired: 20161012T002110Z is now earlier than 20161012T002123Z (20161012T002623Z - 5 min.)
    status code: 403, request id: 816c05a6-9342-11e6-43a6-3156af876a6a
2016/10/12 00:26:23 [ERROR] root: eval: *terraform.EvalOpFilter, err: aws_route53_record.myrecord1-mydomain-com-CNAME: SignatureDoesNotMatch: Signature expired: 20161012T002110Z is now earlier than 20161012T002123Z (20161012T002623Z - 5 min.)
    status code: 403, request id: 816c05a6-9342-11e6-43a6-3156af876a6a
2016/10/12 00:26:23 [ERROR] root: eval: *terraform.EvalSequence, err: aws_route53_record.myrecord1-mydomain-com-CNAME: SignatureDoesNotMatch: Signature expired: 20161012T002110Z is now earlier than 20161012T002123Z (20161012T002623Z - 5 min.)
    status code: 403, request id: 816c05a6-9342-11e6-43a6-3156af876a6a
2016/10/12 00:26:23 [TRACE] [walkRefresh] Exiting eval tree: aws_route53_record.myrecord1-mydomain-com-CNAME
2016/10/12 00:26:23 [DEBUG] plugin: terraform: aws-provider (internal) 2016/10/12 00:26:23 [DEBUG] [aws-sdk-go] DEBUG: Response route53/GetHostedZone Details:
2016/10/12 00:26:23 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------

[udangel-r7]

I encounter a similar issue with a lot of resources etc.

[AMeng]

This was an issue with the AWS Golang SDK that is now fixed. See: aws/aws-sdk-go#876

Terraform 0.7.6 updated the SDK dependency to include that fix.

[gusmat]

Thanks @AMeng. Just tested with 0.7.7 and works fine.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.