Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_sns_topic_subscription apply intermittently fails with "Root resource was present, but now absent" error #16695

Closed
camlow325 opened this issue Dec 10, 2020 · 3 comments · Fixed by #18475
Assignees
Labels
bug Addresses a defect in current functionality. service/sns Issues and PRs that pertain to the sns service.
Milestone

Comments

@camlow325
Copy link
Contributor

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

› terraform -v
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.4.0

Affected Resource(s)

  • aws_sns_topic_subscription

Terraform Configuration Files

resource "aws_sqs_queue" "this" {
  name = "test"
}

resource "aws_sns_topic" "this" {
  name = "test"
}

data "aws_iam_policy_document" "sns" {
  statement {
    actions = [
      "SNS:GetTopicAttributes",
      "SNS:Subscribe",
      "SNS:ListSubscriptionsByTopic",
      "SNS:Publish"
    ]
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    resources = [aws_sns_topic.this.arn]
    effect    = "Allow"
  }
}

resource "aws_sns_topic_policy" "this" {
  arn    = aws_sns_topic.this.arn
  policy = data.aws_iam_policy_document.sns.json
}

resource "aws_sns_topic_subscription" "sqs" {
  topic_arn = aws_sns_topic_policy.this.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.this.arn
}

Debug Output

The debug output shows that the provider was able to create the SNS subscription successfully:

2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] [aws-sdk-go] DEBUG: Request sns/Subscribe Details:
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: ---[ REQUEST POST-SIGN ]-----------------------------
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: POST / HTTP/1.1
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Host: sns.us-west-2.amazonaws.com
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: User-Agent: aws-sdk-go/1.34.10 (go1.14.5; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.13.5 (+https://www.terraform.io)
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Content-Length: 222
...
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Content-Type: application/x-www-form-urlencoded; charset=uff-8
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: X-Amz-Date: 20201210T151440Z
...
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Accept-Encoding: gzip
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Action=Subscribe&Endpoint=arn%3Aaws%3Asqs%3Aus-west-2%3AXXX%3Atest&Protocol=sqs&TopicArn=arn%3Aaws%3Asns%3Aus-west-2%3AXXX%3Atest&Version=2010-03-31
2020-12-10T15:14:40.147Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: -----------------------------------------------------
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] [aws-sdk-go] DEBUG: Response sns/Subscribe Details:
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: ---[ RESPONSE ]--------------------------------------
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: HTTP/1.1 200 OK
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Connection: close
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Content-Length: 373
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Content-Type: text/xml
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Date: Thu, 10 Dec 2020 15:14:40 GMT
...
-----------------------------------------------------
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] [aws-sdk-go] <SubscribeResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   <SubscribeResult>
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:     <SubscriptionArn>arn:aws:sns:us-west-2:XXX:test</SubscriptionArn>
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   </SubscribeResult>
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   <ResponseMetadata>
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:     <RequestId>XXX</RequestId>
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   </ResponseMetadata>
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: </SubscribeResponse>
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] Finished subscribing to topic arn:aws:sns:us-west-2:XXX:test with subscription arn arn:aws:sns:us-west-2:XXX:test:XXX
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] Created new subscription! arn:aws:sns:us-west-2:XXX:test:XXX
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 New subscription ARN: arn:aws:sns:us-west-2:XXX:test:XXX

Immediately after creating the subscription, though, a call to GetSubscriptionAttributes for the subscription returns a 404 error, which leads to the Terraform apply failure:

2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] Loading subscription arn:aws:sns:us-west-2:XXX:test:XXX
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] [aws-sdk-go] DEBUG: Request sns/GetSubscriptionAttributes Details:
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: ---[ REQUEST POST-SIGN ]-----------------------------
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: POST / HTTP/1.1
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Host: sns.us-west-2.amazonaws.com
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: User-Agent: aws-sdk-go/1.34.10 (go1.14.5; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.13.5 (+https://www.terraform.io)
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Content-Length: 175
…
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: X-Amz-Date: 20201210T151440Z
…
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Accept-Encoding: gzip
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Action=GetSubscriptionAttributes&SubscriptionArn=arn%3Aaws%3Asns%3Aus-west-2%3AXXX%3Atest%3AXXX&Version=2010-03-31
2020-12-10T15:14:40.330Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: -----------------------------------------------------
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] [aws-sdk-go] DEBUG: Response sns/GetSubscriptionAttributes Details:
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: ---[ RESPONSE ]--------------------------------------
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: HTTP/1.1 404 Not Found
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Connection: close
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Content-Length: 266
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Content-Type: text/xml
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: Date: Thu, 10 Dec 2020 15:14:39 GMT
…
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: -----------------------------------------------------
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   <Error>
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:     <Type>Sender</Type>
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:     <Code>NotFound</Code>
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:     <Message>Subscription does not exist</Message>
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   </Error>
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   <RequestId>XXX</RequestId>
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: </ErrorResponse>
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [DEBUG] [aws-sdk-go] DEBUG: Validate Response sns/GetSubscriptionAttributes failed, attempt 0/25, error NotFound: Subscription does not exist
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:       status code: 404, request id: XXX
2020-12-10T15:14:40.404Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2020/12/10 15:14:40 [WARN] SNS Topic Subscription (arn:aws:sns:us-west-2:XXX:test:XXX) not found, removing from state
2020/12/10 15:14:40 [DEBUG] aws_sns_topic_subscription.sqs: apply errored, but we're indicating that via the Error pointer rather than returning it: Provider produced inconsistent result after apply: When applying changes to aws_sns_topic_subscription.sqs, provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value: Root resource was present, but now absent.

When I looked in AWS at the SNS topic after the failure, I saw that the subscription did actually exist, so this may just be an eventual consistency-related issue.

Panic Output

Expected Behavior

The apply should have succeeded.

Actual Behavior

Most of the time, the apply succeeds. In at least once instance, though, the aws_sns_topic_subscription resource failed to apply, with this error:

Error: Provider produced inconsistent result after apply

When applying changes to aws_sns_topic_subscription.sqs,
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Note that if a second terraform apply is attempted after the error occurs, the aws_sns_topic_subscription resource is created and the apply succeeds.

Steps to Reproduce

terraform apply

Important Factoids

References

@ghost ghost added service/iam Issues and PRs that pertain to the iam service. service/sns Issues and PRs that pertain to the sns service. service/sqs Issues and PRs that pertain to the sqs service. labels Dec 10, 2020
@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Dec 10, 2020
@ewbankkit ewbankkit added bug Addresses a defect in current functionality. and removed needs-triage Waiting for first response or review from a maintainer. service/iam Issues and PRs that pertain to the iam service. service/sqs Issues and PRs that pertain to the sqs service. labels Jan 27, 2021
bflad added a commit that referenced this issue Mar 30, 2021
…l consistency, enforce lowercase protocol argument validation

Reference: #10225
Reference: #11737
Reference: #12692
Reference: #16695
Reference: #16796

The `protocol` validation update is to catch where the API accepts uppercase values such as `HTTPS`, but prevents proper handling when the API canonicalizes it to lowercase. The API documentation and existing Terraform documentation solely use lowercase.

Output from acceptance testing in AWS Commercial:

```
--- PASS: TestAccAWSSNSTopicSubscription_autoConfirmingEndpoint (95.60s)
--- PASS: TestAccAWSSNSTopicSubscription_autoConfirmingSecuredEndpoint (137.43s)
--- PASS: TestAccAWSSNSTopicSubscription_basic (66.20s)
--- PASS: TestAccAWSSNSTopicSubscription_deliveryPolicy (69.55s)
--- PASS: TestAccAWSSNSTopicSubscription_disappears (74.02s)
--- PASS: TestAccAWSSNSTopicSubscription_disappears_topic (75.13s)
--- PASS: TestAccAWSSNSTopicSubscription_email (16.78s)
--- PASS: TestAccAWSSNSTopicSubscription_filterPolicy (71.62s)
--- PASS: TestAccAWSSNSTopicSubscription_firehose (140.29s)
--- PASS: TestAccAWSSNSTopicSubscription_rawMessageDelivery (69.77s)
--- PASS: TestAccAWSSNSTopicSubscription_redrivePolicy (64.88s)
```

Output from acceptance testing in AWS GovCloud (US):

```
--- PASS: TestAccAWSSNSTopicSubscription_basic (70.13s)
--- PASS: TestAccAWSSNSTopicSubscription_deliveryPolicy (106.45s)
--- PASS: TestAccAWSSNSTopicSubscription_disappears (82.09s)
--- PASS: TestAccAWSSNSTopicSubscription_disappears_topic (68.14s)
--- PASS: TestAccAWSSNSTopicSubscription_email (20.04s)
--- PASS: TestAccAWSSNSTopicSubscription_filterPolicy (106.32s)
--- PASS: TestAccAWSSNSTopicSubscription_rawMessageDelivery (95.36s)
--- PASS: TestAccAWSSNSTopicSubscription_redrivePolicy (110.75s)
--- SKIP: TestAccAWSSNSTopicSubscription_autoConfirmingEndpoint (1.41s)
--- SKIP: TestAccAWSSNSTopicSubscription_autoConfirmingSecuredEndpoint (1.41s)
--- SKIP: TestAccAWSSNSTopicSubscription_firehose (53.36s)
```
@bflad bflad self-assigned this Mar 30, 2021
bflad added a commit that referenced this issue Mar 30, 2021
…l consistency, enforce lowercase protocol argument validation (#18475)

* resource/aws_sns_topic_subscription: Handle read-after-create eventual consistency, enforce lowercase protocol argument validation

Reference: #10225
Reference: #11737
Reference: #12692
Reference: #16695
Reference: #16796

The `protocol` validation update is to catch where the API accepts uppercase values such as `HTTPS`, but prevents proper handling when the API canonicalizes it to lowercase. The API documentation and existing Terraform documentation solely use lowercase.

Output from acceptance testing in AWS Commercial:

```
--- PASS: TestAccAWSSNSTopicSubscription_autoConfirmingEndpoint (95.60s)
--- PASS: TestAccAWSSNSTopicSubscription_autoConfirmingSecuredEndpoint (137.43s)
--- PASS: TestAccAWSSNSTopicSubscription_basic (66.20s)
--- PASS: TestAccAWSSNSTopicSubscription_deliveryPolicy (69.55s)
--- PASS: TestAccAWSSNSTopicSubscription_disappears (74.02s)
--- PASS: TestAccAWSSNSTopicSubscription_disappears_topic (75.13s)
--- PASS: TestAccAWSSNSTopicSubscription_email (16.78s)
--- PASS: TestAccAWSSNSTopicSubscription_filterPolicy (71.62s)
--- PASS: TestAccAWSSNSTopicSubscription_firehose (140.29s)
--- PASS: TestAccAWSSNSTopicSubscription_rawMessageDelivery (69.77s)
--- PASS: TestAccAWSSNSTopicSubscription_redrivePolicy (64.88s)
```

Output from acceptance testing in AWS GovCloud (US):

```
--- PASS: TestAccAWSSNSTopicSubscription_basic (70.13s)
--- PASS: TestAccAWSSNSTopicSubscription_deliveryPolicy (106.45s)
--- PASS: TestAccAWSSNSTopicSubscription_disappears (82.09s)
--- PASS: TestAccAWSSNSTopicSubscription_disappears_topic (68.14s)
--- PASS: TestAccAWSSNSTopicSubscription_email (20.04s)
--- PASS: TestAccAWSSNSTopicSubscription_filterPolicy (106.32s)
--- PASS: TestAccAWSSNSTopicSubscription_rawMessageDelivery (95.36s)
--- PASS: TestAccAWSSNSTopicSubscription_redrivePolicy (110.75s)
--- SKIP: TestAccAWSSNSTopicSubscription_autoConfirmingEndpoint (1.41s)
--- SKIP: TestAccAWSSNSTopicSubscription_autoConfirmingSecuredEndpoint (1.41s)
--- SKIP: TestAccAWSSNSTopicSubscription_firehose (53.36s)
```

* Update CHANGELOG for #18475
@github-actions github-actions bot added this to the v3.35.0 milestone Mar 30, 2021
@ghost
Copy link

ghost commented Apr 1, 2021

This has been released in version 3.35.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Apr 30, 2021

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked as resolved and limited conversation to collaborators Apr 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/sns Issues and PRs that pertain to the sns service.
Projects
None yet
3 participants