aws_cloudwatch_log_stream resource produced new value for was present but now absent

[camlow325]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform -v
Terraform v0.12.3
+ provider.aws v2.37.0

Affected Resource(s)

• aws_cloudwatch_log_stream

Terraform Configuration Files

resource "aws_cloudwatch_log_group" "test" {
  name = "/test"
}

resource "aws_cloudwatch_log_stream" "test" {
  name           = "test"
  log_group_name = aws_cloudwatch_log_group.test.name
}

Debug Output

Here's a snippet of the debug log when the error occurs. The problem appears to occur when AWS returns a 200 OK with an empty list of logStreams for the stream which has just been created. This may be due to an eventual consistency issue that could be overcome by retrying the DescribeLogStreams call some number of times until the resource is available.

2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: 2019/12/30 22:27:14 [DEBUG] [aws-sdk-go] DEBUG: Request logs/DescribeLogStreams Details:
2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: POST / HTTP/1.1
2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: Host: logs.us-west-2.amazonaws.com
2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: User-Agent: aws-sdk-go/1.25.36 (go1.13.3; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.3 (+https://www.terraform.io)
...
2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: X-Amz-Target: Logs_20140328.DescribeLogStreams
2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: Accept-Encoding: gzip
2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: 
2019-12-30T22:27:14.323Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: {"logGroupName":"/test","logStreamNamePrefix":"test"}
...
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: ---[ RESPONSE ]--------------------------------------
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: HTTP/1.1 200 OK
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: Connection: close
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: Content-Length: 17
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: Content-Type: application/x-amz-json-1.1
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: Date: Mon, 30 Dec 2019 22:27:14 GMT
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: X-Amzn-Requestid: XXX
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: 
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: 
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: -----------------------------------------------------
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: 2019/12/30 22:27:14 [DEBUG] [aws-sdk-go] {"logStreams":[]}
2019-12-30T22:27:14.644Z [DEBUG] plugin.terraform-provider-aws_v2.37.0_x4: 2019/12/30 22:27:14 [DEBUG] CloudWatch Stream "test" Not Found. Removing from state
2019/12/30 22:27:14 [DEBUG] aws_cloudwatch_log_stream.test: apply errored, but we're indicating that via the Error pointer rather than returning it: Provider produced inconsistent result after apply: When applying changes to aws_cloudwatch_log_stream.test, provider "aws" produced an unexpected new value for was present, but now absent.

Panic Output

Expected Behavior

The log stream resource should be created without any errors and the terraform apply should succeed.

Actual Behavior

Most of the time, the expected behavior occurs. Intermittently, though, the log stream resource is created in AWS but the terraform apply fails with the following error:

Error: Provider produced inconsistent result after apply

When applying changes to
aws_cloudwatch_log_stream.test,
provider "aws" produced an unexpected new value for was present, but now
absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Subsequent terraform apply attempts fail with the following error:

aws_cloudwatch_log_stream.test: Creating...
Error: Creating CloudWatch Log Stream failed: ResourceAlreadyExistsException: The specified log stream already exists
	status code: 400, request id: XXX

Steps to Reproduce

1. terraform apply

Important Factoids

References

• aws_appautoscaling_policy resource produced new value for was present but now absent #10549 - The behavior for this case seems similar, only for an aws_appautoscaling_policy resource instead of an aws_cloudwatch_log_stream one.

[bflad]

The fix for this has been merged and will release with version 2.48.0 of the Terraform AWS Provider, tomorrow. Thanks to @camlow325 for the implementation. 👍

[ghost]

This has been released in version 2.48.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!