Issues with aws_kinesis_firehose_delivery_stream - externded_s3 state

[pecigonzalo]

Terraform Version

Terraform: v0.11.1
Provider:

• AWS: 1.7.0

Affected Resource(s)

Please list the resources as a list, for example:

• aws_kinesis_firehose_delivery_stream
  (when using extended_s3)

Terraform Configuration Files

resource "aws_kinesis_firehose_delivery_stream" "foo-bar-stream" {
  name        = "${var.FIREHOSE_STREAM_NAME}"
  destination = "extended_s3"

  extended_s3_configuration {
    role_arn   = "${aws_iam_role.foo-bar-role.arn}"
    bucket_arn = "arn:aws:s3:::examplebucket"

    processing_configuration = [
      {
        enabled = "false"
      },
    ]

    prefix          = "raw/foo/"
    buffer_size     = 96
    buffer_interval = 600
  }

  kinesis_source_configuration {
    kinesis_stream_arn = "${aws_kinesis_stream.foo-stream.arn}"
    role_arn           = "${aws_iam_role.foo-bar-role.arn}"
  }

  depends_on = [
    "aws_iam_role_policy.foo-bar-role-policy",
  ]
}

Expected Behavior

terraform state show aws_kinesis_firehose_delivery_stream.foo-bar-stream

id                                                                  = arn:aws:firehose:eu-central-1:123123:deliverystream/foo-bar-stage
arn                                                                 = arn:aws:firehose:eu-central-1:123123:deliverystream/foo-bar-stage
destination                                                         = extended_s3
destination_id                                                      = destinationId-000000000001
extended_s3_configuration.#                                         = 1
extended_s3_configuration.0.bucket_arn                              = arn:aws:s3:::examplebucket
extended_s3_configuration.0.buffer_interval                         = 600
extended_s3_configuration.0.buffer_size                             = 96
extended_s3_configuration.0.cloudwatch_logging_options.#            = 0
extended_s3_configuration.0.compression_format                      = UNCOMPRESSED
extended_s3_configuration.0.kms_key_arn                             = 
extended_s3_configuration.0.prefix                                  = enriched/foo/
extended_s3_configuration.0.processing_configuration.#              = 1
extended_s3_configuration.0.processing_configuration.0.enabled      = false
extended_s3_configuration.0.processing_configuration.0.processors.# = 0
extended_s3_configuration.0.role_arn                                = arn:aws:iam::123123:role/foo-bar-stage
kinesis_source_configuration.#                                      = 1
kinesis_source_configuration.0.kinesis_stream_arn                   = arn:aws:kinesis:eu-central-1:123123:stream/foo-stage
kinesis_source_configuration.0.role_arn                             = arn:aws:iam::123123:role/foo-bar-stage
name                                                                = foo-bar-stage
version_id                                                          = 1

Actual Behavior

terraform state show aws_kinesis_firehose_delivery_stream.foo-bar-stream

id                                                                       = arn:aws:firehose:eu-central-1:123123:deliverystream/foo-bar-stage
arn                                                                      = arn:aws:firehose:eu-central-1:123123:deliverystream/foo-bar-stage
destination                                                              = s3
destination_id                                                           = destinationId-000000000001
extended_s3_configuration.#                                              = 1
extended_s3_configuration.0.bucket_arn                                   = arn:aws:s3:::examplebucket
extended_s3_configuration.0.buffer_interval                              = 600
extended_s3_configuration.0.buffer_size                                  = 96
extended_s3_configuration.0.cloudwatch_logging_options.#                 = 0
extended_s3_configuration.0.compression_format                           = UNCOMPRESSED
extended_s3_configuration.0.kms_key_arn                                  = 
extended_s3_configuration.0.prefix                                       = raw/foo/
extended_s3_configuration.0.processing_configuration.#                   = 1
extended_s3_configuration.0.processing_configuration.0.enabled           = false
extended_s3_configuration.0.processing_configuration.0.processors.#      = 0
extended_s3_configuration.0.role_arn                                     = arn:aws:iam::123123:role/foo-bar-stage
kinesis_source_configuration.#                                           = 1
kinesis_source_configuration.0.kinesis_stream_arn                        = arn:aws:kinesis:eu-central-1:123123:stream/foo-stage
kinesis_source_configuration.0.role_arn                                  = arn:aws:iam::123123:role/foo-bar-stage
name                                                                     = foo-bar-stage
s3_configuration.#                                                       = 1
s3_configuration.0.bucket_arn                                            = arn:aws:s3:::examplebucket
s3_configuration.0.buffer_interval                                       = 600
s3_configuration.0.buffer_size                                           = 96
s3_configuration.0.cloudwatch_logging_options.#                          = 1
s3_configuration.0.cloudwatch_logging_options.2513562885.enabled         = false
s3_configuration.0.cloudwatch_logging_options.2513562885.log_group_name  = 
s3_configuration.0.cloudwatch_logging_options.2513562885.log_stream_name = 
s3_configuration.0.compression_format                                    = UNCOMPRESSED
s3_configuration.0.kms_key_arn                                           = 
s3_configuration.0.prefix                                                = raw/foo/
s3_configuration.0.role_arn                                              = arn:aws:iam::123123:role/foo-bar-stage
version_id                                                               = 1

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Create a minimal firehose configuration with

provider "aws" {
  version = "~> 1.7.0"

  region = "${var.region}"
}

2. Apply
3. Try to apply again or inspect the resource state

You will notice the resource tries to recreate on every apply and fails to destroy with error

aws_kinesis_firehose_delivery_stream.foo-bar-stream: Destroying... (ID: arn:aws:firehose:eu-central-1:123123:deliverystream/foo-bar-stage)
aws_kinesis_firehose_delivery_stream.foo-bar-stream: Still destroying... (ID: arn:aws:firehose:eu-central-1:123123:deliverystream/foo-bar-stage, 10s elapsed)
aws_kinesis_firehose_delivery_stream.foo-bar-stream: Destruction complete after 11s

Error: Error applying plan:

1 error(s) occurred:

* aws_kinesis_firehose_delivery_stream.foo-bar-stream: aws_kinesis_firehose_delivery_stream.foo-bar-stream: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.11.1
    Resource ID: aws_kinesis_firehose_delivery_stream.foo-bar-stream
    Mismatch reason: attribute mismatch: s3_configuration.0.buffer_interval
[...]

Important Factoids

I believe this was added on #2082

References

• Add import support for kinesis firehose delivery stream #2082

[gbergere]

Hi,

Same problem here.

For me, slightly different, it succeed to destroy the firehose but same issue it keep recreating it each it.
I noticed this happen from the upgrade to aws: 1.7. But with version 1.6 it works.

provider "aws" {                                                                 
  version = "< 1.7.0"
  ...
}

Thanks for you fix.

[jetpks]

Hi!

I'm also experiencing this issue. The destruction appears to complete successfully, but the "new" firehose can't be created.

aws_kinesis_firehose_delivery_stream.chunker_firehose: Still destroying... (ID: arn:aws:firehose:us-west-2:123456...chunker-firehose1, 10s elapsed)
aws_kinesis_firehose_delivery_stream.cg_log_chunker_firehose: Still destroying... (ID: arn:aws:firehose:us-west-2:123456...chunker-firehose1, 20s elapsed)
aws_kinesis_firehose_delivery_stream.chunker_firehose: Destruction complete after 30s

Error: Error applying plan:

1 error(s) occurred:

* aws_kinesis_firehose_delivery_stream.chunker_firehose: aws_kinesis_firehose_delivery_stream.chunker_firehose: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.10.8
    Resource ID: aws_kinesis_firehose_delivery_stream.chunker_firehose
    Mismatch reason: attribute mismatch: s3_configuration.0.buffer_interval

I can also confirm that the workaround @gbergere mentioned works like a charm. 👍🏻👍🏻

provider "aws" {
  version = "< 1.7.0"
...
}

Thanks!

[ApsOps]

This should be fixed by #2970

[radeksimko]

Thanks for the follow-up, closing.

[bflad]

This has been released in terraform-provider-aws version 1.7.1. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

Please note we are tracking one additional crash relating to the (lack of) prefix of in a s3_configuration in #3071. Sorry for the trouble with this resource being quite complex and the acceptance testing quite lengthy.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!