Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resource/aws_kinesis_firehose_delivery_stream: Fix for s3 destinations #2970

Merged
merged 8 commits into from
Jan 17, 2018

Conversation

ApsOps
Copy link
Contributor

@ApsOps ApsOps commented Jan 12, 2018

  • import doesn't work for s3 destination

Fixes the breaking functionality caused by #2082

- import doesn't work for s3 destination
@bflad bflad added this to the v1.7.1 milestone Jan 12, 2018
@bflad bflad added bug Addresses a defect in current functionality. service/firehose Issues and PRs that pertain to the firehose service. labels Jan 12, 2018
@dennisatspaceape
Copy link
Contributor

Works for me on a newly created Delivery Stream, also verified that a subsequent plan does not flag any changes

@radeksimko radeksimko self-requested a review January 15, 2018 14:19
@radeksimko radeksimko changed the title Fix kinesis firehose stream breaking for s3 destinations resource/aws_kinesis_firehose_delivery_stream: Fix for s3 destinations Jan 16, 2018
@radeksimko radeksimko self-assigned this Jan 16, 2018
Flattener was previously broken, hence drifts were not detected.
Adding an error check should prevent breaking it in the future.
Firehose API automatically sets some default parameters
which we need to filter out to prevent diffs during routine
use of the resource.
@radeksimko
Copy link
Member

Hi @ApsOps
thanks for the PR. We're trying to get 1.7.1 (bugfix release) out this week so I took the liberty to make some tweaks and push them to your branch directly, instead of following the standard procedure (reviewing & discussing & letting you implement). I hope you don't mind.

I didn't pile up just random commits. The guiding line for me were failing acceptance tests which we already have in place. The goal was to get them all pass again which I managed to do with all attached changes.

I hope my commit messages are self-explanatory - if not I'm happy to explain/update.

There's still one pending bug in this resource, but it's apparently not covered in any test, so we can address it in a separate PR.

Test results

=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3InvalidParameterName
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3InvalidParameterName (1.68s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3InvalidProcessorType
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3InvalidProcessorType (1.77s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (111.15s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (120.53s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_importBasic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_importBasic (267.21s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (298.13s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (300.28s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (306.58s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (314.55s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (396.72s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (922.33s)
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (1158.73s)

@radeksimko radeksimko requested review from bflad and removed request for radeksimko January 17, 2018 12:08
@ApsOps
Copy link
Contributor Author

ApsOps commented Jan 17, 2018

@radeksimko Thanks so much!

It would be awesome if we could have acceptance tests running on travis. I think I wasn't able to get them working properly, and it costs more if tests fail which leaves some resources lying around sometimes.

Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! TC gives the go ahead. Two minor comments which you probably already addressed. 😄

"parameter_value": params.ParameterValue,
name, value := *params.ParameterName, *params.ParameterValue

if t == "Lambda" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: Constant available: firehose.ProcessorTypeLambda

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@@ -269,31 +298,36 @@ func flattenKinesisFirehoseDeliveryStream(d *schema.ResourceData, s *firehose.De
elasticsearchConfList[0] = elasticsearchConfiguration
d.Set("elasticsearch_configuration", elasticsearchConfList)
d.Set("s3_configuration", flattenFirehoseS3Configuration(*destination.ElasticsearchDestinationDescription.S3DestinationDescription))
} else if destination.S3DestinationDescription != nil {
} else if d.Get("destination").(string) == "s3" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: Should we create constants for these?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed on Slack we will eventually phase out this field as it likely doesn't need to be there at all.

@pecigonzalo
Copy link

Could it be related to #2997 ?

@ApsOps
Copy link
Contributor Author

ApsOps commented Jan 22, 2018

@pecigonzalo Yes, this should fix #2997

@bflad
Copy link
Contributor

bflad commented Jan 22, 2018

This has been released in terraform-provider-aws version 1.7.1. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

Please note we are tracking one additional crash relating to the (lack of) prefix of in a s3_configuration in #3071. Sorry for the trouble with this resource being quite complex and the acceptance testing quite lengthy.

drewsonne pushed a commit to drewsonne/terraform-provider-aws that referenced this pull request Mar 3, 2018
hashicorp#2970)

* Fix kinesis firehose stream breaking for s3 destinations
- import doesn't work for s3 destination

* Run tests for extended_s3 config

* Fix invalid test config

* Avoid crash on empty redshift's S3BackupDescription

Fixes hashicorp#3006

* Fix flattenProcessingConfiguration

Flattener was previously broken, hence drifts were not detected.
Adding an error check should prevent breaking it in the future.

* Filter out default params in processing configuration

Firehose API automatically sets some default parameters
which we need to filter out to prevent diffs during routine
use of the resource.

* Suppress diff for redshift_configuration.0.password

* Replace string with constant
@ghost
Copy link

ghost commented Apr 8, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/firehose Issues and PRs that pertain to the firehose service.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants