Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent EMR cluster recreation when javax.jdo.option.ConnectionPassword is used in configuration_json #13

Merged
merged 3 commits into from
Apr 20, 2020
Merged

Prevent EMR cluster recreation when javax.jdo.option.ConnectionPassword is used in configuration_json #13

merged 3 commits into from
Apr 20, 2020

Conversation

3h4x
Copy link

@3h4x 3h4x commented Apr 15, 2020

what

  • Bootstrap action added that executes /bin/echo and as an argument md5 hash of configuration_json - dummy action
  • configuration_json added to lifecycle/ignore_changes

why

  • When configuration_json uses javax.jdo.option.ConnectionPassword then terraform will force recreate cluster with every apply. Reason for that is that AWS API for describe_cluster returns configuration with string '********instead of real password forjavax.jdo.option.ConnectionPassword. terraform` want to replace that "masked" password with actual password.

references

@3h4x 3h4x requested a review from goruha April 15, 2020 16:45
@3h4x 3h4x requested a review from a team as a code owner April 15, 2020 16:45
@3h4x 3h4x changed the title Backward Incompatible - Prevent EMR cluster recreatinwhen javax.jdo.option.ConnectionPassword is used in configuration_json Backward Incompatible - Prevent EMR cluster recreation when javax.jdo.option.ConnectionPassword is used in configuration_json Apr 15, 2020
@3h4x 3h4x requested a review from maximmi April 17, 2020 18:58
maximmi
maximmi previously approved these changes Apr 17, 2020
View reviewed changes
@3h4x 3h4x force-pushed the emr-dummy-bootstrap-action branch from 19d3a16 to 31e8be3 Compare April 20, 2020 14:43
@3h4x 3h4x changed the title Backward Incompatible - Prevent EMR cluster recreation when javax.jdo.option.ConnectionPassword is used in configuration_json Prevent EMR cluster recreation when javax.jdo.option.ConnectionPassword is used in configuration_json Apr 20, 2020
@3h4x 3h4x requested review from aknysh, osterman and a team April 20, 2020 15:08
@3h4x 3h4x merged commit 8564f4e into master Apr 20, 2020
@3h4x 3h4x deleted the emr-dummy-bootstrap-action branch April 20, 2020 15:23
LawrenceWarren added a commit to humn-ai/tf-mod-aws-emr-cluster that referenced this pull request Aug 18, 2022
@LawrenceWarren @aknysh
@navdeep710 @maximmi @osterman @yujunz @actions-bot @Gowiem @calvin-barker @dusty73 @hannahkamundson @tirumerla @cloudpossebot @renovate @renovate-bot @kunickiaj @mergify @shabih-paystack @mattbsea @nitrocode @pecastro @szymonm @erpel @dylanbannon
HT-8112: Merge upstream for EMR cluster (#2)
fb63dfd 
* Initial commit

* Initial implementation (#1)

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Initial implementation

* Add `subnet_type` var

* Add `aws_vpc_endpoint` for private subnets

* Add `aws_vpc_endpoint` for private subnets

* Add `aws_vpc_endpoint` for private subnets

* Add `aws_vpc_endpoint` for private subnets

* Find `aws_route_table` by `subnet_id`

* Find `aws_route_table` by `subnet_id`

* Add `aws_route_table`

* Fix route table

* Fix security groups (#2)

* Use EMR default security groups for all managed security groups

* Use EMR default security groups for all managed security groups

* Use EMR default security groups for all managed security groups

* Add `configurations_json`. Fix `bootstrap_action` (#5)

* Added application-autoscaling.amazonaws.com as Trusted entity for e… (cloudposse#11)

* Added application-autoscaling.amazonaws.com as Trusted entities for emr auto scaling role

Earlier autoscaling policy is failing to attach with the error
Failed to provision the AutoScaling policy: Unable to assume IAM role: arn:aws:iam::216727*****:role/emr-stage-dataorc-emr-ec2-autoscaling
this is due to absence of application-autoscaling.amazonaws.com as Trusted entity

* Ran terraform fmt

* corrected role for autosclaing policy, which requires 'elasticmapreduce.amazonaws.com', 'application-autoscaling.amazonaws.com' as trust entities

* New README.md generated from Package terraform-docs already installed
Generated README.md from /Users/navdeepagarwal/projects/open-projects/terraform-aws-emr-cluster/build-harness/templates/README.md using data from /Users/navdeepagarwal/projects/open-projects/terraform-aws-emr-cluster/build-harness/templates/README.yaml

* Add `slash-command-dispatch` GitHub Actions workflow (cloudposse#12)

* Add `slash-command-dispatch` GitHub Actions workflow

* random suffix added to name to avoid s3 bucket problems with existing buckets

* seed for rand applied

* test fixed

* test fixed

* Update examples/complete/fixtures.us-east-2.tfvars

Co-Authored-By: Erik Osterman <erik@cloudposse.com>

Co-authored-by: Erik Osterman <erik@cloudposse.com>

* Fix `log_uri` in example (cloudposse#10)

* Fix `log_uri` in example

Closes: cloudposse#8

* Updated README.md

Co-authored-by: Maxim Mironenko <simixido@gmail.com>
Co-authored-by: actions-bot <58130806+actions-bot@users.noreply.github.com>

* Prevent EMR cluster recreation when javax.jdo.option.ConnectionPassword is used in configuration_json (cloudposse#13)

* Add dummy bootstrap action that use md5 of configuration_json as an argument to prevent force recreation

* Terratest fixed, empty string as new default for configurations_json

* More descriptive comment for changes around bootstrap_action

* Add `var.create_vpc_endpoint_s3` to control S3 endpoint creation (cloudposse#14)

* Add `var.create_vpc_endpoint_s3` to control S3 endpoint creation

* variable description

* Updated README.md

Co-authored-by: actions-bot <58130806+actions-bot@users.noreply.github.com>

* Added kerberos_attributes configuration (cloudposse#16)

* Added kerberos_attributes configuration

* Readme updated

* Updates to ChatOps - Automated commit (cloudposse#17)

## What
* Adds chatops commands
  - '/test all'
  - '/test bats'
  - '/test readme'
  - '/test terratest'
* Drops codefresh
* Drops slash-command-dispatch
* Removes codefresh badge
* Rebuilds README

## Why
* Change over from codefresh to GH Actions
* Facilitate testing of PRs from forks

* [AUTOMATED] Update terraform-null-label versions to support Terraform 0.13 (cloudposse#19)

* [AUTOMATED] Update terraform-null-label versions to support Terraform 0.13

* Updated README.md

Co-authored-by: actions-bot <58130806+actions-bot@users.noreply.github.com>

* [AUTOMATED] Update Version Pinning for Terraform to support 0.13 (cloudposse#18)

## What

1. Update Version Pinning for Terraform to support 0.13

## Why

1. This is a relatively minor update that the CloudPosse module already likely supports.
1. This allows module consumers to not individually update our Terraform module to support Terraform 0.13.

* Support terraform 0.13 (cloudposse#20)

* fix: unquote references in ignore_changes (cloudposse#21)

* fix: unquote references in ignore_changes

* Updated README.md

Co-authored-by: actions-bot <58130806+actions-bot@users.noreply.github.com>

* Add support for step_concurrency_level (cloudposse#25)

* Adding step concurrency level support

* Dropping rogue "

* dropping potentially confusing 'default is 1'

* Updated README.md

Co-authored-by: actions-bot <58130806+actions-bot@users.noreply.github.com>

* Expose ec2 role (cloudposse#27)

* fix: add ingress rule from managed master to service (cloudposse#22)

* fix: add ingress rule from managed master to service

* Make ingress rule consistent with other rules

* Join security groups ids, instead of using count

Co-authored-by: Andriy Knysh <aknysh@users.noreply.github.com>

* Join security groups ids, instead of using count

Co-authored-by: Andriy Knysh <aknysh@users.noreply.github.com>

* Updated README.md

Co-authored-by: Andriy Knysh <aknysh@users.noreply.github.com>
Co-authored-by: actions-bot <58130806+actions-bot@users.noreply.github.com>

* Update to `context.tf`. Use the latest EMR version in the example. Update GitHub Actions (cloudposse#28)

* Update to `context.tf`. Use the latest EMR version in the example. Update GitHub Actions

* Update to `context.tf`. Use the latest EMR version in the example. Update GitHub Actions

* Include Steps option in aws_emr_cluster (cloudposse#30)

* Updating for issue number 7 so that we can include steps in the emr
cluster

* Executed 'terraform fmt'

* Updated README.md

Co-authored-by: Hannah Amundson <amundson.hannah@heb.com>
Co-authored-by: actions-bot <58130806+actions-bot@users.noreply.github.com>

* update emr module to use existing security groups (cloudposse#33)

* update emr module to use existing security groups

* fix formatting & variable decription based on the review

* Terraform 0.14 upgrade (cloudposse#34)

* Update README.md and docs (cloudposse#37)

Co-authored-by: osterman <osterman@users.noreply.github.com>

* context.tf updated to v0.24.1, minimum required Terraform version bumped to 0.13.0 when needed, readme updated (cloudposse#36)

* chore(deps): update terraform cloudposse/route53-cluster-hostname/aws to v0.12.0 (cloudposse#39)

Co-authored-by: Renovate Bot <bot@renovateapp.com>

* chore(deps): update terraform cloudposse/label/null to v0.24.1 (cloudposse#38)

Co-authored-by: Renovate Bot <bot@renovateapp.com>

* Add resource tags to created IAM roles (cloudposse#32)

* Add resource tags to created IAM roles

* Auto Format

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* Update module to be terraform v0.15 compatible (cloudposse#40)

Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* Update context.tf from origin source (cloudposse#43)

Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* Feature: Allow users to pass their own IAM roles (cloudposse#44)

* Feature: Allow users to pass their own IAM roles

* Rename variables following best practices

* update wording for variables & remove unused variable

* Add permissions boundaries and GovCloud support (cloudposse#42)

* feat: Added permissions boundary variables for iam roles

* feat: Add variable to adjust policy ARNs for use in AWS GovCloud.

* fix: type of govcloud set to bool instead of boolean

* Auto Format

* Auto Format

* partition

* Auto Format

* govcloud partition

* Update variables.tf

* Auto Format

Co-authored-by: nitrocode <nitrocode@users.noreply.github.com>
Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* chore(deps): update terraform cloudposse/label/null to v0.25.0 (cloudposse#48)

* chore(deps): update terraform cloudposse/label/null to v0.25.0

* Auto Format

Co-authored-by: Renovate Bot <bot@renovateapp.com>
Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* Upgrade aws provider to 3.0 (cloudposse#45)

* Upgrade aws provider to 3.0

* Auto Format

* Update versions.tf

* Update main.tf

* Update versions.tf

* Auto Format

Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* chore(deps): update terraform cloudposse/route53-cluster-hostname/aws to v0.12.2 (cloudposse#47)

* chore(deps): update terraform cloudposse/route53-cluster-hostname/aws to v0.12.2

* Auto Format

Co-authored-by: Renovate Bot <bot@renovateapp.com>
Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* Add missing tags to the aws_iam_instance_profile resource. (cloudposse#54)

* Add missing tags to the aws_iam_instance_profile resource.

* Auto Format

Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* Add support for auto termination policy, resolves cloudposse#53 (cloudposse#55)

* Add support for auto termination policy

* Auto Format

* Fix null check

Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* Clarify descriptions to indicate secgroup id is required (cloudposse#50)

* Clarify descriptions to indicate secgroup id is required

* Auto Format

Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* git.io->cloudposse.tools update

* git-xargs programmatic commit

* Auto Format

Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>

* Update all modules, examples and tests to the latest versions and patterns (cloudposse#57)

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* Update all modules, examples and tests to the latest versions and patterns

* fix(TF): Delete dead files, update README.md

* fix(TF): Update docs

* fix(TF): Update config

* fix(TF): Delete Makefile

* fix(TF): Add Makefile

* fix(TF): Cleanup docs

* fix(TF): Cleanup docs

* fix(TF): Cleanup docs

* fix(TF): Add support for `certificate_arn`

Co-authored-by: Andriy Knysh <aknysh@users.noreply.github.com>
Co-authored-by: Navdeep <navdeep710@gmail.com>
Co-authored-by: Maxim Mironenko <maxim@cloudposse.com>
Co-authored-by: Erik Osterman <erik@cloudposse.com>
Co-authored-by: Yujun Zhang <zhangyujun@gmail.com>
Co-authored-by: Maxim Mironenko <simixido@gmail.com>
Co-authored-by: actions-bot <58130806+actions-bot@users.noreply.github.com>
Co-authored-by: marcin <marcin@cloudposse.com>
Co-authored-by: Scott Cabrinha <scott@internaught.io>
Co-authored-by: Matt Gowie <gowie.matt@gmail.com>
Co-authored-by: Stepan Rakitin <svrakitin@yandex.ru>
Co-authored-by: calvin-barker <54291801+calvin-barker@users.noreply.github.com>
Co-authored-by: Igor D'Astolfo <i.dastolfo@gmail.com>
Co-authored-by: Hannah <48397717+hannahkamundson@users.noreply.github.com>
Co-authored-by: Hannah Amundson <amundson.hannah@heb.com>
Co-authored-by: Tirumerla <57160285+tirumerla@users.noreply.github.com>
Co-authored-by: Cloud Posse Bot (CI/CD) <bot@cloudposse.com>
Co-authored-by: osterman <osterman@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Renovate Bot <bot@renovateapp.com>
Co-authored-by: Adam Kunicki <kunickiaj@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>
Co-authored-by: Shabih <81912045+shabih-paystack@users.noreply.github.com>
Co-authored-by: Matt Barclay <mbarclay@gmail.com>
Co-authored-by: nitrocode <nitrocode@users.noreply.github.com>
Co-authored-by: Paulo Edgar Castro <pauloedgarcastro@gmail.com>
Co-authored-by: Szymon Matejczyk <smatejczyk@gmail.com>
Co-authored-by: Philipp Erbelding <philipp@copythat.de>
Co-authored-by: dylanbannon <dylanbannon5@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@osterman osterman osterman approved these changes

@maximmi maximmi maximmi left review comments

@goruha goruha Awaiting requested review from goruha

@aknysh aknysh Awaiting requested review from aknysh

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@3h4x @osterman @maximmi