Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resource/aws_emr_cluster: Add master_instance_group configuration block instance_count argument (support multiple master nodes) #9235

Merged
merged 3 commits into from
Jul 16, 2019
Merged

resource/aws_emr_cluster: Add master_instance_group configuration block instance_count argument (support multiple master nodes) #9235

merged 3 commits into from
Jul 16, 2019

Conversation

bflad
Copy link
Contributor

@bflad bflad commented Jul 4, 2019 •
edited
Loading

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Closes #9226

Release note for CHANGELOG:

* resource/aws_emr_cluster: Add `master_instance_group` configuration block `instance_count` argument (support multiple master nodes)

Output from acceptance testing:

--- PASS: TestAccAWSEMRCluster_additionalInfo (424.70s)
--- PASS: TestAccAWSEMRCluster_basic (466.79s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_Migration_InstanceGroup (472.78s)
--- PASS: TestAccAWSEMRCluster_instance_group_update (512.31s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Migration_InstanceGroup (535.17s)
--- PASS: TestAccAWSEMRCluster_updateAutoScalingPolicy (536.05s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_AutoscalingPolicy (542.83s)
--- PASS: TestAccAWSEMRCluster_instance_group_names (561.06s)
--- PASS: TestAccAWSEMRCluster_instance_group (580.49s)
--- PASS: TestAccAWSEMRCluster_instance_group_EBSVolumeType_st1 (582.52s)
--- PASS: TestAccAWSEMRCluster_Kerberos_ClusterDedicatedKdc (586.17s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Migration_CoreInstanceType (589.09s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_InstanceCount (724.61s)
--- PASS: TestAccAWSEMRCluster_configurationsJson (741.12s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_InstanceType (826.73s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Name (827.15s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_InstanceCount (963.35s)
--- PASS: TestAccAWSEMRCluster_keepJob (405.56s)
--- PASS: TestAccAWSEMRCluster_bootstrap_ordering (443.44s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_Migration_MasterInstanceType (566.70s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_BidPrice (992.48s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_BidPrice (1013.58s)
--- PASS: TestAccAWSEMRCluster_Step_Basic (519.91s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_InstanceType (1043.61s)
--- PASS: TestAccAWSEMRCluster_security_config (576.98s)
--- PASS: TestAccAWSEMRCluster_Step_Multiple (524.76s)
--- PASS: TestAccAWSEMRCluster_terminationProtected (529.49s)
--- PASS: TestAccAWSEMRCluster_custom_ami_id (504.52s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_Name (798.86s)
--- PASS: TestAccAWSEMRCluster_tags (701.49s)
--- PASS: TestAccAWSEMRCluster_s3Logging (712.11s)
--- PASS: TestAccAWSEMRCluster_Step_ConfigMode (868.38s)
--- PASS: TestAccAWSEMRCluster_root_volume_size (791.87s)

@bflad
resource/aws_emr_cluster: Add master_instance_group configuration blo…
5b1e6cc 
…ck instance_count argument (support multiple master nodes)

Reference: #9226

Output from acceptance testing:

```
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_InstanceCount (1062.33s)
```
@bflad bflad added enhancement Requests to existing resources that expand the functionality or scope. service/emr Issues and PRs that pertain to the emr service. labels Jul 4, 2019
@bflad bflad requested a review from a team July 4, 2019 01:00
@ghost ghost added size/L Managed by automation to categorize the size of a PR. documentation Introduces or discusses updates to documentation. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Jul 4, 2019
@bflad bflad mentioned this pull request Jul 4, 2019
Closed
mattburgess
mattburgess reviewed Jul 4, 2019
View reviewed changes
website/docs/r/emr_cluster.html.markdown Outdated
# This configuration is for illustrative purposes and highlights
# only relevant configurations for working with this functionality.

# Map public IP on launch must be enabled for the subnet
Copy link
Collaborator

@mattburgess mattburgess Jul 4, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs, you can use private subnets too. From https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-ha-launch.html:

You can launch an EMR cluster with multiple master nodes in both public and private VPC subnets

Same comment applies to line 347.

Is the requirement for map_public_ip_on_launch = true a restriction of the API? Obviously, assigning a public IP on a private subnet isn't going to do anything particularly useful :)

BTW, thanks very much for such a quick turnaround on the issue I raised!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see it might only be a requirement if the cluster is in a public subnet. Do you have a working private VPC subnet multi-master setup?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet; given that the restriction is only mentioned in the comments though, and not enforced in the implementation I should be able to give this a spin, time permitting, and verify.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. Your change works just fine on a private EMR cluster without the private subnets having map_public_ip_on_launch = true in the config.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, thanks, @mattburgess!

@bflad
docs/resource/aws_emr_cluster: Adjust wording that multi-master only …
4c628e7 
…requires public subnets to have public IP on launch

Reference: #9235 (review)
nywilken
nywilken approved these changes Jul 16, 2019
View reviewed changes
Copy link
Contributor

@nywilken nywilken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a suggestion on the argument documentation, but this is otherwise good to go 👍

website/docs/r/emr_cluster.html.markdown Outdated Show resolved Hide resolved
@nywilken
Copy link
Contributor 

--- PASS: TestAccAWSEMRCluster_Step_Multiple (612.41s)
--- PASS: TestAccAWSEMRCluster_Kerberos_ClusterDedicatedKdc (621.12s)
--- PASS: TestAccAWSEMRCluster_s3Logging (629.62s)
--- PASS: TestAccAWSEMRCluster_bootstrap_ordering (637.53s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_InstanceType (811.04s)
--- PASS: TestAccAWSEMRCluster_visibleToAllUsers (822.40s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_BidPrice (824.04s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_Name (856.52s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Name (911.31s)
--- PASS: TestAccAWSEMRCluster_additionalInfo (386.90s)
--- PASS: TestAccAWSEMRCluster_configurationsJson (410.65s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Migration_InstanceGroup (480.60s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_InstanceCount (992.40s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Migration_CoreInstanceType (496.07s)
--- PASS: TestAccAWSEMRCluster_instance_group (560.05s)
--- PASS: TestAccAWSEMRCluster_Step_ConfigMode (1058.89s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_InstanceCount (538.57s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_AutoscalingPolicy (548.28s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_InstanceType (905.75s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_BidPrice (1007.50s)

@bflad @nywilken
Update website/docs/r/emr_cluster.html.markdown
31a7140 
Co-Authored-By: Wilken Rivera <dev@wilkenrivera.com>
@bflad bflad added this to the v2.20.0 milestone Jul 16, 2019
@bflad bflad merged commit 10dac2f into master Jul 16, 2019
@bflad bflad deleted the f-aws_emr_cluster-master_instance_group-instance_count branch July 16, 2019 18:59
bflad added a commit that referenced this pull request Jul 16, 2019
@bflad
Update CHANGELOG for #9235
eeed649
@ghost
Copy link

ghost commented Nov 2, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Nov 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@mattburgess mattburgess mattburgess left review comments

@nywilken nywilken nywilken approved these changes

Assignees
No one assigned
Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. service/emr Issues and PRs that pertain to the emr service. size/L Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v2.20.0
Development

Successfully merging this pull request may close these issues.

EMR: Support multiple master nodes
3 participants
@bflad @nywilken @mattburgess