resource/aws_emr_cluster: Add master_instance_group and core_instance_group configuration blocks, deprecate other instance group configuration methods

[bflad]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Closes #8245

Output from acceptance testing (will kick off full resource testing shortly):

--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_AutoscalingPolicy (487.78s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_BidPrice (1006.42s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_InstanceCount (1004.90s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_InstanceType (869.47s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Migration_CoreInstanceType (415.45s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Migration_InstanceGroup (485.50s)
--- PASS: TestAccAWSEMRCluster_CoreInstanceGroup_Name (769.30s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_BidPrice (849.98s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_InstanceType (756.42s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_Migration_InstanceGroup (414.36s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_Migration_MasterInstanceType (423.02s)
--- PASS: TestAccAWSEMRCluster_MasterInstanceGroup_Name (735.58s)

[nywilken]

This is looking pretty solid, and the upgrade guide is nicely written 👍🏼

I’m still working through the resource changes but there are a few minor things worth calling out.

[nywilken]

Let’s switch to index notation for the verbs as opposed to positional.

[bflad]

I updated this unrelated test to index notation, but I'd really prefer to treat testing cleanup separate from code changes for this resource so we can simplify/fix all the test configurations together (e.g. use testAccAWSEmrClusterConfigBaseVpc) and keep the review here concise.

--- PASS: TestAccAWSEMRCluster_security_config (404.88s)

[nywilken]

💯

[nywilken]

LGTM 👍 👍

[bflad]

This has been released in version 2.11.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[dthauvin]

Hello @bflad why remove the capabilities of launching an instance_group Core and TASK in a single terraform apply with aws_emr_cluster ressource .
When we used BOTO or other SDK with a single API call we can :

```python
client.run_job_flow(
            Name=cluster_name,
            LogUri=bucket_log,
            ReleaseLabel='emr-5.20.0',
            Instances={
                'KeepJobFlowAliveWhenNoSteps': False,
                'TerminationProtected': False,
                'Ec2SubnetId': subnet_id,
                'Ec2KeyName': 'key_pair',
                'InstanceGroups': [
                    {
                        'Name': 'master',
                                'InstanceRole': 'MASTER',
                        'InstanceType': instance_type,
                        'InstanceCount': 1,
                        'Market': 'SPOT',
                    },
                    {
                        'Name': 'core',
                                'InstanceRole': 'CORE',
                                'InstanceType': instance_type,
                        'InstanceCount': instance_core_count,
                        'Market': 'SPOT',
                        'AutoScalingPolicy': autoscaling_policy
                    },
                    {
                        'Name': 'task',
                                'InstanceRole': 'TASK',
                                'InstanceType': instance_type,
                        'InstanceCount': instance_task_count,
                        'Market': 'SPOT',
                        'AutoScalingPolicy': autoscaling_policy,
                    }
                ]
            },
            VisibleToAllUsers=True,
            Steps=steps,
            BootstrapActions=bootstrapping,
            Applications=applications,
            JobFlowRole='EMR_EC2_DefaultRole',
            ServiceRole='EMR_DefaultRole',
            AutoScalingRole='EMR_AutoScaling_DefaultRole',
            Tags=tags,
            EbsRootVolumeSize=ebs_size,
            SecurityConfiguration=security_configuration
        )

thank you

[anouvel]

Hi,

I think this behaviour could be the same as security group inline rule vs security group rule as resources. @bflad Any opinion / suggestion?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!