Support for AWS Auto Scaling EMR Clusters

[cpower]

Hi there,

It would be great to have support for the new EMR Auto Scaling feature, ideally including support for EBS volumes, and possibly spot market bids. This would of course also require the ability to configure and manage the auto-scaling policy rules (scale out/in, min/max instances).

Affected Resource(s)

• aws_emr_cluster
• aws_emr_instance_group

References

Auto Scaling for EMR Clusters
https://aws.amazon.com/blogs/aws/new-auto-scaling-for-emr-clusters/

[ghost]

+1

[ststarcount]

@cpower a colleague of mine has recently forked and added this functionality. We could write some tests, and open a pr?

[cpower]

That's great, would be happy to take a look and provide some feedback

[stack72]

Hi @shaun-tillyard - we would love a PR for this functionality :)

[hylaride]

+1

[grubernaut]

Re-opening, working on scaling policy now

[cpower]

Just curious, is auto-scaling and ebs volume support going to live in the aws_emr_instance_group resource, or as part of the aws_emr_cluster. If you look at the AWS SDK they've generalized their approach to specifying the nodes that are part of an EMR cluster to instance groups for master, core, and task nodes including scaling policies and ebs volume definitions.

[grubernaut]

@cpower I was planning on adding them to both resources, as currently the aws_emr_cluster manages the core instance groups, and the aws_emr_instance_group manages the task nodes.

[cpower]

One option, although clearly way more invasive would be to remove it from the aws_emr_cluster and using aws_emr_instance_group, which might be a more general and extensible approach, although clearly more invasive and less backwards compatible. However given how new the support in Terraform for EMR is, perhaps. I'd look for input from the Hashicorp folks on this one if we were to go down that path.

[grubernaut]

Currently aws_emr_instance_group only supports creating the TASK instance type, not MASTER or CORE. We could add that functionality to the aws_emr_instance_group resource, but I believe that would create a breaking change for how aws_emr_cluster handles the CORE instance group.

[cpower]

I haven't looked closely enough at the code to see, but would't be surprised if it did. Wondering if it is worth it though to more closely align to the structure of the AWS GO SDK. Although I don't have backwards compatibility constraints myself, so perhaps I'm biased.

[grubernaut]

I'll speak with @catsby and others internally before diving too deep into how we structure these changes, but it would make sense to separate the aws_emr_cluster resource from the aws_emr_instance_group resource, such that the aws_emr_instance_group resource could manage the CORE instance group as well.

[cpower]

Yep, definitely worth talking it through before diving too deeply on that.

[niclic]

Just came across this before opening an issue to add EMR scaling support to Application Auto Scaling. This long standing PR to add Spot Fleet support is about to land today, so might make sense to revisit this in light of the changes in this PR:

#8697

[t-pascal]

We are very interested in this functionality as well and can help test when it's available.

[hylaride]

Hey guys,

Just curious if there are any roadmap plans to add this?

[stack72]

Closed via #12823

[darrenhaken]

Hey all I have a PR set on the AWS provider for this

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.