Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(stepfunctions-tasks): missing tags & perms for emr cluster creation #28327

Merged
merged 3 commits into from
Dec 11, 2023

Conversation

msambol
Copy link
Contributor

@msambol msambol commented Dec 11, 2023

Per the documentation:

  • To use managed policies, pass the user tag for-use-with-amazon-emr-managed-policies = true when you provision a cluster with the CLI, SDK, or another method.
  • For resources that are not created by Amazon EMR, you must add tags to those resources. For example, you must tag Amazon EC2 subnets, EC2 security groups (if not created by Amazon EMR), and VPCs (if you want Amazon EMR to create security groups).

Also, AmazonEMRServicePolicy_v2 only has iam:PassRole on the default EMR roles and needs this on the cluster role created by the CDK.

Running the step function:

Screenshot 2023-12-11 at 5 24 09 AM Screenshot 2023-12-11 at 5 24 03 AM

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added star-contributor [Pilot] contributed between 25-49 PRs to the CDK p2 labels Dec 11, 2023
@aws-cdk-automation aws-cdk-automation requested a review from a team December 11, 2023 14:13
@@ -192,6 +193,14 @@ export class EmrCreateCluster extends sfn.TaskStateBase {
this._serviceRole = this.props.serviceRole ?? this.createServiceRole();
this._clusterRole = this.props.clusterRole ?? this.createClusterRole();

// Service role must be able to iam:PassRole on the cluster role
this._clusterRole.grantPassRole(this._serviceRole);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AmazonEMRServicePolicy_v2 only grants iam:PassRole on the default role. Need this here for the cluster role we create.

'elasticmapreduce:RunJobFlow',
'elasticmapreduce:DescribeCluster',
'elasticmapreduce:TerminateJobFlows',
'elasticmapreduce:AddTags',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added AddTags.

conditions: {
StringEquals: { 'aws:RequestTag/for-use-with-amazon-emr-managed-policies': 'true' },
},
}),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition isn't needed and was actually causing it to fail.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain your reasoning for why it's not needed? Also how was it causing the integ tests to fail?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When included, I get the following error:

EMR service role arn:aws:iam::<REDACTED>:role/aws-cdk-emr-create-cluste-EmrCreateClusterServiceRo-GrrjLUAR1a2Q is invalid

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, I see the part where it's being added to the _base_tags as well. Thanks for clarifying.

new sfn.StateMachine(stack, 'SM', {
definition: step,
});

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this wasn't actually deploying a state machine. I added this and ran the step function.

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Dec 11, 2023
conditions: {
StringEquals: { 'aws:RequestTag/for-use-with-amazon-emr-managed-policies': 'true' },
},
}),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain your reasoning for why it's not needed? Also how was it causing the integ tests to fail?

paulhcsun
paulhcsun previously approved these changes Dec 11, 2023
Copy link
Contributor

@paulhcsun paulhcsun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me. This PR also adds additional integ tests that deploys EMR cluster from a state machine and is working. Thanks for the fix!

@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Dec 11, 2023
@mergify mergify bot dismissed paulhcsun’s stale review December 11, 2023 22:33

Pull request has been modified.

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Dec 11, 2023
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: b125464
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

mergify bot commented Dec 11, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 14e5e50 into aws:main Dec 11, 2023
14 checks passed
@msambol msambol deleted the emr-policy-tag branch December 11, 2023 23:02
@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p2 star-contributor [Pilot] contributed between 25-49 PRs to the CDK
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants