Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(deadline): add ConfigureSpotEventPlugin and SpotEventPluginFleet constructs #279

Merged
merged 23 commits into from
Feb 26, 2021

Conversation

ryyakobe
Copy link
Contributor

@ryyakobe ryyakobe commented Dec 18, 2020

Fixes #131

Adds a new ConfigureSpotEventPlugin construct that can be used to create and save Spot Event Plugin Configurations.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Copy link
Contributor

@ddneilson ddneilson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be really great to have an abstraction layer overtop of the specific GET/POST requests.

ex: Have a EventPluginRequests class. Constructor takes a DeadlineClient, and the class itself has methods that make it easier to construct the specific requests for, say, saving ServerData and Configuring the SpotEventPlugin.

}
}

public GetRequest(path: string, requestOptions?: https.RequestOptions) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should put the return-type declaration on these -- Promise<Response>

Or, alternatively, declare them as public async GetRequest(...): Response

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@jusiskin jusiskin added the contribution/core This is a PR that came from AWS. label Jan 5, 2021
@ryyakobe ryyakobe force-pushed the ryyakobe/sep_configuration branch 3 times, most recently from cb3ac0f to 3b91315 Compare January 28, 2021 04:13
@ryyakobe ryyakobe changed the title feat(deadline): adding dealine client to connect to rcs feat(deadline): add SEPConfigurationSetup construct Jan 30, 2021
@ryyakobe ryyakobe force-pushed the ryyakobe/sep_configuration branch 2 times, most recently from 9dee4b3 to dadc86f Compare January 30, 2021 02:48
@ryyakobe ryyakobe marked this pull request as ready for review February 5, 2021 02:22
@horsmand horsmand self-requested a review February 8, 2021 16:23
Copy link
Contributor

@jusiskin jusiskin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a partial review - I am still going through code, but I thought it would help to post some of my feedback so far.

packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-spotfleet.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-spotfleet.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
Copy link
Contributor

@horsmand horsmand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to get through the examples still, but this is a long one so I thought I'd give some feedback now.

packages/aws-rfdk/lib/core/lib/runtime-info.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/render-queue.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
Copy link
Contributor

@jusiskin jusiskin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second batch of code review feedback.

One other thing I wanted to request is that we add documentation for this new construct in packages/aws-rfdk/lib/deadline/README.md. This is a good place to provide more detailed documentation for constructs that we don't want to necessarily want to cram into the TSDoc string.

packages/aws-rfdk/lib/core/lib/runtime-info.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-configuration.ts Outdated Show resolved Hide resolved
/**
* The Worker Extra Info column to be used to display AWS Instance Status
* if the instance has been marked to be stopped or terminated by EC2 or Spot Event Plugin.
* See "AWS Instance Status" option at https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/event-spot.html?highlight=spot%20even%20plugin#event-plugin-configuration-options
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few things:

  1. This link is pretty useful for the overall configuration of SEP. Can we also add it to the TSDoc string for the SEPConfigurationProperties interface as a general reference?
  2. The URL contains ?highlight=... which makes those terms appear highlighted by our docs. Can we remove that from the link?
  3. Does it make sense to also link to the Worker extra info docs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I though the same earlier and added it before.
  2. Good catch, thanks.
  3. Thanks! Good idea!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I missed it, I believe 2 still needs to be addressed.

packages/aws-rfdk/lib/deadline/lib/sep-spotfleet.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-spotfleet.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-spotfleet.ts Outdated Show resolved Hide resolved
packages/aws-rfdk/lib/deadline/lib/sep-spotfleet.ts Outdated Show resolved Hide resolved
ManagedPolicy.fromAwsManagedPolicyName('AWSThinkboxDeadlineSpotEventPluginWorkerPolicy'),
],
description: `Role for ${id} in region ${this.env.region}`,
roleName: 'DeadlineSpot' + this.calculateResourceTagValue([this]),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to be careful here when specifying what CDK refers to as a physical name:

Assigning physical names to resources has some disadvantages in AWS CloudFormation. Most importantly, any changes to deployed resources that require a resource replacement, such as changes to a resource's properties that are immutable after creation, will fail if a resource has a physical name assigned. If you end up in a state like that, the only solution is to delete the AWS CloudFormation stack, then deploy the AWS CDK app again. See the AWS CloudFormation documentation for details.

Is there any reason why we can't just not specify roleName and have CloudFormation generate a physical name for us?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement of the Spot Event Plugin. It requires that the role name start with "DeadlineSpot"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we have this requirement only because in AWSThinkboxDeadlineSpotEventPluginAdminPolicy
we have:
"Resource": [ "arn:aws:iam::*:role/aws-ec2-spot-fleet-tagging-role", "arn:aws:iam::*:role/DeadlineSpot*" ]
We might then use the approach we used with the fleetRole: just create a new role and add policy to the render queue, but then we would have a few unused statements in the RQ policy, which is not great. I don't think it's worth it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no harm in those statements being unused in the managed policy...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I agree - unused statements are unfortunate but acceptable here. Adding a broadening policy to the RQ seems like a good way to avoid specifying a physical name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be working after allowing PassRole. In AWSThinkboxDeadlineSpotEventPluginAdminPolicy we also allow GetRole for both fleet role and fleet instance role, but it seems that everything is working if I only allow PassRole, so I will keep the permissions strict for now.
We will be able to allow GetRole n the future, if something won't work.

@aws aws deleted a comment from horsmand Feb 10, 2021
Copy link
Contributor

@jusiskin jusiskin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work Roman! This will add huge value for RFDK users.

@jusiskin jusiskin changed the title feat(deadline): add ConfigureSpotEventPlugin construct feat(deadline): add ConfigureSpotEventPlugin and SpotEventPluginFleet constructs Feb 26, 2021
@jusiskin jusiskin merged commit b418e8a into aws:mainline Feb 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

RenderQueue endpoint does not expose fully qualified domain name for the load balancer
5 participants