feat(deadline): configure identity registration settings for deadline clients #576

jusiskin · 2021-09-20T23:01:59Z

Summary

This change makes RFDK automatically configure Deadline Secrets Management identity registration settings based on the RFDK constructs that are present in a CDK application.

The changes are broken out by construct below:

`DeploymentInstance` (new)

This construct is an extraction/generalization of the CDK/CloudFormation pattern used by the Repository construct. It deploys an EC2 Auto-Scaling Group with a size of one. The Auto-Scaling Group is configured to require one CloudFormation success signal which is only sent after the user-data completes without error. The user data can optionally (by default enabled) self-terminate the instance by scaling down its own auto-scaling group to zero.

For now, this construct is being kept internal to RFDK and not exported. It was designed with the possibility of making it a public construct in the future.

`SecretsManagementIdentityRegistration` (new)

The SecretsManagementIdentityRegistration construct is also introduced in this PR. It is responsible for configuring the user data (on a DeploymentInstance) to configure the identity registration settings. It adds IAM permissions and user-data to perform the following steps:

download and install the Deadline Client
configure Deadline to make a direct connection to the Deadline Repository
run a python script that
1. fetches CIDR ranges for subnets involved
2. fetches existing identity registration settings
3. creates/updates/deletse identity registration settings so that they match what's desired

The main API of this construct is the .addSubnetIdentityRegistrationSetting(...) method which a caller can use to specify the subnets of the client, the desired secrets management role and registration status.

`RenderQueue` (modified)

This PR introduces a new configureSecretsManagementAutoRegistration(...) method to IRenderQueue. When called on the RenderQueue, it will lazily create DeploymentInstance and SecretsManagementIdentityRegistration constructs and delegate the work to the SecretsManagementIdentityRegistration construct. It also accepts a dependent prop which is used to enforce a CloudFormation dependency on a dependent resource of the caller's choosing.

The CloudWatch logs for the DeploymentInstance are sent to the /renderfarm/ConfigureRepository log group with the intention that the behavior of the deployment instance can be augmented in the future.

`WorkerInstanceFleet` / `ConfigureSpotEventPlugin` / `UsageBasedLicensing` (modified)

The RFDK constructs that deploy Deadline Clients were modified to call the configureSecretsManagementAutoRegistration(...) method

`IVersion` / `VersionQuery` / `ThinkboxDockerRecipes.version` (modified)

The IVersion interface and classes that implement it were modified to return the path to the Deadline Client installer location. This was required in order for the SecretsManagementIdentityRegistration construct to be able to install the Deadline Client on instance deployed by the DeploymentInstance.

Testing

Unit tests were updated to reflect all of the changes

Automatic identity registration of Deadline Clients was end-to-end tested by deploying the All-In-AWS-Infrastructure-Basic TypeScript example app with Thinkbox Usage-Based Licensing (UBL). Verified that:

Client identities were properly being assigned with the "Client" role and statuses by checking in a connected Deadline Monitor
Logs in the renderfarm/ConfigureRepository log group showed correct behavior
Clients were able to access Deadline-managed secrets. Thinkbox UBL requires the "Client" role to be assigned properly. Renders were successful.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

horsmand

Overall this looks really good, none of my comments should require changing too much in the overall architecture of this. I think the deployment instance could be a really powerful tool.

packages/aws-rfdk/lib/core/lib/deployment-instance.ts

packages/aws-rfdk/lib/deadline/lib/render-queue.ts

packages/aws-rfdk/lib/deadline/lib/secrets-management.ts

packages/aws-rfdk/lib/deadline/scripts/python/configure_identity_registration_settings.py

packages/aws-rfdk/lib/deadline/test/repository.test.ts

packages/aws-rfdk/lib/deadline/test/secrets-management.test.ts

packages/aws-rfdk/lib/core/test/deployment-instance.test.ts

kozlove-aws

Great job. Just a couple moments that I want clarify.

packages/aws-rfdk/lib/deadline/lib/secrets-management-ref.ts

packages/aws-rfdk/lib/deadline/scripts/python/configure_identity_registration_settings.py

jericht

Looks great! Just have a few minor comments

packages/aws-rfdk/lib/deadline/scripts/python/configure_identity_registration_settings.py

packages/aws-rfdk/lib/core/lib/deployment-instance.ts

packages/aws-rfdk/lib/deadline/lib/secrets-management.ts

packages/aws-rfdk/lib/deadline/lib/stage.ts

packages/aws-rfdk/lib/deadline/test/usage-based-licensing.test.ts

packages/aws-rfdk/lib/deadline/scripts/python/configure_identity_registration_settings.py

… client instances

Co-authored-by: David Horsman <56004724+horsmand@users.noreply.github.com>

- reorder DeploymentInstance props - document internal use of RenderQueue.configureSecretsManagementAutoRegistration - move SecretsManagementIdentityRegistration construct ID to constant - Add TSDoc header for SecretsManagementIdentityRegistration construct - Factor out constant for MacOS Deadline path file - Remove artifacts from FileSecret in configure_identity_registration_settings.py - fixes to encoding, logging and error-handling of deadline path on MacOS

…ecret read access

… registration configuration

…nets or private

- TSDoc improvements - Use MachineImage.latestAmazonLinux - Remove unnecessary test parameterization - Add code comments - Fix typos/punctuation/redundant syntax

jericht

LGTM!

I left a few minor comments, but it's mostly tiny formatting/organization in the unit tests, so not holding back the PR for it.

packages/aws-rfdk/lib/deadline/test/secrets-management.test.ts

packages/aws-rfdk/lib/deadline/test/usage-based-licensing.test.ts

packages/aws-rfdk/lib/deadline/test/worker-fleet.test.ts

horsmand

I'm in the same boat as Jericho. This looks good as-is. I'm going to give my approval but leave it for Josh to look over the comments and decide whether to fix them or merge.

packages/aws-rfdk/lib/deadline/test/configure-spot-event-plugin.test.ts

- flatten jest test heirarchies - fix typos - clarify test name

jusiskin added the contribution/core This is a PR that came from AWS. label Sep 21, 2021

jusiskin force-pushed the sm_worker_auto_registration branch 5 times, most recently from f58da2e to 8c4292d Compare September 27, 2021 17:43

jusiskin force-pushed the sm_worker_auto_registration branch from 8c4292d to 26cd4e1 Compare September 30, 2021 22:08

jusiskin marked this pull request as ready for review October 1, 2021 16:32

jusiskin changed the title ~~feat(deadline): configure identity registration settings for Deadline…~~ feat(deadline): configure identity registration settings for deadline clients Oct 1, 2021

jusiskin force-pushed the sm_worker_auto_registration branch from 26cd4e1 to 0cc7f99 Compare October 1, 2021 16:59

jusiskin mentioned this pull request Oct 4, 2021

chore(deadline): added tests for configure_identity_registration_settings.py #593

Merged

horsmand suggested changes Oct 5, 2021

View reviewed changes

kozlove-aws reviewed Oct 5, 2021

View reviewed changes

packages/aws-rfdk/lib/deadline/lib/secrets-management-ref.ts Show resolved Hide resolved

packages/aws-rfdk/lib/deadline/scripts/python/configure_identity_registration_settings.py Show resolved Hide resolved

jericht requested changes Oct 5, 2021

View reviewed changes

kozlove-aws previously approved these changes Oct 6, 2021

View reviewed changes

jericht mentioned this pull request Oct 7, 2021

feat(integ): add secrets management identity registration settings tests #596

Merged

jusiskin and others added 9 commits October 7, 2021 17:23

feat(deadline): configure identity registration settings for Deadline…

f6f8a29

… client instances

Apply suggestions from code review

215226e

Co-authored-by: David Horsman <56004724+horsmand@users.noreply.github.com>

feat(deadline): warn about dedicated subnets during synthesis

c94a27a

chore(deadline): optimize DeploymentInstance tests

2d3c148

test(deadline): ensure Repository.configureClientInstance grants db s…

a8093ce

…ecret read access

chore(deadline): clean up code that generates parameters for identity…

4d9a05b

… registration configuration

fix(deadline): deploy RenderQueue DeploymentInstance to specified sub…

b56dfa0

…nets or private

chore(deadline): apply CR feedback

0cb6210

- TSDoc improvements - Use MachineImage.latestAmazonLinux - Remove unnecessary test parameterization - Add code comments - Fix typos/punctuation/redundant syntax

jusiskin dismissed kozlove-aws’s stale review via 0cb6210 October 7, 2021 17:30

jusiskin force-pushed the sm_worker_auto_registration branch from 4bb8df5 to 0cb6210 Compare October 7, 2021 17:30

jusiskin mentioned this pull request Oct 7, 2021

feat(examples): use dedicated subnets in All-In-AWS-Infrastructure-Basic example #598

Merged

jusiskin added 2 commits October 7, 2021 21:15

fix(deadline): typo in method name

d9a7184

chore(deadline): add test code comment

d35d9a3

fix(deadline): fixed method call signature mismatch

c2be47e

jericht previously approved these changes Oct 7, 2021

View reviewed changes

horsmand previously approved these changes Oct 8, 2021

View reviewed changes

packages/aws-rfdk/lib/deadline/test/configure-spot-event-plugin.test.ts Outdated Show resolved Hide resolved

chore(deadline): address PR feedback

5565a47

- flatten jest test heirarchies - fix typos - clarify test name

jusiskin dismissed stale reviews from horsmand and jericht via 5565a47 October 8, 2021 14:31

horsmand approved these changes Oct 8, 2021

View reviewed changes

jericht approved these changes Oct 8, 2021

View reviewed changes

horsmand merged commit b9082b2 into aws:feature_enable_secret_manager Oct 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(deadline): configure identity registration settings for deadline clients #576

feat(deadline): configure identity registration settings for deadline clients #576

jusiskin commented Sep 20, 2021 •

edited

Loading

horsmand left a comment

kozlove-aws left a comment

jericht left a comment

jericht left a comment

horsmand left a comment

feat(deadline): configure identity registration settings for deadline clients #576

feat(deadline): configure identity registration settings for deadline clients #576

Conversation

jusiskin commented Sep 20, 2021 • edited Loading

Summary

DeploymentInstance (new)

SecretsManagementIdentityRegistration (new)

RenderQueue (modified)

WorkerInstanceFleet / ConfigureSpotEventPlugin / UsageBasedLicensing (modified)

IVersion / VersionQuery / ThinkboxDockerRecipes.version (modified)

Testing

horsmand left a comment

Choose a reason for hiding this comment

kozlove-aws left a comment

Choose a reason for hiding this comment

jericht left a comment

Choose a reason for hiding this comment

jericht left a comment

Choose a reason for hiding this comment

horsmand left a comment

Choose a reason for hiding this comment

jusiskin commented Sep 20, 2021 •

edited

Loading

`DeploymentInstance` (new)

`SecretsManagementIdentityRegistration` (new)

`RenderQueue` (modified)

`WorkerInstanceFleet` / `ConfigureSpotEventPlugin` / `UsageBasedLicensing` (modified)

`IVersion` / `VersionQuery` / `ThinkboxDockerRecipes.version` (modified)