feat(deadline): add SEPConfigurationSetup construct

examples/deadline/All-In-AWS-Infrastructure-SEP/README.md

@@ -10,7 +10,7 @@ _**Note:** This application is an illustrative example to showcase some of the c
 
 ## Architecture
 
-This sample application deploys a basic Deadline Render farm that is configured to use Deadlines [Spot Event Plugin](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/event-spot.html). 
+This sample application deploys a basic Deadline Render farm that is configured to use Deadlines [Spot Event Plugin](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/event-spot.html).
 
 ### Components
 
@@ -22,15 +22,15 @@ The Repository component contains the database and file system that store persis
 
 #### Render Queue
 
-The Render Queue component contains the fleet of [Deadline Remote Connection Server](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/remote-connection-server.html) instances behind an [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html). This acts as the central service for Deadline applications and is the only component that interacts with the Repository.  When comparing this component to the "All in AWS Infrastructure - Basic" example it has been granted additional permissions in order to use the Spot Event Plugin.    
+The Render Queue component contains the fleet of [Deadline Remote Connection Server](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/remote-connection-server.html) instances behind an [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html). This acts as the central service for Deadline applications and is the only component that interacts with the Repository. When comparing this component to the "All in AWS Infrastructure - Basic" example it has been granted additional permissions in order to use the Spot Event Plugin.
 
 #### Spot Event Plugin Configurations
 
-The Spot Event plugin requires additional Roles for both Deadline's Resource Tracker and the Spot Workers that are created and a Security Group to allow your Spot workers the ability to access the Render Queue.
+Spot Event plugin Configuration Setup component generates and saves the [Spot Fleet Request Configurations](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/event-spot.html#spot-fleet-request-configurations). The Spot Workers that are created will be configured to connect to the Render Queue. The Spot Event plugin requires additional Role for Deadline's Resource Tracker.
 
 ## Prerequisites
 
-- The Spot Fleet Configuration requires an [Amazon Machine Image (AMI)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) with the Deadline Worker application installed. This AMI must have Deadline Installed and should be configured to connect to your repository.  For additional information on setting up your AMI please see the [Spot Event Plugin Documentation](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/event-spot.html).
+- The Spot Fleet Configuration requires an [Amazon Machine Image (AMI)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) with the Deadline Worker application installed. This AMI must have Deadline Installed and should be configured to connect to your repository. For additional information on setting up your AMI please see the [Spot Event Plugin Documentation](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/event-spot.html).
 - You have setup and configured the AWS CLI
 - Your AWS account already has CDK bootstrapped in the desired region by running `cdk bootstrap`
 - You must have NodeJS installed on your system

examples/deadline/All-In-AWS-Infrastructure-SEP/python/README.md

@@ -32,7 +32,41 @@ These instructions assume that your working directory is `examples/deadline/All-
     popd
     pip install ../../../../dist/python/aws-rfdk-<version>.tar.gz
     ```
-3. Stage the Docker recipes for `RenderQueue`:
+4.  Change the value in the `deadline_client_linux_ami_map` variable in `package/config.py` to include the region + AMI ID mapping of your EC2 AMI(s) with Deadline Worker. You can use the following AWS CLI query to find AMI ID's:
+    ```bash
+    aws --region <region> ec2 describe-images \
+    --owners 357466774442 \
+    --filters "Name=name,Values=*Worker*" "Name=name,Values=*<version>*" \
+    --query 'Images[*].[ImageId, Name]' \
+    --output text
+    ```
+
+    And enter it into this section of `package/config.py`:
+    ```python
+    # For example, in the us-west-2 region
+    self.deadline_client_linux_ami_map: Mapping[str, str] = {
+        'us-west-2': '<your ami id>'
+    }
+    ```
+
+    ---
+
+    **Note:** The next two steps are optional. You may skip these if you do not need SSH access into your render farm.
+
+    ---
+5.  Create an EC2 key pair to give you SSH access to the render farm:
+
+    ```bash
+    aws ec2 create-key-pair --key-name <key-name>
+    ```
+6.  Change the value of the `key_pair_name` variable in `package/config.py` to your value for `<key-name>` in the previous step:
+
+    **Note:** Save the value of the `"KeyMaterial"` field as a file in a secure location. This is your private key that you can use to SSH into the render farm.
+
+    ```python
+    self.key_pair_name: Optional[str] = '<your key pair name>'
+    ```
+7. Stage the Docker recipes for `RenderQueue`:
 
     ```bash
     # Set this value to the version of RFDK your application targets
@@ -43,22 +77,13 @@ These instructions assume that your working directory is `examples/deadline/All-
 
     npx --package=aws-rfdk@${RFDK_VERSION} stage-deadline --output stage ${RFDK_DEADLINE_VERSION}
     ```
-4. Deploy all the stacks in the sample app:
+8. Deploy all the stacks in the sample app:
 
     ```bash
     cdk deploy "*"
     ```
 
-5. Connect to your Render Farm and open up the Deadline Monitor.
-
-6. Configure the Spot event plugin by following the directions in the [Spot Event Plugin documentation](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/event-spot.html) with the following considerations:
-
-    Use the default security credentials by using turning "Use Local Credentials" to False and leaving both "Access Key ID" and "Secret Access Key" blank.
-    Ensure that the Region your Spot workers will be launched in is the same region as your CDK application.
-    When Creating your Spot Fleet Requests, set the IAM instance profile to "DeadlineSpotWorkerRole" and set the security group to "DeadlineSpotSecurityGroup".
-    Configure your instances to connect to the Render Queue by either creating your AMI after launching your app and preconfiguring the AMI or by setting up a userdata in the Spot Fleet Request. (see the Spot Event Plugin documentation for additional information on configuring this connection.)
-
-7. Once you are finished with the sample app, you can tear it down by running:
+9. Once you are finished with the sample app, you can tear it down by running:
 
     ```bash
     cdk destroy "*"

examples/deadline/All-In-AWS-Infrastructure-SEP/python/package/app.py

@@ -10,11 +10,27 @@
     Environment
 )
 
+from aws_cdk.aws_ec2 import (
+    MachineImage
+)
+
 from .lib import (
     sep_stack,
 )
 
+from .config import config
+
 def main():
+    # ------------------------------
+    # Validate Config Values
+    # ------------------------------
+
+    if not config.key_pair_name:
+        print('EC2 key pair name not specified. You will not have SSH access to the render farm.')
+
+    if 'region' in config.deadline_client_linux_ami_map:
+        raise ValueError('Deadline Client Linux AMI map is required but was not specified.')
+
     # ------------------------------
     # Application
     # ------------------------------
@@ -33,6 +49,8 @@ def main():
     # ------------------------------
     sep_props = sep_stack.SEPStackProps(
         docker_recipes_stage_path=os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir, 'stage'),
+        key_pair_name=config.key_pair_name,
+        worker_machine_image=MachineImage.generic_linux(config.deadline_client_linux_ami_map)
     )
     service = sep_stack.SEPStack(app, 'SEPStack', props=sep_props, env=env)
 

examples/deadline/All-In-AWS-Infrastructure-SEP/python/package/config.py

@@ -0,0 +1,26 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from typing import (
+    List,
+    Mapping,
+    Optional
+)
+
+class AppConfig:
+    """
+    Configuration values for the sample app.
+
+    TODO: Fill these in with your own values.
+    """
+    def __init__(self):
+        # A map of regions to Deadline Client Linux AMIs.As an example, the Linux Deadline 10.1.12.1 AMI ID
+        # from us-west-2 is filled in. It can be used as-is, added to, or replaced. Ideally the version here
+        #  should match the one used for staging the render queue and usage based licensing recipes.
+        self.deadline_client_linux_ami_map: Mapping[str, str] = {'us-east-1': 'ami-0f5650d87270255ae'}
+
+        # (Optional) The name of the EC2 keypair to associate with the instances.
+        self.key_pair_name: Optional[str] = None
+
+
+config: AppConfig = AppConfig()

examples/deadline/All-In-AWS-Infrastructure-SEP/python/package/lib/sep_stack.py

@@ -1,7 +1,9 @@
 # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-import typing
+from typing import (
+    Optional
+)
 from dataclasses import dataclass
 
 from aws_cdk.core import (
@@ -11,19 +13,39 @@
     StackProps
 )
 from aws_cdk.aws_ec2 import (
+    IMachineImage,
+    InstanceClass,
+    InstanceSize,
+    InstanceType,
     SecurityGroup,
-    Vpc,
+    Vpc
 )
 from aws_cdk.aws_iam import (
     ManagedPolicy,
     Role,
-    ServicePrincipal 
+    ServicePrincipal
+)
+from aws_cdk.aws_elasticloadbalancingv2 import (
+    ApplicationProtocol
+)
+from aws_cdk.aws_route53 import (
+    PrivateHostedZone
 )
 from aws_rfdk.deadline import (
+    SEPConfigurationProperties,
     RenderQueue,
+    RenderQueueExternalTLSProps,
+    RenderQueueHostNameProps,
+    RenderQueueTrafficEncryptionProps,
     Repository,
+    SEPConfigurationSetup,
+    SEPSpotFleet,
     Stage,
-    ThinkboxDockerRecipes,
+    ThinkboxDockerRecipes
+)
+from aws_rfdk import (
+    DistinguishedName,
+    X509CertificatePem
 )
 
 
@@ -34,6 +56,10 @@ class SEPStackProps(StackProps):
     """
     # The path to the directory where the staged Deadline Docker recipes are.
     docker_recipes_stage_path: str
+    # The IMachineImage to use for Workers (needs Deadline Client installed).
+    worker_machine_image: IMachineImage
+    # The name of the EC2 keypair to associate with Worker nodes.
+    key_pair_name: Optional[str]
 
 
 class SEPStack(Stack):
@@ -73,46 +99,82 @@ def __init__(self, scope: Construct, stack_id: str, *, props: SEPStackProps, **k
             repository_installation_timeout=Duration.minutes(20)
         )
 
+        # The following code is used to demonstrate how to use the SEPConfigurationSetup if TLS is enabled.
+        host = 'renderqueue'
+        suffix = '.local'
+        # We are calculating the max length we can add to the common name to keep it under the maximum allowed 64
+        # characters and then taking a slice of the stack name so we don't get an error when creating the certificate
+        # with openssl
+        max_length = 64 - len(host) - len('.') - len(suffix) - 1
+        zone_name = Stack.of(self).stack_name[:max_length] + suffix
+
+        # Internal DNS zone for the VPC.
+        dns_zone = PrivateHostedZone(
+            self,
+            'DnsZone',
+            vpc=vpc,
+            zone_name=zone_name
+        )
+
+        # NOTE: this certificate is used by SEPConfigurationSetup construct below.
+        ca_cert = X509CertificatePem(
+            self,
+            'CaCert' + '_SEPConfigurationTest',
+            subject=DistinguishedName(
+                cn='ca.renderfarm' + suffix
+            )
+        )
+
+        server_cert = X509CertificatePem(
+            self,
+            'RenderQueueCertPEM' + '_SEPConfigurationTest',
+            subject=DistinguishedName(
+                cn=host + '.' + zone_name
+            ),
+            signing_certificate=ca_cert
+        )
+
         render_queue = RenderQueue(
             self,
             'RenderQueue',
-            vpc=props.vpc,
+            vpc=vpc,
             version=recipes.version,
             images=recipes.render_queue_images,
             repository=repository,
             # TODO - Evaluate deletion protection for your own needs. This is set to false to
             # cleanly remove everything when this stack is destroyed. If you would like to ensure
             # that this resource is not accidentally deleted, you should set this to true.
-            deletion_protection=False
+            deletion_protection=False,
+            hostname=RenderQueueHostNameProps(
+                hostname=host,
+                zone=dns_zone
+            ),
+            traffic_encryption=RenderQueueTrafficEncryptionProps(
+                external_tls=RenderQueueExternalTLSProps(
+                    rfdk_certificate=server_cert
+                ),
+                internal_protocol=ApplicationProtocol.HTTPS
+            )
+        )
+
+        # Create the IAM Role for the spot fleet.
+        # Note if you already have a worker IAM role in your account you can use it instead.
+        fleet_role = Role(
+            self,
+            'FleetRole',
+            assumed_by=ServicePrincipal('spotfleet.amazonaws.com'),
+            managed_policies=[ManagedPolicy.from_managed_policy_arn(
+                self,
+                'AmazonEC2SpotFleetTaggingRole',
+                'arn:aws:iam::aws:policy/service-role/AmazonEC2SpotFleetTaggingRole'
+            )]
         )
         # Adds the following IAM managed Policies to the Render Queue so it has the necessary permissions
         # to run the Spot Event Plugin and launch a Resource Tracker:
         # * AWSThinkboxDeadlineSpotEventPluginAdminPolicy
         # * AWSThinkboxDeadlineResourceTrackerAdminPolicy
-        render_queue.add_sep_policies()
-
-        # Create the security group that you will assign to your workers
-        worker_security_group = SecurityGroup(
-            self,
-            'SpotSecurityGroup', 
-            vpc=props.vpc,
-            allow_all_outbound=True,
-            security_group_name='DeadlineSpotSecurityGroup',
-        )
-        worker_security_group.connections.allow_to_default_port(
-            render_queue.endpoint
-        )
-
-        # Create the IAM Role for the Spot Event Plugins workers.
-        # Note: This Role MUST have a roleName that begins with "DeadlineSpot"
-        # Note: If you already have a worker IAM role in your account you can remove this code.
-        worker_iam_role = Role(
-            self,
-            'SpotWorkerRole',
-            assumed_by=ServicePrincipal('ec2.amazonaws.com'),
-            managed_policies= [ManagedPolicy.from_aws_managed_policy_name('AWSThinkboxDeadlineSpotEventPluginWorkerPolicy')],
-            role_name= 'DeadlineSpotWorkerRole',
-        )
+        # Also, adds policies that allow the Render Queue to tag spot fleet requests and to pass the spot fleet role.
+        render_queue.add_sep_policies(True, [fleet_role.role_arn])
 
         # Creates the Resource Tracker Access role.  This role is required to exist in your account so the resource tracker will work properly
         # Note: If you already have a Resource Tracker IAM role in your account you can remove this code.
@@ -121,6 +183,32 @@ def __init__(self, scope: Construct, stack_id: str, *, props: SEPStackProps, **k
             'ResourceTrackerRole',
             assumed_by=ServicePrincipal('lambda.amazonaws.com'),
             managed_policies= [ManagedPolicy.from_aws_managed_policy_name('AWSThinkboxDeadlineResourceTrackerAccessPolicy')],
-            role_name= 'DeadlineResourceTrackerAccessRole',
+            role_name= 'DeadlineResourceTrackerAccessRole'
+        )
+
+        fleet = SEPSpotFleet(
+            self,
+            'SEPSpotFleet',
+            vpc=vpc,
+            render_queue=render_queue,
+            fleet_role=fleet_role,
+            deadline_groups=['group_name'],
+            instance_types=[InstanceType.of(InstanceClass.BURSTABLE3, InstanceSize.LARGE)],
+            worker_machine_image=props.worker_machine_image,
+            target_capacity=1,
+            key_name=props.key_pair_name
+        )
+
+        SEPConfigurationSetup(
+            self,
+            'SEPConfigurationSetup',
+            vpc=vpc,
+            render_queue=render_queue,
+            ca_cert=ca_cert.cert,
+            spot_fleet_options=SEPConfigurationProperties(
+                spot_fleets=[fleet],
+                enable_resource_tracker=True,
+                region=self.region
+            )
         )