feat(examples): added ec2 image builder example (#292)

examples/deadline/EC2-Image-Builder/README.md

@@ -0,0 +1,58 @@
+# RFDK Sample Application - EC2 Image Builder
+
+Keeping software updated on custom AMIs can be a pain if you're creating them manually. The EC2 Image Builder service is an option for automating this process. The CDK constructs for Image Builder can be worked into an RFDK app to build the required AMIs for the worker fleet on the fly.
+
+This is a sample RFDK application that utilizes [EC2 Image Builder](https://docs.aws.amazon.com/imagebuilder/latest/userguide/what-is-image-builder.html) to install and configure the Deadline client onto an AMI that can then be used as the image for a worker fleet. The process of creating an AMI can be lengthy, sometimes extending the deployment of a render farm by 30 minutes, but if you have multiple custom AMI's that you maintain, you may find it easier to let your RFDK application install new versions of the Deadline client onto them, rather than having to do it manually.
+
+Another option is using EC2 Image Builder from the AWS Console to create an image pipeline. We don't touch on that here, but you may want to see the [Create an image pipeline user guide](https://docs.aws.amazon.com/imagebuilder/latest/userguide/start-build-image-pipeline.html) and see if this option would be a better fit for you.
+
+---
+
+_**Note:** This application is an illustrative example to showcase some of the capabilities of the RFDK. **It is not intended to be used for production render farms**, which should be built with more consideration of the security and operational needs of the system._
+
+---
+
+## Architecture
+
+This sample application uses an [EC2 ImageBuilder Image](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-imagebuilder-image.html) to create an AMI in the DeadlineMachineImage stack that is used by the WorkerInstanceFleet in the same stack.
+
+### BaseFarmStack
+
+The contents of the BaseFarmStack include all the required components for a render farm, without any worker nodes. To learn more about these components, refer to the [All-In-AWS-Infrastructure-Basic](../All-In-AWS-Infrastructure-Basic/README.md) example.
+
+### ComputeStack
+
+The ComputeStack holds the images being created as well as the worker fleets the images get used by. It's important to keep the image and worker fleet in the same stack if you want to be able to do updates to the image. If you were to move the worker fleet into a dependent stack, the update wouldn't be able to be performed while it was currently in use in a different stack. In this case, the worker fleet would need to be taken down first.
+
+### DeadlineMachineImage
+
+This construct creates all the infrastructure required by Image Builder to install Deadline onto an existing AMI and then create a WorkerInstanceFleet that uses it. The configuration for the image creation is as simple as providing the Deadline version, a parent AMI, the OS of the parent AMI, and a version number for your AMI. The parent AMI can be supplied by anything that supplied an `IMachineImage`, such as `MachineImage.genericLinux()` or `MachineImage.lookup()`. An image version also needs to be supplied. One strategy to use with versioning your image is to start with version `1.0.0` and bump the version if you need to change any of the input parameters, such as changing the parent AMI or Deadline version. Images for different OSes can be versioned separately.
+
+CDK does not have L2 constructs for Image Builder yet, so we are using the L1 constructs. An L1 construct is generated directly from the CloudFormation definition of the service, so referring to the [AWS CloudFormation user guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_ImageBuilder.html) will provide more details. You can read more about L1 vs L2 constructs in the [CDK Constructs user guide](https://docs.aws.amazon.com/cdk/latest/guide/constructs.html#constructs_lib).
+
+#### Updates
+
+If you would like to upgrade the version of Deadline your worker fleet is using, you will need to bump the image version to a value such as `1.0.1` along with changing the Deadline version. Since the worker fleet is backed by an auto scaling group, the new image will get built and the auto scaling group's launch configuration will get updated; however, this doesn't replace existing instances, it will only affect new ones that get deployed. To have current workers get replaced with the new version, you have a few options:
+
+1. Set your `desiredCapacity` and `minCapacity` on the worker fleet to `0` before you perform the redeployment that will create your new AMI, and then do a follow up deployment with these fields set to their previous values (or removed).
+1. Before performing your redeployment that will generate the new AMI, manually terminate the worker instances from the console. This will cause new ones to get started during the redeployment.
+
+#### AMI Storage
+
+When performing an update to, or deletion of, the DeadlineMachineImage construct, any AMI that was created by a previous deployment of the construct will not be deleted. They are still available in EC2 and can be seen under `Images > AMIs` in the EC2 console or in the `My AMIs` section of the Launch instance wizard. You can continue to use them like any other AMI, or deregister them if you no longer require them. The cost of storing these AMIs depends on the size of the disk you took a snapshot of to create them, for EBS-backed AMIs you can find snapshot costs on their [EBS pricing page](https://aws.amazon.com/ebs/pricing/). For S3-backed AMI's, you'll pay for the storage fees of the data that needs to be stored based on [S3 pricing](https://aws.amazon.com/s3/pricing/), whether you have an instance running or not.
+
+## EC2 Image Builder Pipeline
+
+An alternative to creating single images during the deployment of your render farm is to create an [EC2 Image Builder pipeline](https://docs.aws.amazon.com/imagebuilder/latest/userguide/start-build-image-pipeline.html) that would run on a specified schedule. You can choose to either only create an image if any components have new versions released since the last run, or create a new image regardless.
+
+At this point in time, we do not provide Amazon-managed Deadline components for you to consume, so if you do choose to go this route, you would need to create a new version of your Deadline component for each release of Deadline, but you may decide this is worthwhile, depending on how many image variations you use for your workers. Due to the Image Builder pipeline still being an L1 construct in CDK, the AWS Console is recommended for building your pipeline.
+
+Once you have an image created from a pipeline, it can be consumed in your RFDK application using a [LookupMachineImage](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.LookupMachineImage.html). The result of this lookup will always pick the most recent image if there are multiple matches, so you can always be sure that when your pipeline creates a newer image, your RFDK app deployment will pick it up. Note that the lookup stores the selected image in the CDK context, so if you would like to try and pick up a new image on an update deployment, you'll have to [clear the context](https://docs.aws.amazon.com/cdk/latest/guide/context.html#context_viewing) to get it.
+
+## Typescript
+
+[Continue to Typescript specific documentation.](ts/README.md)
+
+## Python
+
+[Continue to Python specific documentation.](python/README.md)

examples/deadline/EC2-Image-Builder/components/.gitignore

@@ -0,0 +1 @@
+*.component

examples/deadline/EC2-Image-Builder/components/deadline-linux.component.template

@@ -0,0 +1,45 @@
+name: Deadline-${version}
+schemaVersion: 1.0
+
+phases:
+  -
+    name: 'build'
+    steps:
+      -
+        name: DownloadDeadlineClient
+        action: S3Download
+        timeoutSeconds: 120
+        onFailure: Abort
+        maxAttempts: 3
+        inputs:
+          -
+            source: '${s3uri}'
+            destination: '/tmp/DeadlineClient-${version}-linux-x64-installer.run'
+      -
+        name: InstallDeadline
+        action: ExecuteBash
+        timeoutSeconds: 600
+        onFailure: Abort
+        maxAttempts: 1
+        inputs:
+          commands:
+            - "chmod +x {{ build.DownloadDeadlineClient.inputs[0].destination }}"
+            - "{{ build.DownloadDeadlineClient.inputs[0].destination }} --mode unattended \
+                  --connectiontype Remote \
+                  --noguimode true \
+                  --slavestartup false \
+                  --launcherdaemon true \
+                  --restartstalled true \
+                  --autoupdateoverride false"
+      -
+        name: Delete
+        action: ExecuteBash
+        timeoutSeconds: 120
+        onFailure: Continue
+        maxAttempts: 3
+        inputs:
+          commands:
+            - "rm {{ build.DownloadDeadlineClient.inputs[0].destination }}"
+            - "rm /var/log/Thinkbox/Deadline10/deadlineslave*.log"
+            - "rm /var/log/Thinkbox/Deadline10/deadlinelauncher*.log"
+            - "rm /var/log/cloud-init-output.log"

examples/deadline/EC2-Image-Builder/components/deadline-windows.component.template

@@ -0,0 +1,62 @@
+name: Deadline-${version}
+schemaVersion: 1.0
+
+phases:
+  -
+    name: 'build'
+    steps:
+      -
+        name: DownloadDeadlineClient
+        action: S3Download
+        timeoutSeconds: 120
+        onFailure: Abort
+        maxAttempts: 3
+        inputs:
+          -
+            source: '${s3uri}'
+            destination: 'C:\DeadlineClient-${version}-windows-installer.exe'
+      -
+        name: InstallDeadline
+        action: ExecutePowerShell
+        timeoutSeconds: 600
+        onFailure: Abort
+        maxAttempts: 1
+        inputs:
+          commands:
+            - '$argList = "--mode unattended --licensemode UsageBased  --connectiontype Remote --noguimode true --slavestartup false --restartstalled true --launcherservice true --serviceuser `"NT AUTHORITY\NetworkService`" --autoupdateoverride false"'
+            - 'Start-Process -FilePath {{ build.DownloadDeadlineClient.inputs[0].destination }} -ArgumentList $argList -Wait'
+      -
+        name: ConfigureAdmin
+        action: ExecutePowerShell
+        timeoutSeconds: 600
+        onFailure: Abort
+        maxAttempts: 1
+        inputs:
+          commands:
+            - 'if (-not (net localgroup administrators | Select-String "^NT AUTHORITY\\NETWORK SERVICE$" -Quiet)) {
+                net localgroup administrators /add "NT AUTHORITY\NETWORK SERVICE"
+              }'
+      -
+        name: ConfigureDeadlineLaunch
+        action: ExecutePowerShell
+        timeoutSeconds: 600
+        onFailure: Abort
+        maxAttempts: 1
+        inputs:
+          commands:
+            - 'New-ItemProperty -Path "HKLM:\System\CurrentControlSet\Control" -Name "ServicesPipeTimeout" -Value 60000 -PropertyType DWORD -Force | Out-Null'
+            - '(Get-Content C:\ProgramData\Amazon\EC2-Windows\Launch\Config\LaunchConfig.json).replace("`"setComputerName`": false", "`"setComputerName`": true") | Set-Content C:\ProgramData\Amazon\EC2-Windows\Launch\Config\LaunchConfig.json'
+            - 'C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule'
+            - 'Stop-Service -Name "deadline10launcherservice"'
+      -
+        name: Delete
+        action: ExecutePowerShell
+        timeoutSeconds: 120
+        onFailure: Continue
+        maxAttempts: 3
+        inputs:
+          commands:
+            - 'Remove-Item -Path "{{ build.DownloadDeadlineClient.inputs[0].destination }}" -Force'
+            - 'Remove-Item -Path "C:\ProgramData\Thinkbox\Deadline10\logs\deadlinelauncher*.log" -Force'
+            - 'Remove-Item -Path "C:\ProgramData\Thinkbox\Deadline10\logs\deadlineslave*.log" -Force'
+            - 'Remove-Item -Path "C:\ProgramData\Amazon\EC2-Windows\Launch\Log\UserdataExecution*.log" -Force'

examples/deadline/EC2-Image-Builder/python/.gitignore

@@ -0,0 +1,17 @@
+*.swp
+package-lock.json
+__pycache__
+.pytest_cache
+.env
+*.egg-info
+*.pyc
+Pipfile
+Pipfile.lock
+pyproject.toml
+build/
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out/
+cdk.context.json
+stage/

examples/deadline/EC2-Image-Builder/python/README.md

@@ -0,0 +1,58 @@
+# RFDK Sample Application - EC2 Image Builder - Python
+
+## Overview
+[Back to overview](../README.md)
+
+## Instructions
+
+---
+**NOTE**
+
+These instructions assume that your working directory is `examples/deadline/EC2-Image-Builder/python/` relative to the root of the AWS-RFDK package.
+
+---
+
+1.  This sample app on the `mainline` branch may contain features that have not yet been officially released, and may not be available in the `aws-rfdk` package installed through pip from PyPI. To work from an example of the latest release, please switch to the `release` branch. If you would like to try out unreleased features, you can stay on `mainline` and follow the instructions for building, packing, and installing the `aws-rfdk` from your local repository.
+
+2.  Install the dependencies of the sample app:
+
+    ```bash
+    pip install -r requirements.txt
+    ```
+
+3.  If working on the `release` branch, this step can be skipped. If working on `mainline`, navigate to the base directory where the build and packaging scripts are, then run them and install the result over top of the `aws-rfdk` version that was installed in the previous step:
+    ```bash
+    # Navigate to the root directory of the RFDK repository
+    pushd ../../../..
+    # Enter the Docker container to run the build and pack scripts
+    ./scripts/rfdk_build_environment.sh
+    ./build.sh
+    ./pack.sh
+    # Exit the Docker container
+    exit
+    # Navigate back to the example directory
+    popd
+    pip install ../../../../dist/python/aws-rfdk-<version>.tar.gz
+    ```
+
+4.  You must read and accept the [AWS Thinkbox End-User License Agreement (EULA)](https://www.awsthinkbox.com/end-user-license-agreement) to deploy and run Deadline. To do so, change the value of the `accept_aws_thinkbox_eula` in `package/config.py` like this:
+
+    ```py
+    self.accept_aws_thinkbox_eula: AwsThinkboxEulaAcceptance = AwsThinkboxEulaAcceptance.USER_ACCEPTS_AWS_THINKBOX_EULA
+    ```
+
+5. Change the value of the `deadline_version` variable in `package/config.py` to specify the desired version of Deadline to be deployed to your render farm. RFDK is compatible with Deadline versions 10.1.9.x and later. To see the available versions of Deadline, consult the [Deadline release notes](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/release-notes.html). It is recommended to use the latest version of Deadline available when building your farm, but to pin this version when the farm is ready for production use. For example, to pin to the `10.1.13` release of Deadline, use `10.1.13.2`.
+
+6. Also in `package/config.py`, you can set the version of your image recipe that you'll create by changing the value of `image_recipe_version`. The default value here should be fine to start. The image recipe version would only need to be changed if you're changing any inputs for the image creation that will cause a new image to be made.
+
+7. Deploy all the stacks in the sample app:
+
+    ```bash
+    cdk deploy "*"
+    ```
+
+8. Once you are finished with the sample app, you can tear it down by running:
+
+    ```bash
+    cdk destroy "*"
+    ```

examples/deadline/EC2-Image-Builder/python/cdk.json

@@ -0,0 +1,3 @@
+{
+  "app": "python -m package.app"
+}

examples/deadline/EC2-Image-Builder/python/package/app.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+
+from aws_cdk.core import (
+    App,
+    Environment
+)
+
+from .config import config
+from .lib import (
+    base_farm_stack,
+    compute_stack
+)
+
+def main():
+    app = App()
+
+    if 'CDK_DEPLOY_ACCOUNT' not in os.environ and 'CDK_DEFAULT_ACCOUNT' not in os.environ:
+        raise ValueError('You must define either CDK_DEPLOY_ACCOUNT or CDK_DEFAULT_ACCOUNT in the environment.')
+    if 'CDK_DEPLOY_REGION' not in os.environ and 'CDK_DEFAULT_REGION' not in os.environ:
+        raise ValueError('You must define either CDK_DEPLOY_REGION or CDK_DEFAULT_REGION in the environment.')
+    env = Environment(
+        account=os.environ.get('CDK_DEPLOY_ACCOUNT', os.environ.get('CDK_DEFAULT_ACCOUNT')),
+        region=os.environ.get('CDK_DEPLOY_REGION', os.environ.get('CDK_DEFAULT_REGION'))
+    )
+
+    farm_props = base_farm_stack.BaseFarmStackProps(
+        deadline_version=config.deadline_version,
+        accept_aws_thinkbox_eula=config.accept_aws_thinkbox_eula
+    )
+    farm_stack = base_farm_stack.BaseFarmStack(app, 'BaseFarmStack', props=farm_props, env=env)
+
+    compute_stack_props = compute_stack.ComputeStackProps(
+        deadline_version=config.deadline_version,
+        image_recipe_version=config.image_recipe_version,
+        render_queue=farm_stack.render_queue,
+        vpc=farm_stack.vpc
+    )
+    compute_stack.ComputeStack(app, 'ComputeStack', props=compute_stack_props, env=env)
+
+    app.synth()
+
+
+if __name__ == '__main__':
+    main()

examples/deadline/EC2-Image-Builder/python/package/config.py

@@ -0,0 +1,28 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from aws_rfdk.deadline import AwsThinkboxEulaAcceptance
+
+class AppConfig:
+    """
+    Configuration values for the sample app.
+
+    TODO: Fill these in with your own values.
+    """
+    def __init__(self):
+        # Change this value to AwsThinkboxEulaAcceptance.USER_ACCEPTS_AWS_THINKBOX_EULA if you wish to accept the EULA for
+        # Deadline and proceed with Deadline deployment. Users must explicitly accept the AWS Thinkbox EULA before using the
+        # AWS Thinkbox Deadline container images.
+        #
+        # See https://www.awsthinkbox.com/end-user-license-agreement for the terms of the agreement.
+        self.accept_aws_thinkbox_eula: AwsThinkboxEulaAcceptance = AwsThinkboxEulaAcceptance.USER_REJECTS_AWS_THINKBOX_EULA
+
+        # The version of Deadline to install on the AMI. This can be either a partial version that will use the latest patch, such as
+        # '10.1' or '10.1.13', or a full version that will be pinned to a specific patch release, such as '10.1.13.1'.
+        self.deadline_version: str = '10.1'
+
+        # This version is used for the version of the Deadline component and the image recipe in the DeadlineMachineImage construct.
+        # It must be bumped manually whenever changes are made to the recipe.
+        self.image_recipe_version: str = '1.0.0'
+
+config: AppConfig = AppConfig()