diff --git a/examples/deadline/Local-Zone/README.md b/examples/deadline/Local-Zone/README.md new file mode 100644 index 000000000..b55a5cd63 --- /dev/null +++ b/examples/deadline/Local-Zone/README.md @@ -0,0 +1,41 @@ +# RFDK Sample Application - Local Zones + +If you have large asset files that your Worker instances need to access from your on-prem infrastructure, deploying the Workers to a geographically close AWS Local Zone can reduce the latency and increase the speed of your renders. This example will walk you through setting up your workers in a local zone while leaving the rest of the render farm in standard availability zones. Currently Amazon has launched a local zone in Los Angeles that is a part of the us-west-2 region, but they have more on the way. For more information on where local zones are available, how to get access, and what services they provide, refer to the [AWS Local Zones about page](https://aws.amazon.com/about-aws/global-infrastructure/localzones/). + +Before deploying your farm, you may want to read our [Connecting to the Render Farm](https://docs.aws.amazon.com/rfdk/latest/guide/connecting-to-render-farm.html#connecting-with-site-to-site-vpn) developer guide for guidance on how to create a connection from your local network to the farm using something like a VPN. All of the techniques listed in the guide require changes to the networking tier of your RFDK app to allow the connection. After your connection is set up, you will be able to configure your network file server to be available on your workers, so any local assets you have can be transferred as needed by the jobs they perform. + +--- + +_**Note:** This application is an illustrative example to showcase some of the capabilities of the RFDK. **It is not intended to be used for production render farms**, which should be built with more consideration of the security and operational needs of the system._ + +--- + +## Architecture + +This example app assumes you're familiar with the general architecture of an RFDK render farm. If not, please refer to the [All-In-AWS-Infrastructure-Basic](../All-In-AWS-Infrastructure-Basic/README.md) example for the basics. + +### Components + +#### Network Tier + +The network tier sets up a [VPC](https://aws.amazon.com/vpc/) that spans across all of the standard availability zones and local zones that are used, but the NAT Gateway for the VPC is only added to the standard zones, as it is not available in any local zones at this time. In this tier we override the Stack's `availabilityZones()` method, which returns the list of availability zones the Stack can use. It's by this mechanism that we control which zones the VPC will be deployed to. + +#### Security Tier + +This holds the root CA certificate used for signing any certificates required by the farm, such as the one used by the render queue. + +#### Service Tier + +The service tier contains the repository and render queue, both of which are provided the selection of standard availability zone subnets to be deployed into. The DocumentDB and EFS filesystem are not available in the local zones at this time, so the repository cannot be moved there. Since the repository needs to be in a standard availability zone, there isn't any benefit to moving the render queue to a local zone. + +#### Compute Tier + +This tier holds the worker fleet and its health monitor. The health monitor contains a [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) used to perform application-level health checks and the worker fleet contains an [Auto Scaling Group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html). Currently, these services are available in all launched local zones, so the construct can be placed in those zones. + +## Typescript + +[Continue to Typescript specific documentation.](ts/README.md) + +## Python + +[Continue to Python specific documentation.](python/README.md) \ No newline at end of file diff --git a/examples/deadline/Local-Zone/python/.gitignore b/examples/deadline/Local-Zone/python/.gitignore new file mode 100644 index 000000000..769022221 --- /dev/null +++ b/examples/deadline/Local-Zone/python/.gitignore @@ -0,0 +1,14 @@ +*.swp +package-lock.json +__pycache__ +.pytest_cache +.env +*.egg-info +venv +build + +# CDK asset staging directory +.cdk.staging +cdk.out +cdk.context.json +stage diff --git a/examples/deadline/Local-Zone/python/README.md b/examples/deadline/Local-Zone/python/README.md new file mode 100644 index 000000000..c5a72831b --- /dev/null +++ b/examples/deadline/Local-Zone/python/README.md @@ -0,0 +1,84 @@ +# RFDK Sample Application - Local Zones - Python + +## Overview +[Back to overview](../README.md) + +## Instructions + +--- +**NOTE** + +These instructions assume that your working directory is `examples/deadline/Local-Zones/python/` relative to the root of the AWS-RFDK package. + +--- + +1. This sample app on the `mainline` branch may contain features that have not yet been officially released, and may not be available in the `aws-rfdk` package installed through pip from PyPI. To work from an example of the latest release, please switch to the `release` branch. If you would like to try out unreleased features, you can stay on `mainline` and follow the instructions for building, packing, and installing the `aws-rfdk` from your local repository. + +2. Install the dependencies of the sample app: + + ```bash + pip install -r requirements.txt + ``` + +3. If working on the `release` branch, this step can be skipped. If working on `mainline`, navigate to the base directory where the build and packaging scripts are, then run them and install the result over top of the `aws-rfdk` version that was installed in the previous step: + ```bash + # Navigate to the root directory of the RFDK repository + pushd ../../../.. + # Enter the Docker container to run the build and pack scripts + ./scripts/rfdk_build_environment.sh + ./build.sh + ./pack.sh + # Exit the Docker container + exit + # Navigate back to the example directory + popd + pip install ../../../../dist/python/aws-rfdk-.tar.gz + ``` + +4. You must read and accept the [AWS Thinkbox End-User License Agreement (EULA)](https://www.awsthinkbox.com/end-user-license-agreement) to deploy and run Deadline. To do so, change the value of the `accept_aws_thinkbox_eula` in `package/lib/config.py` like this: + + ```py + self.accept_aws_thinkbox_eula: AwsThinkboxEulaAcceptance = AwsThinkboxEulaAcceptance.USER_ACCEPTS_AWS_THINKBOX_EULA + ``` + +5. Change the value of the `deadline_version` variable in `package/config.py` to specify the desired version of Deadline to be deployed to your render farm. RFDK is compatible with Deadline versions 10.1.9.x and later. To see the available versions of Deadline, consult the [Deadline release notes](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/release-notes.html). It is recommended to use the latest version of Deadline available when building your farm, but to pin this version when the farm is ready for production use. For example, to pin to the latest `10.1.15.x` release of Deadline, use: + + ```python + self.deadline_version: str = '10.1.15' + ``` + +6. Change the value of the `deadline_client_linux_ami_map` variable in `package/config.py` to include the region + AMI ID mapping of your EC2 AMI(s) with Deadline Worker. You can use the following AWS CLI command to look up AMIs, replacing the `` and `` to match the AWS region and Deadline version you're looking for: + + ```bash + aws --region ec2 describe-images --owners 357466774442 --filters "Name=name,Values=*Worker*" "Name=name,Values=**" --query 'Images[*].[ImageId, Name]' --output text + ``` + +7. Also in `package/lib/config.py`, you can set the `availability_zones_standard` and `availability_zones_local` values to the availability zones you want to use. These values must all be from the same region. It's required that you use at least two standard zones, but you can use more if you'd like. For the local zones, you can use one or more. + +8. To gain the benefits of putting your workers in a local zone close to your asset server, you are going to want to set up a connection from your local network to the one you're creating in AWS. + 1. You should start by reading through the [Connecting to the Render Farm](https://docs.aws.amazon.com/rfdk/latest/guide/connecting-to-render-farm.html) documentation and implementing one of the methods for connecting your network to your AWS VPC described there. + 2. With whichever option you choose, you'll want to make sure you are propagating the worker subnets to your local network. All the options in the document show how to propagate all the private subnets, which will include the ones used by the workers. + 3. Ensure your worker fleet's security group allows traffic from your network on the correct ports that your NFS requires to be open. The documentation shows how to [allow connections to the Render Queue](https://docs.aws.amazon.com/rfdk/latest/guide/connecting-to-render-farm.html#allowing-connection-to-the-render-queue), which you may also want to enable if you plan on connecting any of your local machines to your render farm, but you would also want to do something similar for the worker fleet, for example, ports `22` and `2049` are commonly required for NFS, so this code could be added to the `ComputeTier`: + + ```python + # The customer-prefix-cidr-range needs to be replaced by the CIDR range for your local network that you used when configuring the VPC connection + self.worker_fleet.connections.allow_from(Peer.ipv4('customer-prefix-cidr-range'), Port.tcp(22)) + self.worker_fleet.connections.allow_from(Peer.ipv4('customer-prefix-cidr-range'), Port.udp(22)) + self.worker_fleet.connections.allow_from(Peer.ipv4('customer-prefix-cidr-range'), Port.tcp(2049)) + self.worker_fleet.connections.allow_from(Peer.ipv4('customer-prefix-cidr-range'), Port.tcp(2049)) + ``` + + 4. Add user-data to mount the NFS on the compute tier. This can be provided in the `UserDataProvider` in the `ComputeTier`. + 5. (optional) Set up [path mapping rules in Deadline](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/cross-platform.html). + +9. Deploy all the stacks in the sample app: + + ```bash + cdk deploy "*" + ``` + +10. Once you are finished with the sample app, you can tear it down by running: + + ```bash + cdk destroy "*" + ``` \ No newline at end of file diff --git a/examples/deadline/Local-Zone/python/cdk.json b/examples/deadline/Local-Zone/python/cdk.json new file mode 100644 index 000000000..ee4a9b01f --- /dev/null +++ b/examples/deadline/Local-Zone/python/cdk.json @@ -0,0 +1,3 @@ +{ + "app": "python -m package.app" +} diff --git a/examples/deadline/Local-Zone/python/package/__init__.py b/examples/deadline/Local-Zone/python/package/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/examples/deadline/Local-Zone/python/package/app.py b/examples/deadline/Local-Zone/python/package/app.py new file mode 100644 index 000000000..e89aeac53 --- /dev/null +++ b/examples/deadline/Local-Zone/python/package/app.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 + +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +import os + +from aws_cdk.core import ( + App, + Environment +) +from aws_cdk.aws_ec2 import ( + MachineImage +) + +from .lib import ( + config, + network_tier, + security_tier, + service_tier, + compute_tier +) + + +def main(): + # ------------------------------ + # Validate Config Values + # ------------------------------ + if not config.config.key_pair_name: + print('EC2 key pair name not specified. You will not have SSH access to the render farm.') + + # ------------------------------ + # Application + # ------------------------------ + app = App() + + if 'CDK_DEPLOY_ACCOUNT' not in os.environ and 'CDK_DEFAULT_ACCOUNT' not in os.environ: + raise ValueError('You must define either CDK_DEPLOY_ACCOUNT or CDK_DEFAULT_ACCOUNT in the environment.') + if 'CDK_DEPLOY_REGION' not in os.environ and 'CDK_DEFAULT_REGION' not in os.environ: + raise ValueError('You must define either CDK_DEPLOY_REGION or CDK_DEFAULT_REGION in the environment.') + env = Environment( + account=os.environ.get('CDK_DEPLOY_ACCOUNT', os.environ.get('CDK_DEFAULT_ACCOUNT')), + region=os.environ.get('CDK_DEPLOY_REGION', os.environ.get('CDK_DEFAULT_REGION')) + ) + + # ------------------------------ + # Network Tier + # ------------------------------ + network = network_tier.NetworkTier( + app, + 'NetworkTier', + env=env + ) + + # ------------------------------ + # Security Tier + # ------------------------------ + security = security_tier.SecurityTier( + app, + 'SecurityTier', + env=env + ) + + # ------------------------------ + # Service Tier + # ------------------------------ + service_props = service_tier.ServiceTierProps( + vpc=network.vpc, + availability_zones=config.config.availability_zones_standard, + root_ca=security.root_ca, + dns_zone=network.dns_zone, + deadline_version=config.config.deadline_version, + accept_aws_thinkbox_eula=config.config.accept_aws_thinkbox_eula + ) + service = service_tier.ServiceTier(app, 'ServiceTier', props=service_props, env=env) + + # ------------------------------ + # Compute Tier + # ------------------------------ + deadline_client_image = MachineImage.generic_linux(config.config.deadline_client_linux_ami_map) + compute_props = compute_tier.ComputeTierProps( + vpc=network.vpc, + availability_zones=config.config.availability_zones_local, + render_queue=service.render_queue, + worker_machine_image=deadline_client_image, + key_pair_name=config.config.key_pair_name, + ) + _compute = compute_tier.ComputeTier(app, 'ComputeTier', props=compute_props, env=env) + + app.synth() + + +if __name__ == '__main__': + main() diff --git a/examples/deadline/Local-Zone/python/package/lib/__init__.py b/examples/deadline/Local-Zone/python/package/lib/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/examples/deadline/Local-Zone/python/package/lib/compute_tier.py b/examples/deadline/Local-Zone/python/package/lib/compute_tier.py new file mode 100644 index 000000000..f14eb71a7 --- /dev/null +++ b/examples/deadline/Local-Zone/python/package/lib/compute_tier.py @@ -0,0 +1,111 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +from dataclasses import dataclass +from typing import ( + List, + Optional +) + +from aws_cdk.core import ( + Construct, + Stack, + StackProps +) +from aws_cdk.aws_ec2 import ( + IMachineImage, + InstanceClass, + InstanceSize, + InstanceType, + IVpc, + SubnetSelection, + SubnetType +) + +from aws_rfdk import ( + HealthMonitor, + SessionManagerHelper +) +from aws_rfdk.deadline import ( + InstanceUserDataProvider, + IRenderQueue, + WorkerInstanceFleet +) + + +@dataclass +class ComputeTierProps(StackProps): + """ + Properties for ComputeTier + """ + # The VPC to deploy resources into. + vpc: IVpc + # The availability zones the worker instances will be deployed to. This can include your local + # zones, but they must belong to the same region as the standard zones used in other stacks in + # this application. + availability_zones: List[str] + # The IRenderQueue that Deadline Workers connect to. + render_queue: IRenderQueue + # The IMachineImage to use for Workers (needs Deadline Client installed). + worker_machine_image: IMachineImage + # The name of the EC2 keypair to associate with Worker nodes. + key_pair_name: Optional[str] + + +class UserDataProvider(InstanceUserDataProvider): + def __init__(self, scope: Construct, stack_id: str): + super().__init__(scope, stack_id) + + def pre_worker_configuration(self, host) -> None: + # Add code here for mounting your NFS to the workers + host.user_data.add_commands("echo preWorkerConfiguration") + + +class ComputeTier(Stack): + """ + The computer tier consists of the worker fleets. We'll be deploying the workers into the + local zone we're using. + """ + def __init__(self, scope: Construct, stack_id: str, *, props: ComputeTierProps, **kwargs): + """ + Initializes a new instance of ComputeTier + :param scope: The Scope of this construct. + :param stack_id: The ID of this construct. + :param props: The properties of this construct. + :param kwargs: Any kwargs that need to be passed on to the parent class. + """ + super().__init__(scope, stack_id, **kwargs) + + # We can put the health monitor and worker fleet in all of the local zones we're using + subnets = SubnetSelection( + availability_zones=props.availability_zones, + subnet_type=SubnetType.PRIVATE, + one_per_az=True + ) + + # We can put the health monitor in all of the local zones we're using for the worker fleet + self.health_monitor = HealthMonitor( + self, + 'HealthMonitor', + vpc=props.vpc, + vpc_subnets=subnets, + deletion_protection=False + ) + + self.worker_fleet = WorkerInstanceFleet( + self, + 'WorkerFleet', + vpc=props.vpc, + vpc_subnets=subnets, + render_queue=props.render_queue, + # Not all instance types will be available in local zones. For a list of the instance types + # available in each local zone, you can refer to: + # https://aws.amazon.com/about-aws/global-infrastructure/localzones/features/#AWS_Services + # BURSTABLE3 is a T3; the third generation of burstable instances + instance_type=InstanceType.of(InstanceClass.BURSTABLE3, InstanceSize.LARGE), + worker_machine_image=props.worker_machine_image, + health_monitor=self.health_monitor, + key_name=props.key_pair_name, + user_data_provider=UserDataProvider(self, 'UserDataProvider') + ) + SessionManagerHelper.grant_permissions_to(self.worker_fleet) diff --git a/examples/deadline/Local-Zone/python/package/lib/config.py b/examples/deadline/Local-Zone/python/package/lib/config.py new file mode 100644 index 000000000..2fa57172a --- /dev/null +++ b/examples/deadline/Local-Zone/python/package/lib/config.py @@ -0,0 +1,51 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +from typing import ( + List, + Mapping, + Optional +) + +from aws_rfdk.deadline import ( + AwsThinkboxEulaAcceptance, +) + + +class AppConfig: + """ + Configuration values for the sample app. + + TODO: Fill these in with your own values. + """ + def __init__(self): + # Change this value to AwsThinkboxEulaAcceptance.USER_ACCEPTS_AWS_THINKBOX_EULA if you wish to accept the EULA + # for Deadline and proceed with Deadline deployment. Users must explicitly accept the AWS Thinkbox EULA before + # using the AWS Thinkbox Deadline container images. + # + # See https://www.awsthinkbox.com/end-user-license-agreement for the terms of the agreement. + self.accept_aws_thinkbox_eula: AwsThinkboxEulaAcceptance = AwsThinkboxEulaAcceptance.USER_REJECTS_AWS_THINKBOX_EULA + + # The standard availability zones that the render farm will deploy into. It is recommended to use at least + # two and they must be from the same region. The default values being provided are two of the four standard + # zones in us-west-2, located in Oregon. + self.availability_zones_standard: List[str] = ['us-west-2a', 'us-west-2b'] + + # The local availability zones that will hold the worker fleet. They must belong to the same region as the standard + # zones. The default value being provided here is one of the two local zones in us-west-2, located in Los Angeles. + self.availability_zones_local: List[str] = ['us-west-2-lax-1a'] + + # The version of Deadline to use on the render farm. Leave as None for the latest release or specify a version + # to pin to. Some examples of pinned version values are "10", "10.1", or "10.1.16" + self.deadline_version: Optional[str] = '10.1.16' + + # A map of regions to Deadline Client Linux AMIs. As an example, the Linux Deadline 10.1.16.8 AMI ID + # from us-west-2 is filled in. It can be used as-is, added to, or replaced. Ideally the version here + # should match the one used for staging the render queue and usage based licensing recipes. + self.deadline_client_linux_ami_map: Mapping[str, str] = {'us-west-2': 'ami-0aa3610842bc3534d'} + + # (Optional) The name of the EC2 keypair to associate with the instances. + self.key_pair_name: Optional[str] = None + + +config: AppConfig = AppConfig() diff --git a/examples/deadline/Local-Zone/python/package/lib/network_tier.py b/examples/deadline/Local-Zone/python/package/lib/network_tier.py new file mode 100644 index 000000000..504e6551b --- /dev/null +++ b/examples/deadline/Local-Zone/python/package/lib/network_tier.py @@ -0,0 +1,122 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +import builtins +import typing + +from aws_cdk.aws_ec2 import ( + GatewayVpcEndpointAwsService, + InterfaceVpcEndpointAwsService, + SubnetConfiguration, + SubnetSelection, + SubnetType, + Vpc +) +from aws_cdk.aws_route53 import ( + PrivateHostedZone +) +from aws_cdk.core import ( + Construct, + Stack +) +import jsii + +from .config import config + +_INTERFACE_ENDPOINT_SERVICES = [ + {'name': 'CLOUDWATCH', 'service': InterfaceVpcEndpointAwsService.CLOUDWATCH}, + {'name': 'CLOUDWATCH_EVENTS', 'service': InterfaceVpcEndpointAwsService.CLOUDWATCH_EVENTS}, + {'name': 'CLOUDWATCH_LOGS', 'service': InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS}, + {'name': 'EC2', 'service': InterfaceVpcEndpointAwsService.EC2}, + {'name': 'ECR', 'service': InterfaceVpcEndpointAwsService.ECR}, + {'name': 'ECS', 'service': InterfaceVpcEndpointAwsService.ECS}, + {'name': 'KMS', 'service': InterfaceVpcEndpointAwsService.KMS}, + {'name': 'SECRETS_MANAGER', 'service': InterfaceVpcEndpointAwsService.SECRETS_MANAGER}, + {'name': 'SNS', 'service': InterfaceVpcEndpointAwsService.SNS}, + {'name': 'STS', 'service': InterfaceVpcEndpointAwsService.STS} +] + +_GATEWAY_ENDPOINT_SERVICES = [ + {'name': 'S3', 'service': GatewayVpcEndpointAwsService.S3}, + {'name': 'DYNAMODB', 'service': GatewayVpcEndpointAwsService.DYNAMODB} +] + + +class NetworkTier(Stack): + """ + The network tier consists of all constructs that are required for the foundational + networking between the various components of the Deadline render farm. + """ + + @builtins.property # type: ignore + @jsii.member(jsii_name="availabilityZones") + def availability_zones(self) -> typing.List[builtins.str]: + """ + This overrides the availability zones the Stack will use. The zones that we set here are what + our VPC will use, so adding local zones to this return value will enable us to then deploy + infrastructure to them. + """ + return config.availability_zones_standard + config.availability_zones_local + + def __init__(self, scope: Construct, stack_id: str, **kwargs) -> None: + """ + Initializes a new instance of NetworkTier + """ + super().__init__(scope, stack_id, **kwargs) + + # We're creating a SubnetSelection with only the standard availability zones to be used to put + # the NAT gateway in and the VPC interface endpoints, because the local zones do no have + # these available. + standard_zone_subnets = SubnetSelection( + availability_zones=config.availability_zones_standard, + subnet_type=SubnetType.PUBLIC + ) + + # The VPC that all components of the render farm will be created in. We are using the `availability_zones()` + # method to override the availability zones that this VPC will use. + self.vpc = Vpc( + self, + 'Vpc', + max_azs=len(self.availability_zones), + subnet_configuration=[ + SubnetConfiguration( + name='Public', + subnet_type=SubnetType.PUBLIC, + cidr_mask=28 + ), + SubnetConfiguration( + name='Private', + subnet_type=SubnetType.PRIVATE, + cidr_mask=18 + ) + ], + nat_gateway_subnets=standard_zone_subnets + ) + + # Add interface endpoints + for idx, service_info in enumerate(_INTERFACE_ENDPOINT_SERVICES): + service_name = service_info['name'] + service = service_info['service'] + self.vpc.add_interface_endpoint( + service_name, + service=service, + subnets=standard_zone_subnets + ) + + # Add gateway endpoints + for idx, service_info in enumerate(_GATEWAY_ENDPOINT_SERVICES): + service_name = service_info['name'] + service = service_info['service'] + self.vpc.add_gateway_endpoint( + service_name, + service=service, + subnets=[standard_zone_subnets] + ) + + # Internal DNS zone for the VPC. + self.dns_zone = PrivateHostedZone( + self, + 'DnsZone', + vpc=self.vpc, + zone_name='deadline-test.internal' + ) diff --git a/examples/deadline/Local-Zone/python/package/lib/security_tier.py b/examples/deadline/Local-Zone/python/package/lib/security_tier.py new file mode 100644 index 000000000..c14169f0f --- /dev/null +++ b/examples/deadline/Local-Zone/python/package/lib/security_tier.py @@ -0,0 +1,36 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +from aws_cdk.core import ( + Construct, + Stack, +) +from aws_rfdk import ( + DistinguishedName, + X509CertificatePem +) + + +class SecurityTier(Stack): + """ + The security tier of the render farm. + This stack contains resources used to ensure the render farm is secure. + """ + def __init__(self, scope: Construct, stack_id: str, **kwargs): + """ + Initialize a new instance of ServiceTier + :param scope: The scope of this construct. + :param stack_id: The ID of this construct. + :param props: The properties for this construct. + :param kwargs: Any kwargs that need to be passed on to the parent class. + """ + super().__init__(scope, stack_id, **kwargs) + + # Our self-signed root CA certificate for the internal endpoints in the farm. + self.root_ca = X509CertificatePem( + self, + 'RootCA', + subject=DistinguishedName( + cn='SampleRootCA' + ) + ) diff --git a/examples/deadline/Local-Zone/python/package/lib/service_tier.py b/examples/deadline/Local-Zone/python/package/lib/service_tier.py new file mode 100644 index 000000000..f459feb78 --- /dev/null +++ b/examples/deadline/Local-Zone/python/package/lib/service_tier.py @@ -0,0 +1,157 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +from dataclasses import dataclass +from typing import List + +from aws_cdk.aws_ec2 import ( + IVpc, + SubnetSelection, + SubnetType +) +from aws_cdk.aws_elasticloadbalancingv2 import ( + ApplicationProtocol +) +from aws_cdk.aws_route53 import ( + IPrivateHostedZone +) +from aws_cdk.core import ( + Construct, + Duration, + RemovalPolicy, + Stack, + StackProps +) + +from aws_rfdk import ( + DistinguishedName, + SessionManagerHelper, + X509CertificatePem +) +from aws_rfdk.deadline import ( + AwsThinkboxEulaAcceptance, + RenderQueue, + RenderQueueHostNameProps, + RenderQueueTrafficEncryptionProps, + RenderQueueExternalTLSProps, + Repository, + RepositoryRemovalPolicies, + ThinkboxDockerImages, + VersionQuery +) + + +@dataclass +class ServiceTierProps(StackProps): + """ + Properties for ServiceTier + """ + # The VPC to deploy service tier resources into. + vpc: IVpc + # Whether the AWS Thinkbox End-User License Agreement is accepted or not + accept_aws_thinkbox_eula: AwsThinkboxEulaAcceptance + # The availability zones that components in this stack will be deployed into. These should all be in the same + # region and only be standard availability zones, as some constucts use services that aren't available in + # local zones yet. + availability_zones: List[str] + # Version of Deadline to use + deadline_version: str + # Internal DNS zone for the VPC + dns_zone: IPrivateHostedZone + # Our self-signed root CA certificate for the internal endpoints in the farm. + root_ca: X509CertificatePem + + +class ServiceTier(Stack): + """ + The service tier contains all "business-logic" constructs + (e.g. Repository, Render Queue, etc.) + """ + + def __init__(self, scope: Construct, stack_id: str, *, props: ServiceTierProps, **kwargs): + """ + Initialize a new instance of ServiceTier + :param scope: The scope of this construct. + :param stack_id: The ID of this construct. + :param props: The properties for this construct. + :param kwargs: Any kwargs that need to be passed on to the parent class. + """ + super().__init__(scope, stack_id, **kwargs) + + self.version = VersionQuery( + self, + 'Version', + version=props.deadline_version + ) + + # We are excluding the local zones from the Repository. This construct will create an + # EFS filesystem and DocDB cluster, both of which aren't available in any local zones at this time. + repository_subnets = SubnetSelection( + availability_zones=props.availability_zones, + subnet_type=SubnetType.PRIVATE + ) + repository = Repository( + self, + 'Repository', + vpc=props.vpc, + repository_installation_timeout=Duration.minutes(20), + removal_policy=RepositoryRemovalPolicies( + database=RemovalPolicy.DESTROY, + filesystem=RemovalPolicy.DESTROY, + ), + version=self.version, + vpc_subnets=repository_subnets + ) + + images = ThinkboxDockerImages( + self, + 'Images', + version=self.version, + user_aws_thinkbox_eula_acceptance=props.accept_aws_thinkbox_eula + ) + + server_cert = X509CertificatePem( + self, + 'RQCert', + subject=DistinguishedName( + cn=f'renderqueue.{props.dns_zone.zone_name}', + o='RFDK-Sample', + ou='RenderQueueExternal' + ), + signing_certificate=props.root_ca + ) + + # The render queue is also put only in the standard availability zones. The service itself + # is run in a single zone, while the load balancer that sits in front of it can be provided + # all the standard zones we're using. + render_queue_subnets = SubnetSelection( + availability_zones=[props.availability_zones[0]], + subnet_type=SubnetType.PRIVATE + ) + render_queue_alb_subnets = SubnetSelection( + availability_zones=props.availability_zones, + subnet_type=SubnetType.PRIVATE, + one_per_az=True, + ) + self.render_queue = RenderQueue( + self, + 'RenderQueue', + vpc=props.vpc, + images=images, + repository=repository, + hostname=RenderQueueHostNameProps( + hostname='renderqueue', + zone=props.dns_zone + ), + traffic_encryption=RenderQueueTrafficEncryptionProps( + external_tls=RenderQueueExternalTLSProps( + rfdk_certificate=server_cert + ), + internal_protocol=ApplicationProtocol.HTTPS + ), + version=self.version, + vpc_subnets=render_queue_subnets, + vpc_subnets_alb=render_queue_alb_subnets, + deletion_protection=False + ) + SessionManagerHelper.grant_permissions_to(self.render_queue.asg) diff --git a/examples/deadline/Local-Zone/python/requirements.txt b/examples/deadline/Local-Zone/python/requirements.txt new file mode 100644 index 000000000..d6e1198b1 --- /dev/null +++ b/examples/deadline/Local-Zone/python/requirements.txt @@ -0,0 +1 @@ +-e . diff --git a/examples/deadline/Local-Zone/python/setup.py b/examples/deadline/Local-Zone/python/setup.py new file mode 100644 index 000000000..fd5fdb9c9 --- /dev/null +++ b/examples/deadline/Local-Zone/python/setup.py @@ -0,0 +1,29 @@ +import setuptools + + +with open("README.md") as fp: + long_description = fp.read() + + +setuptools.setup( + name="all_in_aws_local_zones", + version="0.0.1", + + description="RFDK All In AWS using Local Zones", + long_description=long_description, + long_description_content_type="text/markdown", + + package_dir={"": "package"}, + packages=setuptools.find_packages(where="package"), + + install_requires=[ + "aws-cdk.aws-ec2==1.108.1", + "aws-cdk.aws-elasticloadbalancingv2==1.108.1", + "aws-cdk.aws-route53==1.108.1", + "aws-cdk.core==1.108.1", + "aws-rfdk==0.35.0", + "jsii==1.30.0", + ], + + python_requires=">=3.7", +) diff --git a/examples/deadline/Local-Zone/python/source.bat b/examples/deadline/Local-Zone/python/source.bat new file mode 100644 index 000000000..8f5744291 --- /dev/null +++ b/examples/deadline/Local-Zone/python/source.bat @@ -0,0 +1,13 @@ +@echo off + +rem The sole purpose of this script is to make the command +rem +rem source .env/bin/activate +rem +rem (which activates a Python virtualenv on Linux or Mac OS X) work on Windows. +rem On Windows, this command just runs this batch file (the argument is ignored). +rem +rem Now we don't need to document a Windows command for activating a virtualenv. + +echo Executing .env\Scripts\activate.bat for you +.env\Scripts\activate.bat diff --git a/examples/deadline/Local-Zone/ts/.gitignore b/examples/deadline/Local-Zone/ts/.gitignore new file mode 100644 index 000000000..7dd5058fe --- /dev/null +++ b/examples/deadline/Local-Zone/ts/.gitignore @@ -0,0 +1,20 @@ +*.d.ts +*.js +cdk.out +cdk.context.json + +# Used by nyc +.nyc_output +coverage +.nycrc + +# Exclude package artifacts +dist +*.tgz +.LAST_PACKAGE + +#Defines license that must be present +!license-header.js + +# The staged files for Deadline +stage diff --git a/examples/deadline/Local-Zone/ts/.npmignore b/examples/deadline/Local-Zone/ts/.npmignore new file mode 100644 index 000000000..a0a349e9e --- /dev/null +++ b/examples/deadline/Local-Zone/ts/.npmignore @@ -0,0 +1,18 @@ +################################################ +# if .npmignore exists, NPM ignores .gitignore # +################################################ + +# Exclude typescript source and config files +*.ts +tsconfig.json +*.tsbuildinfo + +# Include typescript declarations +!*.d.ts + +# Exclude jest config +jest.config.js + +# Exclude packaging artifacts +dist +.LAST_PACKAGE diff --git a/examples/deadline/Local-Zone/ts/README.md b/examples/deadline/Local-Zone/ts/README.md new file mode 100644 index 000000000..f9773bc80 --- /dev/null +++ b/examples/deadline/Local-Zone/ts/README.md @@ -0,0 +1,76 @@ +# RFDK Sample Application - Local Zones - Typescript + +## Overview +[Back to overview](../README.md) + +## Instructions + +--- +**NOTE** + +These instructions assume that your working directory is `examples/deadline/Local-Zones/ts/` relative to the root of the RFDK package. + +--- +1. This sample app on the `mainline` branch may contain features that have not yet been officially released, and may not be available in the `aws-rfdk` package installed through npm from npmjs. To work from an example of the latest release, please switch to the `release` branch. If you would like to try out unreleased features, you can stay on `mainline` and follow the instructions for building and using the `aws-rfdk` from your local repository. + +2. You must read and accept the [AWS Thinkbox End-User License Agreement (EULA)](https://www.awsthinkbox.com/end-user-license-agreement) to deploy and run Deadline. To do so, change the value of the `acceptAwsThinkboxEula` in `bin/config.ts` like this: + + ```ts + public readonly acceptAwsThinkboxEula: AwsThinkboxEulaAcceptance = AwsThinkboxEulaAcceptance.USER_ACCEPTS_AWS_THINKBOX_EULA; + ``` + +3. Change the value of the `deadlineVersion` variable in `bin/config.ts` to specify the desired version of Deadline to be deployed to your render farm. RFDK is compatible with Deadline versions 10.1.9.x and later. To see the available versions of Deadline, consult the [Deadline release notes](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/release-notes.html). It is recommended to use the latest version of Deadline available when building your farm, but to pin this version when the farm is ready for production use. For example, to pin to the latest `10.1.15.x` release of Deadline, use: + + ```ts + public readonly deadlineVersion: string = '10.1.15'; + ``` + +4. Change the value of the `deadlineClientLinuxAmiMap` variable in `bin/config.ts` to include the region + AMI ID mapping of your EC2 AMI(s) with Deadline Worker. You can use the following AWS CLI command to look up AMIs, replacing the `` and `` to match the AWS region and Deadline version you're looking for: + + ```bash + aws --region ec2 describe-images --owners 357466774442 --filters "Name=name,Values=*Worker*" "Name=name,Values=**" --query 'Images[*].[ImageId, Name]' --output text + ``` + +5. Also in `bin/config.ts`, you can set the `availabilityZonesStandard` and `availabilityZonesLocal` values to the availability zones you want to use. These values must all be from the same region. It's required that you use at least two standard zones, but you can use more if you'd like. For the local zones, you can use one or more. + +6. To gain the benefits of putting your workers in a local zone close to your asset server, you are going to want to set up a connection from your local network to the one you're creating in AWS. + 1. You should start by reading through the [Connecting to the Render Farm](https://docs.aws.amazon.com/rfdk/latest/guide/connecting-to-render-farm.html) documentation and implementing one of the methods for connecting your network to your AWS VPC described there. + 2. With whichever option you choose, you'll want to make sure you are propagating the worker subnets to your local network. All the options in the document show how to propagate all the private subnets, which will include the ones used by the workers. + 3. Ensure your worker fleet's security group allows traffic from your network on the correct ports that your NFS requires to be open. The documentation shows how to [allow connections to the Render Queue](https://docs.aws.amazon.com/rfdk/latest/guide/connecting-to-render-farm.html#allowing-connection-to-the-render-queue), which you may also want to enable if you plan on connecting any of your local machines to your render farm, but you would also want to do something similar for the worker fleet, for example, ports `22` and `2049` are commonly required for NFS, so this code could be added to the `ComputeTier`: + + ```ts + // The customer-prefix-cidr-range needs to be replaced by the CIDR range for your local network that you used when configuring the VPC connection + this.workerFleet.connections.allowFrom(Peer.ipv4('customer-prefix-cidr-range'), Port.tcp(22)); + this.workerFleet.connections.allowFrom(Peer.ipv4('customer-prefix-cidr-range'), Port.udp(22)); + this.workerFleet.connections.allowFrom(Peer.ipv4('customer-prefix-cidr-range'), Port.tcp(2049)); + this.workerFleet.connections.allowFrom(Peer.ipv4('customer-prefix-cidr-range'), Port.tcp(2049)); + ``` + + 4. Add user-data to mount the NFS on the compute tier. This can be provided in the `UserDataProvider` in the `ComputeTier`. + 5. (optional) Set up [path mapping rules in Deadline](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/cross-platform.html). + +7. Build the `aws-rfdk` package, and then build the sample app. The `tsconfig.json` for this example app contains a reference to the local `aws-rfdk` package and will link your build artifacts: + + ```bash + # Navigate to the root directory of the RFDK repository (assumes you started in the example's directory) + pushd ../../../.. + # Enter the Docker container, run the build, and then exit + ./scripts/rfdk_build_environment.sh + ./build.sh + exit + # Navigate back to the example directory + popd + # Run the example's build + yarn build + ``` + +8. Deploy all the stacks in the sample app: + + ``` + cdk deploy "*" + ``` +9. Once you are finished with the sample app, you can tear it down by running: + + ``` + cdk destroy "*" + ``` \ No newline at end of file diff --git a/examples/deadline/Local-Zone/ts/bin/app.ts b/examples/deadline/Local-Zone/ts/bin/app.ts new file mode 100644 index 000000000..dc075f05c --- /dev/null +++ b/examples/deadline/Local-Zone/ts/bin/app.ts @@ -0,0 +1,58 @@ +#!/usr/bin/env node +/** + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +import 'source-map-support/register'; +import { + MachineImage, +} from '@aws-cdk/aws-ec2'; +import * as cdk from '@aws-cdk/core'; + +import { config } from './config'; +import { ComputeTier } from '../lib/compute-tier'; +import { NetworkTier } from '../lib/network-tier'; +import { SecurityTier } from '../lib/security-tier'; +import { ServiceTier } from '../lib/service-tier'; + + // ------------------------------ // + // --- Validate Config Values --- // + // ------------------------------ // + if (!config.keyPairName) { + console.log('EC2 key pair name not specified. You will not have SSH access to the render farm.'); + } + +// ------------------- // +// --- Application --- // +// ------------------- // + +const env = { + account: process.env.CDK_DEPLOY_ACCOUNT ?? process.env.CDK_DEFAULT_ACCOUNT, + region: process.env.CDK_DEPLOY_REGION ?? process.env.CDK_DEFAULT_REGION, +}; + +const app = new cdk.App(); + +const network = new NetworkTier(app, 'NetworkTier', { env }); + +const security = new SecurityTier(app, 'SecurityTier', { env }); + +const service = new ServiceTier(app, 'ServiceTier', { + env, + vpc: network.vpc, + availabilityZones: config.availabilityZonesStandard, + deadlineVersion: config.deadlineVersion, + rootCa: security.rootCa, + dnsZone: network.dnsZone, + acceptAwsThinkboxEula: config.acceptAwsThinkboxEula, +}); + +new ComputeTier(app, 'ComputeTier', { + env, + vpc: network.vpc, + availabilityZones: config.availabilityZonesLocal, + renderQueue: service.renderQueue, + workerMachineImage: MachineImage.genericLinux(config.deadlineClientLinuxAmiMap), + keyPairName: config.keyPairName, +}); diff --git a/examples/deadline/Local-Zone/ts/bin/config.ts b/examples/deadline/Local-Zone/ts/bin/config.ts new file mode 100644 index 000000000..6470c74e2 --- /dev/null +++ b/examples/deadline/Local-Zone/ts/bin/config.ts @@ -0,0 +1,57 @@ +/** + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +import 'source-map-support/register'; +import { AwsThinkboxEulaAcceptance } from 'aws-rfdk/deadline'; + +/** + * Configuration values for the sample app. + * + * TODO: Fill these in with your own values. + */ +class AppConfig { + /** + * Change this value to AwsThinkboxEulaAcceptance.USER_ACCEPTS_AWS_THINKBOX_EULA if you wish to accept the EULA for + * Deadline and proceed with Deadline deployment. Users must explicitly accept the AWS Thinkbox EULA before using the + * AWS Thinkbox Deadline container images. + * + * See https://www.awsthinkbox.com/end-user-license-agreement for the terms of the agreement. + */ + public readonly acceptAwsThinkboxEula: AwsThinkboxEulaAcceptance = AwsThinkboxEulaAcceptance.USER_REJECTS_AWS_THINKBOX_EULA; + + /** + * The standard availability zones that the render farm will deploy into. It is recommended to use at least + * two and they must be from the same region. The default values being provided are two of the four standard + * zones in us-west-2, located in Oregon. + */ + public readonly availabilityZonesStandard: string[] = ['us-west-2a', 'us-west-2b']; + + /** + * The local availability zones that will hold the worker fleet. They must belong to the same region as the standard + * zones. The default value being provided here is one of the two local zones in us-west-2, located in Los Angeles. + */ + public readonly availabilityZonesLocal: string[] = ['us-west-2-lax-1a']; + + /** + * The version of Deadline to use on the render farm. Some examples of pinned version values are "10", "10.1", or + * "10.1.16" + * @default 10.1.16 is used, to match the worker AMI ID provided below + */ + public readonly deadlineVersion: string = '10.1.16'; + + /** + * A map of regions to Deadline Client Linux AMIs. As an example, the Linux Deadline 10.1.16.8 AMI ID from us-west-2 + * is filled in. It can be used as-is, added to, or replaced. Ideally the version here should match the one in + * package.json used for staging the render queue and usage based licensing recipes. + */ + public readonly deadlineClientLinuxAmiMap: Record = {['us-west-2']: 'ami-0aa3610842bc3534d'}; + + /** + * (Optional) The name of the EC2 keypair to associate with instances. + */ + public readonly keyPairName?: string; +} + +export const config = new AppConfig(); diff --git a/examples/deadline/Local-Zone/ts/cdk.json b/examples/deadline/Local-Zone/ts/cdk.json new file mode 100644 index 000000000..97bafc2cb --- /dev/null +++ b/examples/deadline/Local-Zone/ts/cdk.json @@ -0,0 +1,7 @@ +{ + "app": "npx ts-node bin/app.ts", + "context": { + "@aws-cdk/core:enableStackNameDuplicates": "true", + "aws-cdk:enableDiffNoFail": "true" + } +} diff --git a/examples/deadline/Local-Zone/ts/clean.sh b/examples/deadline/Local-Zone/ts/clean.sh new file mode 100755 index 000000000..0a907b641 --- /dev/null +++ b/examples/deadline/Local-Zone/ts/clean.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +set -euo pipefail + +rm -rf node_modules/ cdk.out/ cdk.context.json stage/ diff --git a/examples/deadline/Local-Zone/ts/lib/compute-tier.ts b/examples/deadline/Local-Zone/ts/lib/compute-tier.ts new file mode 100644 index 000000000..d0635a991 --- /dev/null +++ b/examples/deadline/Local-Zone/ts/lib/compute-tier.ts @@ -0,0 +1,126 @@ +/** + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +import { + BastionHostLinux, + IMachineImage, + InstanceClass, + InstanceSize, + InstanceType, + IVpc, + SubnetSelection, + SubnetType, +} from '@aws-cdk/aws-ec2'; +import * as cdk from '@aws-cdk/core'; +import { + HealthMonitor, + IHealthMonitor, + SessionManagerHelper, +} from 'aws-rfdk'; +import { + IHost, + InstanceUserDataProvider, + IRenderQueue, + IWorkerFleet, + WorkerInstanceFleet, +} from 'aws-rfdk/deadline'; + +/** + * Properties for {@link ComputeTier}. + */ +export interface ComputeTierProps extends cdk.StackProps { + /** + * The VPC to deploy resources into. + */ + readonly vpc: IVpc; + + /** + * The availability zones the worker instances will be deployed to. This can include your local + * zones, but they must belong to the same region as the standard zones used in other stacks in + * this application. + */ + readonly availabilityZones: string[], + + /** + * The {@link IRenderQueue} that Deadline Workers connect to. + */ + readonly renderQueue: IRenderQueue; + + /** + * The {@link IMachineImage} to use for Workers (needs Deadline Client installed). + */ + readonly workerMachineImage: IMachineImage; + + /** + * The name of the EC2 keypair to associate with Worker nodes. + */ + readonly keyPairName?: string; + + /** + * The bastion host to allow connection to Worker nodes. + */ + readonly bastion?: BastionHostLinux; +} + +class UserDataProvider extends InstanceUserDataProvider { + preWorkerConfiguration(host: IHost): void { + // Add code here for mounting your NFS to the workers + host.userData.addCommands('echo preWorkerConfiguration'); + } +} + +/** + * The computer tier consists of the worker fleets. We'll be deploying the workers into the + * local zone we're using. + */ +export class ComputeTier extends cdk.Stack { + /** + * The {@link IWorkerFleet}. + */ + public readonly workerFleet: IWorkerFleet; + + /** + * The {@link IHealthMonitor} used to maintain the worker fleet. + */ + public readonly healthMonitor: IHealthMonitor; + + /** + * Initializes a new instance of {@link ComputeTier}. + */ + constructor(scope: cdk.Construct, id: string, props: ComputeTierProps) { + super(scope, id, props); + + // We can put the health monitor and worker fleet in all of the local zones we're using + const subnets: SubnetSelection = { + availabilityZones: props.availabilityZones, + subnetType: SubnetType.PRIVATE, + onePerAz: true, + }; + + this.healthMonitor = new HealthMonitor(this, 'HealthMonitor', { + vpc: props.vpc, + vpcSubnets: subnets, + deletionProtection: false, + }); + + this.workerFleet = new WorkerInstanceFleet(this, 'WorkerFleet', { + vpc: props.vpc, + renderQueue: props.renderQueue, + workerMachineImage: props.workerMachineImage, + healthMonitor: this.healthMonitor, + keyName: props.keyPairName, + // Not all instance types will be available in local zones. For a list of the instance types + // available in each local zone, you can refer to: + // https://aws.amazon.com/about-aws/global-infrastructure/localzones/features/#AWS_Services + // BURSTABLE3 is a T3; the third generation of burstable instances + instanceType: InstanceType.of(InstanceClass.BURSTABLE3, InstanceSize.LARGE), + userDataProvider: new UserDataProvider(this, 'UserDataProvider'), + vpcSubnets: subnets, + }); + + SessionManagerHelper.grantPermissionsTo(this.workerFleet); + } +} + diff --git a/examples/deadline/Local-Zone/ts/lib/network-tier.ts b/examples/deadline/Local-Zone/ts/lib/network-tier.ts new file mode 100644 index 000000000..be80d8f86 --- /dev/null +++ b/examples/deadline/Local-Zone/ts/lib/network-tier.ts @@ -0,0 +1,123 @@ +/** + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +import { + GatewayVpcEndpointAwsService, + IInterfaceVpcEndpointService, + IGatewayVpcEndpointService, + InterfaceVpcEndpointAwsService, + IVpc, + SubnetSelection, + SubnetType, + Vpc, +} from '@aws-cdk/aws-ec2'; +import { + PrivateHostedZone, +} from '@aws-cdk/aws-route53'; +import * as cdk from '@aws-cdk/core'; + +import { config } from '../bin/config'; + +/** + * The network tier is where we define our VPC that will host all the other components that will be + * deployed. Adding our local zones to this VPC will allow us to use them for our worker fleets in + * a dependent stack. + */ +export class NetworkTier extends cdk.Stack { + /** + * The VPC that all components of the render farm will be created in. + */ + public readonly vpc: IVpc; + + /** + * Internal DNS zone for the VPC. + */ + public readonly dnsZone: PrivateHostedZone; + + /** + * The interface endpoints for the AWS services used in this app. + */ + private static readonly INTERFACE_ENDPOINT_SERVICES: { name: string, service: IInterfaceVpcEndpointService }[] = [ + { name: 'CLOUDWATCH', service: InterfaceVpcEndpointAwsService.CLOUDWATCH }, + { name: 'CLOUDWATCH_EVENTS', service: InterfaceVpcEndpointAwsService.CLOUDWATCH_EVENTS }, + { name: 'CLOUDWATCH_LOGS', service: InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS }, + { name: 'EC2', service: InterfaceVpcEndpointAwsService.EC2 }, + { name: 'ECR', service: InterfaceVpcEndpointAwsService.ECR }, + { name: 'ECS', service: InterfaceVpcEndpointAwsService.ECS }, + { name: 'KMS', service: InterfaceVpcEndpointAwsService.KMS }, + { name: 'SECRETS_MANAGER', service: InterfaceVpcEndpointAwsService.SECRETS_MANAGER }, + { name: 'SNS', service: InterfaceVpcEndpointAwsService.SNS }, + { name: 'STS', service: InterfaceVpcEndpointAwsService.STS }, + ]; + + /** + * The gateway endpoints for the AWS services used in this app. + */ + private static readonly GATEWAY_ENDPOINT_SERVICES: { name: string, service: IGatewayVpcEndpointService }[] = [ + { name: 'S3', service: GatewayVpcEndpointAwsService.S3 }, + { name: 'DYNAMODB', service: GatewayVpcEndpointAwsService.DYNAMODB }, + ]; + + /** + * This overrides the availability zones the Stack will use. The zones that we set here are what + * our VPC will use, so adding local zones to this return value will enable us to then deploy + * infrastructure to them. + */ + public get availabilityZones(): string[] { + return config.availabilityZonesStandard.concat(config.availabilityZonesLocal); + } + + /** + * Initializes a new instance of {@link NetworkTier}. + */ + constructor(scope: cdk.Construct, id: string, props: cdk.StackProps) { + super(scope, id, props); + + // We're creating a SubnetSelection with only the standard availability zones to be + // used to put the NAT gateway in and the VPC endpoints, because the local zones do no + // have these available. + const standardZoneSubnets: SubnetSelection = { + availabilityZones: config.availabilityZonesStandard, + subnetType: SubnetType.PUBLIC, + }; + + // The VPC that all components of the render farm will be created in. We are using the `availabilityZones()` + // method to override the availability zones that this VPC will use. + this.vpc = new Vpc(this, 'Vpc', { + maxAzs: this.availabilityZones.length, + subnetConfiguration: [ + { + name: 'Public', + subnetType: SubnetType.PUBLIC, + cidrMask: 28, + }, + { + name: 'Private', + subnetType: SubnetType.PRIVATE, + cidrMask: 18, + }, + ], + natGatewaySubnets: standardZoneSubnets, + }); + + NetworkTier.INTERFACE_ENDPOINT_SERVICES.forEach(serviceInfo => { + this.vpc.addInterfaceEndpoint(serviceInfo.name, { + service: serviceInfo.service, + subnets: standardZoneSubnets, + }); + }); + NetworkTier.GATEWAY_ENDPOINT_SERVICES.forEach(serviceInfo => { + this.vpc.addGatewayEndpoint(serviceInfo.name, { + service: serviceInfo.service, + subnets: [ standardZoneSubnets ], + }); + }); + + this.dnsZone = new PrivateHostedZone(this, 'DnsZone', { + vpc: this.vpc, + zoneName: 'deadline-test.internal', + }); + } +} diff --git a/examples/deadline/Local-Zone/ts/lib/security-tier.ts b/examples/deadline/Local-Zone/ts/lib/security-tier.ts new file mode 100644 index 000000000..f4e4376af --- /dev/null +++ b/examples/deadline/Local-Zone/ts/lib/security-tier.ts @@ -0,0 +1,36 @@ +/** + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +import * as cdk from '@aws-cdk/core'; +import { + X509CertificatePem, +} from 'aws-rfdk'; + +/** + * The security tier of the render farm. This stack contains resources used to + * ensure the render farm is secure. + */ +export class SecurityTier extends cdk.Stack { + /** + * Our self-signed root CA certificate for the internal endpoints in the farm. + */ + readonly rootCa: X509CertificatePem; + + /** + * Initializes a new instance of {@link SecurityTier}. + * @param scope The scope of this construct. + * @param id The ID of this construct. + * @param props The properties for the security tier. + */ + constructor(scope: cdk.Construct, id: string, props: cdk.StackProps) { + super(scope, id, props); + + this.rootCa = new X509CertificatePem(this, 'RootCA', { + subject: { + cn: 'SampleRootCA', + }, + }); + } +}; diff --git a/examples/deadline/Local-Zone/ts/lib/service-tier.ts b/examples/deadline/Local-Zone/ts/lib/service-tier.ts new file mode 100644 index 000000000..eed78e23d --- /dev/null +++ b/examples/deadline/Local-Zone/ts/lib/service-tier.ts @@ -0,0 +1,162 @@ +/** + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +import { + IVpc, + SubnetSelection, + SubnetType, +} from '@aws-cdk/aws-ec2'; +import { + ApplicationProtocol, +} from '@aws-cdk/aws-elasticloadbalancingv2'; +import { + IPrivateHostedZone, +} from '@aws-cdk/aws-route53'; +import { + Construct, + Duration, + RemovalPolicy, + Stack, + StackProps, +} from '@aws-cdk/core'; +import { + SessionManagerHelper, + X509CertificatePem, +} from 'aws-rfdk'; +import { + AwsThinkboxEulaAcceptance, + RenderQueue, + Repository, + ThinkboxDockerImages, + VersionQuery, +} from 'aws-rfdk/deadline'; + +/** + * Properties for {@link ServiceTier}. + */ +export interface ServiceTierProps extends StackProps { + /** + * The VPC to deploy service tier resources into. + */ + readonly vpc: IVpc; + + /** + * Whether the AWS Thinkbox End-User License Agreement is accepted or not + */ + readonly acceptAwsThinkboxEula: AwsThinkboxEulaAcceptance; + + /** + * The availability zones that components in this stack will be deployed into. These should all be in the same + * region and only be standard availability zones, as some constucts use services that aren't available in + * local zones yet. + */ + readonly availabilityZones: string[], + + /** + * Internal DNS zone for the VPC. + */ + readonly dnsZone: IPrivateHostedZone; + + /** + * Our self-signed root CA certificate for the internal endpoints in the farm. + */ + readonly rootCa: X509CertificatePem; + + /** + * Version of Deadline to use. + * @default The latest available release of Deadline is used + */ + readonly deadlineVersion?: string; +} + +/** + * The service tier contains all "business-logic" constructs (e.g. Repository, Render Queue, etc.). + */ +export class ServiceTier extends Stack { + /** + * The render queue. + */ + public readonly renderQueue: RenderQueue; + + /** + * The version of Deadline configured by the app. + */ + public readonly version: VersionQuery; + + /** + * Initializes a new instance of {@link ServiceTier}. + */ + constructor(scope: Construct, id: string, props: ServiceTierProps) { + super(scope, id, props); + + this.version = new VersionQuery(this, 'Version', { + version: props.deadlineVersion, + }); + + // We are excluding the local zones from the Repository. This construct will create an + // EFS filesystem and DocDB cluster, both of which aren't available in any local zones at this time. + const repositorySubnets: SubnetSelection = { + availabilityZones: props.availabilityZones, + subnetType: SubnetType.PRIVATE, + }; + const repository = new Repository(this, 'Repository', { + vpc: props.vpc, + version: this.version, + removalPolicy: { + database: RemovalPolicy.DESTROY, + filesystem: RemovalPolicy.DESTROY, + }, + repositoryInstallationTimeout: Duration.minutes(20), + vpcSubnets: repositorySubnets, + }); + + const images = new ThinkboxDockerImages(this, 'Images', { + version: this.version, + userAwsThinkboxEulaAcceptance: props.acceptAwsThinkboxEula, + }); + + const serverCert = new X509CertificatePem(this, 'RQCert', { + subject: { + cn: `renderqueue.${props.dnsZone.zoneName}`, + o: 'RFDK-Sample', + ou: 'RenderQueueExternal', + }, + signingCertificate: props.rootCa, + }); + + // The render queue is also put only in the standard availability zones. The service itself + // is run in a single zone, while the load balancer that sits in front of it can be provided + // all the standard zones we're using. + const renderQueueSubnets: SubnetSelection = { + availabilityZones: [ props.availabilityZones[0] ], + subnetType: SubnetType.PRIVATE, + }; + const renderQueueAlbSubnets: SubnetSelection = { + availabilityZones: props.availabilityZones, + subnetType: SubnetType.PRIVATE, + onePerAz: true, + }; + this.renderQueue = new RenderQueue(this, 'RenderQueue', { + vpc: props.vpc, + images: images, + repository, + hostname: { + hostname: 'renderqueue', + zone: props.dnsZone, + }, + trafficEncryption: { + externalTLS: { + rfdkCertificate: serverCert, + }, + internalProtocol: ApplicationProtocol.HTTPS, + }, + version: this.version, + vpcSubnets: renderQueueSubnets, + vpcSubnetsAlb: renderQueueAlbSubnets, + deletionProtection: false, + }); + SessionManagerHelper.grantPermissionsTo(this.renderQueue.asg); + } +} diff --git a/examples/deadline/Local-Zone/ts/package.json b/examples/deadline/Local-Zone/ts/package.json new file mode 100644 index 000000000..efba592d8 --- /dev/null +++ b/examples/deadline/Local-Zone/ts/package.json @@ -0,0 +1,29 @@ +{ + "name": "all-in-farm-local-zone", + "version": "0.35.0", + "bin": { + "app": "bin/app.js" + }, + "scripts": { + "build": "tsc", + "build+test": "yarn build && yarn test", + "cdk": "cdk", + "clean": "tsc --build --clean && bash ./clean.sh", + "test": "echo 'no tests to run'", + "watch": "tsc -w" + }, + "devDependencies": { + "@types/node": "^15.12.2", + "aws-cdk": "1.108.1", + "ts-node": "^9.1.1", + "typescript": "~4.3.2" + }, + "dependencies": { + "@aws-cdk/aws-ec2": "1.108.1", + "@aws-cdk/aws-elasticloadbalancingv2": "1.108.1", + "@aws-cdk/aws-route53": "1.108.1", + "@aws-cdk/core": "1.108.1", + "aws-rfdk": "0.35.0", + "source-map-support": "^0.5.19" + } +} diff --git a/examples/deadline/Local-Zone/ts/tsconfig.json b/examples/deadline/Local-Zone/ts/tsconfig.json new file mode 100644 index 000000000..f247c93ca --- /dev/null +++ b/examples/deadline/Local-Zone/ts/tsconfig.json @@ -0,0 +1,35 @@ +{ + "compilerOptions": { + "target": "ES2018", + "module": "commonjs", + "lib": [ + "es2018" + ], + "strict": true, + "alwaysStrict": true, + "declaration": true, + "inlineSourceMap": true, + "inlineSources": true, + "noUnusedLocals": true, + "noUnusedParameters": true, + "noImplicitReturns": true, + "noFallthroughCasesInSwitch": true, + "resolveJsonModule": true, + "composite": true, + "incremental": true + }, + "exclude": [ + "cdk.out" + ], + "include": [ + "**/*.ts" + ], + "files": [ + "package.json" + ], + "references": [ + { + "path": "../../../../packages/aws-rfdk" + } + ] +} diff --git a/scripts/rfdk_build_environment.sh b/scripts/rfdk_build_environment.sh index b96fc7767..d21454987 100755 --- a/scripts/rfdk_build_environment.sh +++ b/scripts/rfdk_build_environment.sh @@ -4,7 +4,7 @@ # SPDX-License-Identifier: Apache-2.0 # This is a helper script for entering a docker build environment suitable for -# building the RFDK. To use: Run this script from the root directory of the RFDK +# building the RFDK. To use: Run this script from the root directory of the RFDK # repository. # Make sure we're running from the root of the CDK repo