feat(examples): added ec2 image builder example #292

horsmand · 2021-01-19T19:06:43Z

Fixes #304

The apps in both languages have been deployed successfully.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

examples/deadline/EC2-Image-Builder/README.md

ddneilson

This is pretty slick. Thanks for creating this example, David.

ddneilson · 2021-02-12T17:49:34Z

examples/deadline/EC2-Image-Builder/README.md

@@ -0,0 +1,60 @@
+# RFDK Sample Application - EC2 Image Builder
+
+Keeping software updated on custom AMIs can be a pain if you're creating them manually. The EC2 Image Builder service has matured to the point that it is a viable option for automating the process. While the support in CloudFormation and CDK is still a bit lacking, it is usable and can be worked into an RFDK app to build the required AMIs for the worker fleet on the fly.


I'd avoid the value/judgement wording.

Suggested change

Keeping software updated on custom AMIs can be a pain if you're creating them manually. The EC2 Image Builder service has matured to the point that it is a viable option for automating the process. While the support in CloudFormation and CDK is still a bit lacking, it is usable and can be worked into an RFDK app to build the required AMIs for the worker fleet on the fly.

Keeping software updated on custom AMIs can be a pain if you're creating them manually. The EC2 Image Builder service is an option for automating the process. The CDK constructs for Image Builder can be worked into an RFDK app to build the required AMIs for the worker fleet on the fly.

ddneilson · 2021-02-12T17:53:37Z

examples/deadline/EC2-Image-Builder/README.md

+
+This construct creates all the infrastructure required by Image Builder to install Deadline onto an existing AMI and then create a WorkerInstanceFleet that uses it. The configuration for the image creation is as simple as providing the Deadline version, ID of the parent AMI, and the OS. An image version also needs to be supplied. One strategy to use with versioning your image is to start with version `1.0.0` and bump the version if you need to change any of the input parameters, such as changin the parent AMI or Deadline version. Images for different OSes can be versioned separately.
+
+CDK does not have L2 constructs for Image Builder yet, so we are using the L1 constructs. The documentation for these in CDK is sparse, so referring to the [AWS CloudFormation user guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_ImageBuilder.html) will provide more details.


I'd suggest linking to https://docs.aws.amazon.com/cdk/latest/guide/constructs.html#constructs_lib here, if you're going to talk about L1 vs L2 constructs. I wouldn't expect most readers to be familiar with the distinction.

ddneilson · 2021-02-12T17:54:19Z

examples/deadline/EC2-Image-Builder/README.md

+If you would like to upgrade the version of Deadline your worker fleet is using, you will need to bump the image version to a value such as `1.0.1` along with changing the Deadline version. Since the worker fleet is backed by an autoscaling group, the new image will get built and the autoscaling group's launch configuration will get updated; however, this doesn't replace existing instances, it will only affect new ones that get deployed. To have current workers get replaced with the new version, you have a few options:
+
+1. Set your `desiredCapacity` and `minCapacity` on the worker fleet to `0` before you perform the redeployment that will create your new AMI, and then do a follow up deployment with these fields set to their previous values (or removed).
+1. Manually terminate the worker instances from the console before performing your redeployment.


Suggested change

1. Manually terminate the worker instances from the console before performing your redeployment.

1. Manually terminate the worker instances from the console after performing your redeployment.

??

I did mean "before" but I see how it's confusing. I clarified a bit.

ddneilson · 2021-02-12T17:54:59Z

examples/deadline/EC2-Image-Builder/README.md

+
+#### AMI Storage
+
+When performing an update to or deletion of the DeadlineImage constuct, any AMI that was created by a previous deployment of the construct will not be deleted. They are still available in EC2 and can be seen under `Images > AMIs` in the EC2 console or in the `My AMIs` section of the Launch instance wizard. You can continue to use them like any other AMI, or deregister them if you no longer require them. The cost of storing these AMIs depends on the size of the disk you took a snapshot of to create them, for EBS-backed AMIs you can find snapshot costs on their [EBS pricing page](https://aws.amazon.com/ebs/pricing/). For S3-backed AMI's, you'll pay for the storage fees of the data that needs to be stored based on (S3 pricing)[https://aws.amazon.com/s3/pricing/], whether you have an instance running or not.


Suggested change

When performing an update to or deletion of the DeadlineImage constuct, any AMI that was created by a previous deployment of the construct will not be deleted. They are still available in EC2 and can be seen under `Images > AMIs` in the EC2 console or in the `My AMIs` section of the Launch instance wizard. You can continue to use them like any other AMI, or deregister them if you no longer require them. The cost of storing these AMIs depends on the size of the disk you took a snapshot of to create them, for EBS-backed AMIs you can find snapshot costs on their [EBS pricing page](https://aws.amazon.com/ebs/pricing/). For S3-backed AMI's, you'll pay for the storage fees of the data that needs to be stored based on (S3 pricing)[https://aws.amazon.com/s3/pricing/], whether you have an instance running or not.

When performing an update to, or deletion of, the DeadlineImage constuct, any AMI that was created by a previous deployment of the construct will not be deleted. They are still available in EC2 and can be seen under `Images > AMIs` in the EC2 console or in the `My AMIs` section of the Launch instance wizard. You can continue to use them like any other AMI, or deregister them if you no longer require them. The cost of storing these AMIs depends on the size of the disk you took a snapshot of to create them, for EBS-backed AMIs you can find snapshot costs on their [EBS pricing page](https://aws.amazon.com/ebs/pricing/). For S3-backed AMI's, you'll pay for the storage fees of the data that needs to be stored based on (S3 pricing)[https://aws.amazon.com/s3/pricing/], whether you have an instance running or not.

ddneilson · 2021-02-12T17:56:38Z

examples/deadline/EC2-Image-Builder/README.md

+
+At this point in time, we do not provide Amazon-managed Deadline components for you to consume, so if you do choose to go this route, you would need to create a new version of
+your Deadline component for each release of Deadline, but you may decide this is worthwhile, depending on how many image variations you use for your workers. The management of
+EC2 Image Builder pipelines with CDK isn't very well developed yet, so if you do decide to create one, the AWS Console is recommended.


Suggested change

EC2 Image Builder pipelines with CDK isn't very well developed yet, so if you do decide to create one, the AWS Console is recommended.

EC2 Image Builder pipelines with CDK is not developed at this time, so if you do decide to create one, the AWS Console is recommended.

ddneilson · 2021-02-12T18:03:42Z

examples/deadline/EC2-Image-Builder/python/package/lib/deadline_image.py

+                'ssmmessages:OpenControlChannel',
+                'ssmmessages:OpenDataChannel',
+            ],
+            resources=['*'],


Hmm. Could these permissions be restricted any more than just '*' ?
e.g. the s3 permissions scoped to a particular bucket, or bucket prefix? ( I think these are for the Deadline installer?)

I'm not sure why I didn't use the Amazon managed policies to begin with. I switched to those and added the S3 permissions specifically for our bucket the installers are in.

examples/deadline/EC2-Image-Builder/README.md

examples/deadline/EC2-Image-Builder/components/deadline-linux.component.template

examples/deadline/EC2-Image-Builder/components/deadline-windows.component.template

examples/deadline/EC2-Image-Builder/python/package/config.py

examples/deadline/EC2-Image-Builder/python/package/lib/deadline_image.py

examples/deadline/EC2-Image-Builder/python/setup.py

examples/deadline/EC2-Image-Builder/ts/lib/template.ts

examples/deadline/EC2-Image-Builder/README.md

examples/deadline/EC2-Image-Builder/components/deadline-windows.component.template

examples/deadline/EC2-Image-Builder/ts/lib/deadline-machine-image.ts

yashda

The changes looks good. Also, the example is simple and easy to understand.
Approving. Thanks David!

horsmand · 2021-03-03T21:11:26Z

My latest push updated the version of RFDK and CDK but there were no functional changes.

jericht

Just some minor comments, but overall this is a nice example that's easy to follow!

examples/deadline/EC2-Image-Builder/README.md

examples/deadline/EC2-Image-Builder/python/README.md

examples/deadline/EC2-Image-Builder/python/package/lib/base_farm_stack.py

examples/deadline/EC2-Image-Builder/ts/lib/compute-stack.ts

examples/deadline/EC2-Image-Builder/ts/lib/deadline-machine-image.ts

examples/deadline/EC2-Image-Builder/python/package/lib/deadline_machine_image.py

examples/deadline/EC2-Image-Builder/ts/lib/deadline-machine-image.ts

examples/deadline/EC2-Image-Builder/ts/package.json

jusiskin added the contribution/core This is a PR that came from AWS. label Jan 22, 2021

horsmand force-pushed the ib-example branch 2 times, most recently from 464a674 to 95694b3 Compare January 30, 2021 01:10

horsmand marked this pull request as ready for review January 30, 2021 01:24

horsmand force-pushed the ib-example branch from 95694b3 to adb34c2 Compare February 1, 2021 15:38

horsmand commented Feb 9, 2021

View reviewed changes

examples/deadline/EC2-Image-Builder/README.md Outdated Show resolved Hide resolved

jusiskin self-requested a review February 10, 2021 22:35

ddneilson suggested changes Feb 12, 2021

View reviewed changes

yashda suggested changes Feb 12, 2021

View reviewed changes

jusiskin removed their request for review February 12, 2021 23:18

horsmand force-pushed the ib-example branch from adb34c2 to 00ea30c Compare February 17, 2021 23:16

horsmand requested review from yashda and ddneilson February 17, 2021 23:17

yashda reviewed Feb 18, 2021

View reviewed changes

horsmand force-pushed the ib-example branch from 00ea30c to d62b5e3 Compare February 23, 2021 23:53

horsmand requested a review from yashda February 23, 2021 23:57

yashda previously approved these changes Feb 25, 2021

View reviewed changes

horsmand added 4 commits March 3, 2021 18:52

feat(examples): added ec2 image builder example

1d084cb

Addressed feedback

7bf98d9

Addressed feedback round 2

0e8f81c

Updated to new RFDK version

fd7efba

horsmand dismissed yashda’s stale review via fd7efba March 3, 2021 21:08

horsmand force-pushed the ib-example branch from d62b5e3 to fd7efba Compare March 3, 2021 21:08

horsmand requested a review from yashda March 3, 2021 21:10

jericht requested changes Mar 4, 2021

View reviewed changes

Addressed feedback round 3

42c27ab

jericht approved these changes Mar 10, 2021

View reviewed changes

ddneilson approved these changes Mar 12, 2021

View reviewed changes

horsmand merged commit 2375439 into aws:mainline Mar 16, 2021

horsmand deleted the ib-example branch March 16, 2021 15:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(examples): added ec2 image builder example #292

feat(examples): added ec2 image builder example #292

horsmand commented Jan 19, 2021 •

edited

Loading

ddneilson left a comment

ddneilson Feb 12, 2021

horsmand Feb 23, 2021

ddneilson Feb 12, 2021

horsmand Feb 23, 2021

ddneilson Feb 12, 2021

horsmand Feb 17, 2021

ddneilson Feb 12, 2021

horsmand Feb 23, 2021

ddneilson Feb 12, 2021

horsmand Feb 23, 2021

ddneilson Feb 12, 2021

horsmand Feb 17, 2021

yashda left a comment

horsmand commented Mar 3, 2021

jericht left a comment

		@@ -0,0 +1,60 @@
		# RFDK Sample Application - EC2 Image Builder

		Keeping software updated on custom AMIs can be a pain if you're creating them manually. The EC2 Image Builder service has matured to the point that it is a viable option for automating the process. While the support in CloudFormation and CDK is still a bit lacking, it is usable and can be worked into an RFDK app to build the required AMIs for the worker fleet on the fly.


		This construct creates all the infrastructure required by Image Builder to install Deadline onto an existing AMI and then create a WorkerInstanceFleet that uses it. The configuration for the image creation is as simple as providing the Deadline version, ID of the parent AMI, and the OS. An image version also needs to be supplied. One strategy to use with versioning your image is to start with version `1.0.0` and bump the version if you need to change any of the input parameters, such as changin the parent AMI or Deadline version. Images for different OSes can be versioned separately.

		CDK does not have L2 constructs for Image Builder yet, so we are using the L1 constructs. The documentation for these in CDK is sparse, so referring to the [AWS CloudFormation user guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_ImageBuilder.html) will provide more details.

	1. Manually terminate the worker instances from the console before performing your redeployment.
	1. Manually terminate the worker instances from the console after performing your redeployment.


		#### AMI Storage

		When performing an update to or deletion of the DeadlineImage constuct, any AMI that was created by a previous deployment of the construct will not be deleted. They are still available in EC2 and can be seen under `Images > AMIs` in the EC2 console or in the `My AMIs` section of the Launch instance wizard. You can continue to use them like any other AMI, or deregister them if you no longer require them. The cost of storing these AMIs depends on the size of the disk you took a snapshot of to create them, for EBS-backed AMIs you can find snapshot costs on their [EBS pricing page](https://aws.amazon.com/ebs/pricing/). For S3-backed AMI's, you'll pay for the storage fees of the data that needs to be stored based on (S3 pricing)[https://aws.amazon.com/s3/pricing/], whether you have an instance running or not.

	EC2 Image Builder pipelines with CDK isn't very well developed yet, so if you do decide to create one, the AWS Console is recommended.
	EC2 Image Builder pipelines with CDK is not developed at this time, so if you do decide to create one, the AWS Console is recommended.

feat(examples): added ec2 image builder example #292

feat(examples): added ec2 image builder example #292

Conversation

horsmand commented Jan 19, 2021 • edited Loading

ddneilson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yashda left a comment

Choose a reason for hiding this comment

horsmand commented Mar 3, 2021

jericht left a comment

Choose a reason for hiding this comment

horsmand commented Jan 19, 2021 •

edited

Loading