Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): support CloudFormation simplified resource import #29087

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
0f8965f
feat(cli): support ImportExstingResources
tmokmss Nov 19, 2023
e42494e
update aws-sdk, but I'm not sure if it's allowed
tmokmss Nov 19, 2023
ca9cd25
Update cli.integtest.ts
tmokmss Nov 19, 2023
e803106
fix integ
tmokmss Nov 20, 2023
a2fa8b4
also update aws-sdk
tmokmss Nov 20, 2023
d66b937
Update deployments.ts
tmokmss Nov 20, 2023
7e04e54
possible refactor to remove dead code
tmokmss Nov 20, 2023
30c1e50
revert refactor; might do this in another PR
tmokmss Nov 20, 2023
86f6ebc
Merge branch 'main' into allow_import_resources
tmokmss Nov 20, 2023
9229d92
Merge branch 'main' into allow_import_resources
tmokmss Dec 7, 2023
3a2a739
Update yarn.lock
tmokmss Dec 7, 2023
2a6aae5
Update README.md
tmokmss Dec 13, 2023
07fa306
Update cli.integtest.ts
tmokmss Dec 13, 2023
526daed
Update README.md
tmokmss Dec 13, 2023
d5f41e6
Update deploy-stack.test.ts
tmokmss Dec 13, 2023
2abaeba
Update deploy-stack.test.ts
tmokmss Dec 13, 2023
3cc542d
reject only when method=direct && importExistingResources==true
tmokmss Dec 14, 2023
e812eaa
Update packages/aws-cdk/README.md
tmokmss Dec 26, 2023
6e8813c
Update packages/aws-cdk/README.md
tmokmss Dec 26, 2023
dd75942
Update packages/aws-cdk/README.md
tmokmss Dec 26, 2023
8b5ea7e
Merge branch 'main' into allow_import_resources
tmokmss Dec 27, 2023
552e0eb
Merge branch 'main' into allow_import_resources
tmokmss Jan 21, 2024
8803a58
Merge branch 'main' into allow_import_resources
tmokmss Feb 3, 2024
1051e8f
Merge branch 'main' into allow_import_resources
tmokmss Feb 13, 2024
af43d96
Merge branch 'main' into allow_import_resources
SankyRed Apr 9, 2024
c8be1f5
merge
tmokmss Sep 17, 2024
8eb1549
Update packages/aws-cdk/README.md
tmokmss Sep 20, 2024
293a61f
Apply suggestions from code review
tmokmss Sep 20, 2024
ced4fb1
Update README.md
tmokmss Sep 20, 2024
5b1db10
add missing periods in docs
tmokmss Oct 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -470,6 +470,65 @@ integTest(
}),
);

integTest('deploy with import-existing-resources true', withDefaultFixture(async (fixture) => {
const stackArn = await fixture.cdkDeploy('test-2', {
options: ['--no-execute', '--import-existing-resources'],
captureStderr: false,
});
// verify that we only deployed a single stack (there's a single ARN in the output)
expect(stackArn.split('\n').length).toEqual(1);

const response = await fixture.aws.cloudFormation.send(new DescribeStacksCommand({
StackName: stackArn,
}));
expect(response.Stacks?.[0].StackStatus).toEqual('REVIEW_IN_PROGRESS');

// verify a change set was successfully created
// Here, we do not test whether a resource is actually imported, because that is a CloudFormation feature, not a CDK feature.
const changeSetResponse = await fixture.aws.cloudFormation.send(new ListChangeSetsCommand({
StackName: stackArn,
}));
const changeSets = changeSetResponse.Summaries || [];
expect(changeSets.length).toEqual(1);
expect(changeSets[0].Status).toEqual('CREATE_COMPLETE');
expect(changeSets[0].ImportExistingResources).toEqual(true);
}));

integTest('deploy without import-existing-resources', withDefaultFixture(async (fixture) => {
const stackArn = await fixture.cdkDeploy('test-2', {
options: ['--no-execute'],
captureStderr: false,
});
// verify that we only deployed a single stack (there's a single ARN in the output)
expect(stackArn.split('\n').length).toEqual(1);

const response = await fixture.aws.cloudFormation.send(new DescribeStacksCommand({
StackName: stackArn,
}));
expect(response.Stacks?.[0].StackStatus).toEqual('REVIEW_IN_PROGRESS');

// verify a change set was successfully created and ImportExistingResources = false
const changeSetResponse = await fixture.aws.cloudFormation.send(new ListChangeSetsCommand({
StackName: stackArn,
}));
const changeSets = changeSetResponse.Summaries || [];
expect(changeSets.length).toEqual(1);
expect(changeSets[0].Status).toEqual('CREATE_COMPLETE');
expect(changeSets[0].ImportExistingResources).toEqual(false);
}));

integTest('deploy with method=direct and import-existing-resources fails', withDefaultFixture(async (fixture) => {
const stackName = 'iam-test';
await expect(fixture.cdkDeploy(stackName, {
options: ['--import-existing-resources', '--method=direct'],
})).rejects.toThrow('exited with error');

// Ensure stack was not deployed
await expect(fixture.aws.cloudFormation.send(new DescribeStacksCommand({
StackName: fixture.fullStackName(stackName),
}))).rejects.toThrow('does not exist');
}));

integTest(
'update to stack in ROLLBACK_COMPLETE state will delete stack and create a new one',
withDefaultFixture(async (fixture) => {
Expand Down
38 changes: 38 additions & 0 deletions packages/aws-cdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,39 @@ $ cdk deploy --method=prepare-change-set --change-set-name MyChangeSetName
For more control over when stack changes are deployed, the CDK can generate a
CloudFormation change set but not execute it.

#### Import existing resources

You can pass the `--import-existing-resources` flag to the `deploy` command:

```console
$ cdk deploy --import-existing-resources
```

Automatically import resources in your CDK application which represent
unmanaged resources in your account.
Reduces the manual effort of import operations and avoids
deployment failures due to naming conflicts with unmanaged resources in your account.

Use `--method=prepare-change-set` flag to review which resources are imported or not before deploying
You can inspect the change set created by CDK from the management console or other external tools.

```console
$ cdk deploy --import-existing-resources --method=prepare-change-set
```

Use the `--exclusively` flag to enable this feature for a specific stack

```console
$ cdk deploy --import-existing-resources --exclusively StackName
```

Only resources that have custom names can be imported using `--import-existing-resources`
For more information, see [name type](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-name.html).
To import resources that do not accept custom names, such as EC2 instances,
use the `cdk import` instead.
Visit [Bringing existing resources into CloudFormation management](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html)
for more details.

#### Ignore No Stacks

You may have an app with multiple environments, e.g., dev and prod. When starting
Expand Down Expand Up @@ -579,6 +612,11 @@ To import an existing resource to a CDK stack, follow the following steps:
5. When `cdk import` reports success, the resource is managed by CDK. Any subsequent
changes in the construct configuration will be reflected on the resource.

NOTE: You can also import existing resources by passing `--import-existing-resources` to `cdk deploy`.
This parameter only works for resources that support custom physical names,
such as S3 bucket, DynamoDB table, etc...
For more information, see [Request Parameters](https://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_CreateChangeSet.html#API_CreateChangeSet_RequestParameters).

#### Limitations

This feature currently has the following limitations:
Expand Down
13 changes: 11 additions & 2 deletions packages/aws-cdk/lib/api/deploy-stack.ts
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,13 @@ export interface ChangeSetDeploymentMethod {
* If not provided, a name will be generated automatically.
*/
readonly changeSetName?: string;

/**
* Indicates if the change set imports resources that already exist.
*
* @default false
*/
readonly importExistingResources?: boolean;
}

export async function deployStack(options: DeployStackOptions): Promise<DeployStackResult> {
Expand Down Expand Up @@ -374,7 +381,8 @@ class FullCloudFormationDeployment {
private async changeSetDeployment(deploymentMethod: ChangeSetDeploymentMethod): Promise<DeployStackResult> {
const changeSetName = deploymentMethod.changeSetName ?? 'cdk-deploy-change-set';
const execute = deploymentMethod.execute ?? true;
const changeSetDescription = await this.createChangeSet(changeSetName, execute);
const importExistingResources = deploymentMethod.importExistingResources ?? false;
const changeSetDescription = await this.createChangeSet(changeSetName, execute, importExistingResources);
await this.updateTerminationProtection();

if (changeSetHasNoChanges(changeSetDescription)) {
Expand Down Expand Up @@ -405,7 +413,7 @@ class FullCloudFormationDeployment {
return this.executeChangeSet(changeSetDescription);
}

private async createChangeSet(changeSetName: string, willExecute: boolean) {
private async createChangeSet(changeSetName: string, willExecute: boolean, importExistingResources: boolean) {
await this.cleanupOldChangeset(changeSetName);

debug(`Attempting to create ChangeSet with name ${changeSetName} to ${this.verb} stack ${this.stackName}`);
Expand All @@ -417,6 +425,7 @@ class FullCloudFormationDeployment {
ResourcesToImport: this.options.resourcesToImport,
Description: `CDK Changeset for execution ${this.uuid}`,
ClientToken: `create${this.uuid}`,
ImportExistingResources: importExistingResources,
Comment on lines 425 to +428
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if they specify both ResourcesToImport and ImportExistingResources?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it will not work, but the spec says nothing about such restriction, so I would like to keep it as-is.

Actually users cannot set both ImportExistingResources and ResourcesToImport from the CDK CLI, because the former is only used in cdk deploy, and cdk deploy does not have any command argument to directly set ResourcesToImport.

Copy link
Contributor

@HBobertz HBobertz Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec isn't really the source of truth we all wish it was so it's not uncommon we enforce logical things the spec does not. However in this specific instance I believe it is correct not to enforce these arguments be exclusive because there is a difference between the resources supported by importExistingResources and ResourcesToImport. Auto-import is a subset of resources supported by ResourcesToImport so it is entirely possible to have both. It is entirely possible then someone could build a command which fails, but I believe this won't really be a common occurence and this is fine to leave as is

...this.commonPrepareOptions(),
}).promise();

Expand Down
10 changes: 7 additions & 3 deletions packages/aws-cdk/lib/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ async function parseCommandLineArguments(args: string[]) {
requiresArg: true,
desc: 'How to perform the deployment. Direct is a bit faster but lacks progress information',
})
.option('import-existing-resources', { type: 'boolean', desc: 'Indicates if the stack set imports resources that already exist.', default: false })
.option('force', { alias: 'f', type: 'boolean', desc: 'Always deploy stack even if templates are identical', default: false })
.option('parameters', { type: 'array', desc: 'Additional parameters passed to CloudFormation at deploy time (STACK:KEY=VALUE)', nargs: 1, requiresArg: true, default: {} })
.option('outputs-file', { type: 'string', alias: 'O', desc: 'Path to file where stack outputs will be written as JSON', requiresArg: true })
Expand Down Expand Up @@ -576,16 +577,19 @@ export async function exec(args: string[], synthesizer?: Synthesizer): Promise<n
if (args.changeSetName) {
throw new Error('--change-set-name cannot be used with method=direct');
}
if (args.importExistingResources) {
throw new Error('--import-existing-resources cannot be enabled with method=direct');
}
deploymentMethod = { method: 'direct' };
break;
case 'change-set':
deploymentMethod = { method: 'change-set', execute: true, changeSetName: args.changeSetName };
deploymentMethod = { method: 'change-set', execute: true, changeSetName: args.changeSetName, importExistingResources: args.importExistingResources };
break;
case 'prepare-change-set':
deploymentMethod = { method: 'change-set', execute: false, changeSetName: args.changeSetName };
deploymentMethod = { method: 'change-set', execute: false, changeSetName: args.changeSetName, importExistingResources: args.importExistingResources };
break;
case undefined:
deploymentMethod = { method: 'change-set', execute: args.execute ?? true, changeSetName: args.changeSetName };
deploymentMethod = { method: 'change-set', execute: args.execute ?? true, changeSetName: args.changeSetName, importExistingResources: args.importExistingResources };
break;
}

Expand Down
35 changes: 35 additions & 0 deletions packages/aws-cdk/test/api/deploy-stack.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -880,6 +880,41 @@ describe('disable rollback', () => {

});

describe('import-existing-resources', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a CLI integ test that creates a single resource and then imports it with this new flag?

Copy link
Contributor Author

@tmokmss tmokmss Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@comcalvi
Thanks, but I don't fully agree with implementing such test on CDK side. Because:

  1. Such test would more or less complicate the test code, not noly the success case, but also ensuring there is no orphaned resource when test failed halfway through. This possibly increases maintenance cost of the cli integ test code.
  2. It is not CDK's responsibility to ensure that a resource is successfully imported when the flag is set. It is CFn's responsibility to guarantee the functionality. CDK should just make sure the flag is set properly (as tested in the current code.)

What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not CDK's responsibility to ensure that a resource is successfully imported when the flag is set. It is CFn's responsibility to guarantee the functionality. CDK should just make sure the flag is set properly (as tested in the current code.)

This is true, however ultimately any test involving any cloudformation api usage (such as most CDK deploy tests) also falls under that same umbrella. Even though we don't own cloudformation nor are we responsible to fix any issue blocking our service in cloudformation, it does allow us to report the impact our customer's impact to appropriate CFN teams. Regardless of all that though it's already a just line in the sand we've crossed ages ago, so I believe it's appropriate to add that test

Such test would more or less complicate the test code, not noly the success case, but also ensuring there is no orphaned resource when test failed halfway through. This possibly increases maintenance cost of the cli integ test code.

This is true, it would be far from the most complex integ test we've written but any integ tests which are verifying deployments tend to require some setup/cleanup so it does get a bit complex. Orphaning is technically possible but as long as you set up a finally to manually cleanup you manually created, and the stack rolls back, it should be g2g. You can look at other tests in CDK Deploy or perhaps CDK Migrate for some ideas of how we've done this in the past.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is ultimately just a very long winded way of me saying I believe the test is appropriate 😅.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HBobertz Thanks, hmmm but what is the actual value to adding the test? As long as we set the importExistingResources flag, it should result in a successfull deployment, unless CFn is not working properly. If there happens to be the case when CDK fails although CFn is working as expected, that is when we should add a new test for the very case. At this moment, however, I think such test does not add any reliability to the behavior of CDK; it just adds unneessary complexity to the cli integ test code base.

test('by default, import-existing-resources is disabled', async () => {
// WHEN
await deployStack({
...standardDeployStackArguments(),
deploymentMethod: {
method: 'change-set',
},
});

// THEN
expect(cfnMocks.createChangeSet).toHaveBeenCalledTimes(1);
expect(cfnMocks.createChangeSet).toHaveBeenCalledWith(expect.objectContaining({
ImportExistingResources: false,
}));
});

test('import-existing-resources is enabled', async () => {
// WHEN
await deployStack({
...standardDeployStackArguments(),
deploymentMethod: {
method: 'change-set',
importExistingResources: true,
},
});

// THEN
expect(cfnMocks.createChangeSet).toHaveBeenCalledTimes(1);
expect(cfnMocks.createChangeSet).toHaveBeenCalledWith(expect.objectContaining({
ImportExistingResources: true,
}));
});
});

/**
* Set up the mocks so that it looks like the stack exists to start with
*
Expand Down