Limit on the number of resources in a single stack operation exceeded (V2 Transformer Migration)

[sacrampton]

Before opening, please confirm:

• I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
• I have searched for duplicate or closed issues.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v17.1.0

Amplify CLI Version

7.6.19

What operating system are you using?

Linux

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No

Amplify Categories

Not applicable

Amplify Commands

push

Describe the bug

I am trying to migrate a large system from V1 to V2 Transformer. I have upgraded ElasticSearch 6.2 to OpenSearch 1.1. I have performed the "amplify migrate api" and stripped away all the custom resolvers we have.

The amplify push gets a long way through the deployment, but ultimately fails with the message "Limit on the number of resources in a single stack operation exceeded".

None of my individual stacks are above 500 resources, so I figure this is something to do with concurrent resources. So I checked the current limits on my account, but don't see any way to change that (have opened up a ticket with the CloudFormation Team):

$aws cloudformation describe-account-limits
ACCOUNTLIMITS   StackLimit      2000
ACCOUNTLIMITS   StackOutputsLimit       200
ACCOUNTLIMITS   ConcurrentResourcesLimit        8500

Not sure how to proceed here.

Expected behavior

Push does not fail

Reproduction steps

See above

GraphQL schema(s)

# Put schemas below this line

Log output

# Put your logs below this line

Additional information

No response

[daniellorenzin]

We have just hit this too.

We moved from V1 to V2 and have a large GraphQL deployment at the moment. ( 45 models ). It almost gets to the end of the deployment and fails on a stack with: Limit on the number of resources in a single stack operation exceeded

We are using the CDK output functionality so I was able to get a count and it seems that the deployment is attempting to deploy 74 CloudFormation templates with 2604 resources over all of them.

[SebSchwartz]

We also have a large graphql schema (55 tables) :/

Didn't try it yet (will do it later this day) but I saw on the doc something that could help:
-> https://docs.amplify.aws/cli/graphql/override/#place-appsync-resolvers-in-custom-named-stacks

[edwardfoyle]

Hi all, we have a mechanism for placing resources that the graphql transformer generates into custom named stacks: https://docs.amplify.aws/cli/graphql/override/#place-appsync-resolvers-in-custom-named-stacks. This mechanism can be used to keep a single stack from accumulating too many resources.

That being said, this is something that we should handle in a more automated way.

[sacrampton]

Hi @edwardfoyle - in the table below you'll see that I've split stacks such that they are all less than the 500 maximum. With the V2 transformer however the number of resources, particularly in the connectionStack* is significantly higher. This means I have a total number of resources across all stacks is now 3,528.

Amplify submits this to CloudFormation as a nested stack. I opened a ticket with AWS CloudFormation support and they came back with this feedback.

"...There also exists a limit on the total number of resources for nested stack which is 2500 per chain and if it goes beyond that, we may encounter the aforementioned error message. If this is the case, you can also split the stacks and use cross stack references [5] - It exports an output from one stack and import it in another and you can access resources in a different stack, as long as they are in the same account and AWS Region...."

It would appear from this that with the significant expansion of resources with the V2 transformer that we are now exceeding (by a significant amount) the 2500 resource limit per nested stack - even if the individual stacks are less than 500 (as is my case).

Can Amplify be configured to support / enable cross stack references to move past this limitation. With the significant increase in resources caused by V2 it would appear that any production user with a significant database is going to hit this limitation pretty quickly.

Because it stops a production upgrade I'd put this in as a bug rather than a feature request as it is currently classified.

<style> </style>

No. Resources	Stack
52	Table 1
44	Table 2
32	Table 3
31	Table 4
32	Table 5
40	Table 6
44	Table 7
48	Table 8
44	Table 9
44	Table 10
36	Table 11
6	Connection Stack
435	Connection Stack 2
438	Connection Stack 3
402	Connection Stack 4
32	Table 12
32	Table 13
36	Table 14
36	Table 15
44	Table 16
48	Table 17
32	Table 18
36	Table 19
32	Table 20
32	Table 21
32	Table 22
32	Table 23
44	Table 24
44	Table 25
48	Table 26
48	Table 27
36	Table 28
44	Table 29
44	Table 30
32	Table 31
32	Table 32
36	Table 33
31	Table 34
32	Table 35
44	Table 36
32	Table 37
32	Table 38
76	Table 39
48	Table 40
44	Table 41
32	Table 42
31	Table 43
32	Table 44
36	Table 45
44	Table 46
44	Table 47
258	Searchable Stack
32	Table 48
32	Table 49
44	Table 50
44	Table 51

[daniellorenzin]

@sacrampton I was about to comment with the same feedback.

I had a feeling that there was a 2500 resource limit in a chain, but it is good to have it confirmed. Because I have a CDK export version ( which is just json template files), I'm going to try to split it up for a deployment.

But I agree, the changes in V2 would require Amplify to create seperate deployments for the model parts to remain under the 2500 limit.

[SebSchwartz]

Hi all, we have a mechanism for placing resources that the graphql transformer generates into custom named stacks: https://docs.amplify.aws/cli/graphql/override/#place-appsync-resolvers-in-custom-named-stacks. This mechanism can be used to keep a single stack from accumulating too many resources.

That being said, this is something that we should handle in a more automated way.

I've tried the solution but I got this error: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 9b7c445a-1b63-4c9d-9164-9c6e327e7141; Proxy: null)

I put something like this in transform.conf.json:

"StackMapping": {
    "GetInvoiceResolver": "InvoiceResolverCustomStack",
    "ListInvoiceResolver": "InvoiceResolverCustomStack",
    "SyncInvoiceResolver": "InvoiceResolverCustomStack",
    "CreateInvoiceResolver": "InvoiceResolverCustomStack",
    "UpdateInvoiceResolver": "InvoiceResolverCustomStack",
    "DeleteInvoiceResolver": "InvoiceResolverCustomStack",
    "InvoicebankAccountResolver": "InvoiceResolverCustomStack",
...

[sacrampton]

Hi @SebSchwartz - thanks for the input. I'm actually doing exactly that with the connectionStack.json (see table in the above comment (#9762 (comment)) - splitting the 1,281 resources in connectionStack.json into 4 files with transform.conf.json - and as can be seen, that is working fine. That however is not this issue.

The issue is that the TOTAL number of resources across all stacks can't exceed 2,500 resources. To deal with this, Amplify needs to "...split the stacks and use cross stack references..."

[sacrampton]

Hi @edwardfoyle - just wondering if you have any feedback on this one. Its going to prevent anyone with a significant database migrating to the V2 platform (of course in my opinion). I know we are stopped in our tracks. So any viable workaround also appreciated.

[sacrampton]

Hi @undefobj / @josefaidt - appreciate your updates to status

[AustinAmoruso]

I am also hitting this issue. Is creating the different stacks going to blow away my data in Dynamodb and the GSI? I'm really concerned with this migration in general.

[sacrampton]

Hi @undefobj / @josefaidt - just checking back to see if there are any updates on this? This is obviously a killer when migrating any significant system. Appreciate any feedback you can provide.

[josefaidt]

Hey @sacrampton apologies for the delay! I've added this to the team bug bash board to be prioritized as part of our weekly bug bash 🙂

[sacrampton]

Hi @josefaidt - feedback much appreciated

[sacrampton]

Hi @josefaidt - any update on this - we are dead in the water with this and I expect anyone with a significant database to migrate will be facing the same issue.

[undefobj]

@sacrampton we're still researching this, however to be transparent our initial investigation shows the resolution will take some time. That being said it is at the top of our priority list nonetheless.

[sacrampton]

Thanks @undefobj - the transparency helps a lot - thanks

[akshbhu]

Hi @sacrampton

We are currently investigating options to reduce resources in larger schemas.Can you share your schema or mail your schema at amplify-cli@amazon.com after redacting any secret info from it.

[oe-bayram]

Hi @akshbhu

we also have a large schema and are facing the same issue.

I have shared our schema with you via the mentioned mail address. Thanks.

[sacrampton]

Hi @akshbhu - I have emailed our schema to you. Thanks

[AustinAmoruso]

@akshbhu I resent my schema it essentially breaks/creates all kinds of v2 migration issues.

[lseemann]

We're bumping into this now, too. Am emailing our schema. Our limit appears to be getting affected by a number of custom resolvers, too, of which we have about 40.

[akshbhu]

Hi @lseemann

If you are blocked on 500 resource limit , You can follow this to divide few resources in other cfn files.
https://docs.amplify.aws/cli/graphql/override/#place-appsync-resolvers-in-custom-named-stacks

[lseemann]

Thanks. I've been trying that all afternoon but it’s failing. For example I've added the following to the StackMapping object in transform.conf.json: ` "CreateNotificationResolver": "MyCustomStack", ` and the build will start (progress!) but it complains that "Only one resolver is allowed per field." ``` CREATE_FAILED CreateNotificationResolver AWS::AppSync::Resolver Wed Apr 20 2022 14:06:42 GMT-0500 (Central Daylight Time) Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: c5497f55-c047-4cdf-bc1e-4f757197cf6a; Proxy: null) CREATE_FAILED amplify-meltzerhellrungporta-lukeooo-141419-apimhgraphqlapi-74SQKAWXV2BP-MyCustomStack-LW5HI7UHTCMJ AWS::CloudFormation::Stack Wed Apr 20 2022 14:06:43 GMT-0500 (Central Daylight Time) The following resource(s) failed to create: [CreateNotificationResolver]. ```

[akshbhu]

Do you also same resolver defined in customResources.json file?

I think this resolver is defined in two stacks in your case , one in MyCustomStack.json and second in CustomResources.json . Can you confirm ?

Check this issue , this seems related: #9284

[akshbhu]

Hi @sacrampton

Do you have V2 schema which is giving the limit exceeded error ?

I think you have previously emailed complied schemas and v1 schema

[sacrampton]

Hi @akshbhu - my bad, I'll resend

[lseemann]

Do you also same resolver defined in customResources.json file?

I think this resolver is defined in two stacks in your case , one in MyCustomStack.json and second in CustomResources.json . Can you confirm ?

We haven't made any edits to CustomResources.json, and there is no MyCustomStack.json as far as I can tell.

[akshbhu]

Thanks. I've been trying that all afternoon but it’s failing. For example I've added the following to the StackMapping object in transform.conf.json: "CreateNotificationResolver": "MyCustomStack", and the build will start (progress!) but it complains that "Only one resolver is allowed per field." CREATE_FAILED CreateNotificationResolver AWS::AppSync::Resolver Wed Apr 20 2022 14:06:42 GMT-0500 (Central Daylight Time) Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: c5497f55-c047-4cdf-bc1e-4f757197cf6a; Proxy: null) CREATE_FAILED amplify-meltzerhellrungporta-lukeooo-141419-apimhgraphqlapi-74SQKAWXV2BP-MyCustomStack-LW5HI7UHTCMJ AWS::CloudFormation::Stack Wed Apr 20 2022 14:06:43 GMT-0500 (Central Daylight Time) The following resource(s) failed to create: [CreateNotificationResolver].

When you add this entry in transform.conf.json file , this creates another cfn file with value as name of the file .
Can you check if this file getting generated after running amplify api gql-compile

[lseemann]

Ah, yes, I see it in /build/stacks now.

FWIW, I tried again without the Create*Resolver and instead added some assorted relational resolvers, and this appears to have worked: It brought us under the 500 cap, and it built successfully. So knock on wood, I may be unblocked.

[InnovateWithEric]

This is resolved in build 8.1.0 with de-duplication of resolver functions.

[oe-bayram]

I'm still getting this error message after upgrading to 8.1.0
The deduplication of resolver functions seems to be not enough for the limit exceed.
Are there any other options than defining custom stacks?
Because adding each resolver to a custom stack also causes this error.

[mikeswain]

We struggled with this for a while (2500 items in stack limit). Our solution to get our build fixed was to add an override.ts to slightly optimise the way individual table resources are created. We moved the 2 policy resources per table (DynamoDBAccess and modelIamRolePolicy into the modelRole itself as inline policies - this reduced our resource count enough to bring it under the 2500 limit (we had about 50 tables and thus reducing by 100 avoided the error). Not a total solution but it got us out of a jam