Failure to build stack #1854

DougalW · 2023-09-11T02:24:55Z

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

19.5.0

Amplify CLI Version

12.4.0

What operating system are you using?

MacOS 12.6.3

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

None

Describe the bug

We have a schema.graphql with 54 tables and 31 enums. There are 40 many:many relations across the tables.

This is by no means a massive schema, and a good size for a moderately complex SaaS app of this type. I would expect that Amplify could handle this size schema.

We are experiencing four types of errors:

the EMFILE error
ConnectionStack errors
"Limit on the number of resources in a single stack operation exceeded" errors
Parameter count > than 200 allowed max
Failure modes occur with both amplify push and push with force directive.

In detail:

Issue 1. - Have been getting this error after the schema table count went over around 40:
Raised in support Case ID: 13727852561

I tried to push a GraphQL schema update to Amplify with the following command: amplify push --force --allow-destructive-graphql-schema-updates
It keeps failing with the following error:

Deployment status
Deployment failed 04/09/2023, 19:45:10: EMFILE: too many open files, open '/tmp/amplify-07edc1f5-8083-426b-9391-a7ce62209b51/amplify/backend/api/ClimateDisclosure/build/states/01/resolvers/MetricsDataSet.Assets.res.vtl' .

Then rolls back my push. This issue has been raised before here: aws-amplify/amplify-studio#414
and appeared to be resolved by increasing the ulimit on the Job container.

Issue 2. - This error is intermittent.

E.g. when creating a totally new stack:

Embedded stack arn:aws:cloudformation:ap-southeast-2:253256353881:stack/amplify-climatedisclosure-deleteme-130810-apiClimateDisclosure-YGEXC2DFOWA3/b42ad8e0-5040-11ee-864c-026a6ec0826a was not successfully created: The following resource(s) failed to create: [ConnectionStack].

...and...

Name: ConnectionStack (AWS::CloudFormation::Stack), Event Type: create, Reason: Embedded stack arn:aws:cloudformation:ap-southeast-2:253256353881:stack/amplify-climatedisclosure-deleteme-130810-apiClimateDisclosure-YGEXC2D-ConnectionStack-U6F63L520F4L/6fad0390-5041-11ee-8807-0ab22e2dcf8a was not successfully created: Limit on the number of resources in a single stack operation exceeded, IsCustomResource: false

Issue 3. - Have been getting this error after the schema table count went over 50:

amplify push --force --allow-destructive-graphql-schema-updates kept failing:

Reason: Embedded stack arn:aws:cloudformation:ap-southeast-2:253256353881:stack/amplify-climatedisclosure-dev-112157-apiClimateDisclosure-KQIOIHG9OUB-ConnectionStack-YZNJE7CIQID9/2a714690-e293-11ed-8126-0a25b4139496 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: Limit on the number of resources in a single stack operation exceeded


Issue 4. - Have been getting this error after the schema table count went over around 50:

"ResourceType": "AWS::CloudFormation::Stack",
            "Timestamp": "2023-09-06T00:16:36.119000+00:00",
            "ResourceStatus": "UPDATE_FAILED",
            "ResourceStatusReason": "Template format error: Parameter count 214 is greater than max allowed 200",
            "ResourceProperties": "{\"TemplateURL\":\"https://s3.ap-southeast-2.amazonaws.com/amplify-climatedisclosure-dev-112157-deployment/amplify-appsync-files/53f8e3dffea7a051e934983e6a45f6b8f8619c5d/stacks/ConnectionStack.json\",\"Parameters
\":

We also see other errors like this:

🛑 Cannot iteratively rollback as the following step does not contain a previousMetaKey: {"status":"DEPLOYING"}

Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

IterativeRollbackError: Cannot iteratively rollback as the following step does not contain a previousMetaKey: {"status":"DEPLOYING"}
    at runIterativeRollback (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/iterative-deployment/iterative-rollback.js:44:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/push-resources.js:127:9)
    at async /snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/extensions/amplify-helpers/push-resources.js:137:16
    at async Promise.all (index 0)
    at async providersPush (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/extensions/amplify-helpers/push-resources.js:133:5)
    at async AmplifyToolkit.pushResources (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/extensions/amplify-helpers/push-resources.js:107:13)
    at async Object.executeAmplifyCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:194:9)
    at async executePluginModuleCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:139:5)
    at async executeCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:37:9)
    at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:121:5)

Session Identifier: 8ef8a9c1-ec0f-41f0-b8b3-f767ff85312e

and this:

Deploying iterative update 2 of 2 into schemadev environment. This will take a few minutes. ⠼
Deploying api ClimateDisclosure [ =--------------------------------------- ] 2/107
	GraphQLAPIDefaultApiKey215A6D… AWS::AppSync::ApiKey           UPDATE_IN_PROGRESS             Wed Sep 06 2023 11:23:06…     
	GraphQLAPITransformerSchema3C… AWS::AppSync::GraphQLSchema    UPDATE_COMPLETE                Wed Sep 06 2023 11:23:41…     
🛑 table name should be passed

Resolution: Please report this issue at https://github.com/aws-amplify/amplify-cli/issues and include the project identifier from: 'amplify diagnose --send-report'
Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

UnknownNodeJSFault: table name should be passed
    at nodeErrorToAmplifyException (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/amplify-exception-handler.js:144:12)
    at handleException (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/amplify-exception-handler.js:24:28)
    at process.<anonymous> (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:52:93)
    at process.emit (node:events:513:28)
    at process.processEmit [as emit] (/snapshot/amplify-cli/build/node_modules/signal-exit/index.js:199:34)
    at process._fatalException (node:internal/process/execution:149:25)

table name should be passed
AssertionError [ERR_ASSERTION]: table name should be passed
    at DeploymentManager.getTableStatus (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/iterative-deployment/deployment-manager.js:325:28)
    at invokeFunc (/snapshot/amplify-cli/build/node_modules/lodash.throttle/index.js:160:19)

Each of these errors has increased in frequency as the table count increased.

These seem to be caused by Amplify not correctly creating CloudFormation stacks inside CloudFormation resource limits. Some of these have been raised in these issues:
-aws-amplify/amplify-cli#9762 (comment) (@ Straubulous closed that issue on 4 May 2022 but we are still getting both errors with the latest version of the CLI).

aws-amplify/amplify-studio#414
but there is no real fix.
I've seen this suggested, but tbh it feels like a hack and doesn't work around the resource limit issue:
https://docs.amplify.aws/cli/graphql/override/#place-appsync-resolvers-in-custom-named-stacks

Please advise what to do - this makes Amplify impractical for any moderately sophisticated schema. The resource issue also makes Amplify impractical because we will very quickly hit the resource limit per-accountID.

Expected behavior

Amplify should successfully parse the schema and generated needed backend resources. It currently parses the schema just fine, but fails with the resources.

Reproduction steps

Reproducing requires trying to run a build of our project - reach out to me for access to this.

Project Identifier

5c7bc147366fc1111fe82b901e8f35f8

Log output

# Put your logs below this line

Additional information

No response

Before submitting, please confirm:

I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
I have removed any sensitive information from my code snippets and submission.

The text was updated successfully, but these errors were encountered:

DougalW · 2023-09-11T02:25:42Z

Correction: 55 tables

DougalW · 2023-09-11T05:55:21Z

I reduced the number of tables to 52, pushed again, and it failed:
🛑 The following resources failed to deploy:
Resource Name: ConnectionStack (AWS::CloudFormation::Stack)
Event Type: update
Reason: Embedded stack arn:aws:cloudformation:ap-southeast-2:253256353881:stack/amplify-climatedisclosure-schemadev-184417-apiClimateDisclosure-1JJNPZ-ConnectionStack-ZGUHWWI1V3D6/0e470f50-3e5e-11ee-a46f-02d236a3bb2e was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: Limit on the number of resources in a single stack operation exceeded
URL: https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/arn%3Aaws%3Acloudformation%3Aap-southeast-2%3A253256353881%3Astack%2Famplify-climatedisclosure-schemadev-184417-apiClimateDisclosure-1JJNPZWB03U59%2Fe0d43120-3e5c-11ee-a569-02e6440f255c/events

🛑 Resource is not in the state stackUpdateComplete
Name: ConnectionStack (AWS::CloudFormation::Stack), Event Type: update, Reason: Embedded stack arn:aws:cloudformation:ap-southeast-2:253256353881:stack/amplify-climatedisclosure-schemadev-184417-apiClimateDisclosure-1JJNPZ-ConnectionStack-ZGUHWWI1V3D6/0e470f50-3e5e-11ee-a46f-02d236a3bb2e was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: Limit on the number of resources in a single stack operation exceeded, IsCustomResource: false

Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

Session Identifier: 38b21643-f8e0-44f5-ab7d-a3c7acddf7de

I can email the report if needed

DougalW · 2023-09-11T06:38:56Z

We also tried the recommended approach in:

https://docs.amplify.aws/cli/graphql/override/#place-appsync-resolvers-in-custom-named-stacks

But this failed with Reason: Circular dependency between resources:

same as in #32

ykethan · 2023-09-11T17:48:43Z

Hey @DougalW, 👋 thanks for raising this! I'm going to transfer this over to our API repository for better assistance 🙂.

dpilch · 2023-09-12T14:33:57Z

Here is a possible solution to unblock the custom named stack workaround.

Consider this example:

Models A, B, and C
Model A hasOne B (dependency)
Model B hasOne C (dependency)
Each model creates a resolver (GetAResolver, GetBResolver, GetCResolver).
Two custom stacks (StackFoo, StackBar).

"StackMapping": {
  "GetAResolver": "StackFoo",
  "GetBResolver": "StackBar",
  "GetCResolver": "StackFoo" // creates circular dependency
}

With this configuration I have created a circular dependency.

StackFoo depends on StackBar via A to B relationship
StackBar depends on StackFoo via B to C relationship
Changing GetCResolver to StackBar would remove the circular dependency.

chrisbonifacio · 2023-09-19T19:19:16Z

Hi 👋 Closing this as we have not heard back from you. If you are still experiencing this issue and in need of assistance, please feel free to comment and provide us with any information previously requested by our team members so we can re-open this issue and be better able to assist you.

Thank you!

DougalW · 2023-09-19T20:40:30Z

I wasn't aware anyone had requested additional information! Please point me to where this has been requested.

@dpilich provided a suggestion, which we looked at but this will only solve the connection stack issue and won't alleviate the resource issue (Amplify creates too many cloudformation resources). This approach is also very fragile and manual.

We have instead been pursuing an approach to reduce the dimensionality of our data model, which is very far from ideal. I will report back once we have better data.

What I was hoping to see is a response to reduce the number of resources Amplify creates, built in to Amplify itself. It's not reasonable to expect your customers to keep making these fundamental tweaks just to get something to deploy successfully.

I don't think you should close this issue - it's clearly still there and experienced by others, and something you need to fix in Amplify otherwise your customers will stop using Amplify for anything more than toy apps.

DougalW · 2023-09-20T02:27:22Z

I also just saw this:
#1859
Which is related to the issue we're facing so I don't understand why you have closed my issue.

chrisbonifacio · 2023-09-20T16:29:19Z

Hi @DougalW 👋 I closed the issue simply because we hadn't received a response within a week but we will always re-open an issue once we do. I understand the provided solution is only a workaround and does not result in a good developer experience. I will reach out to the team with the feedback provided so that we can come to a better solution.

BBopanna · 2023-12-31T11:33:44Z

+1 to the whole paradigm of amplify+appsync falling apart around 50+ tables - AWS team how do you expect enterprise grade apps to scale with these kinds of limitations ? and just to be backed by RUSH to close valid scenarios. Work arounds are just cumbersome and painful, organizations question this tech stack offered by you Amplify!! Pathetic to say the least Amplify team!

INCREASE THE DAMN LIMITS AWS AMPLIFY TEAM !

Amirbahal · 2024-01-12T21:56:27Z

Have you tried removing the amplify folder in your project and running an amplify pull again? :)

chirpavel · 2024-08-09T09:52:38Z

I have the same error while deploy:

2024-08-09T06:15:18.144Z [INFO]: 🛑 The following resources failed to deploy:
2024-08-09T06:15:18.153Z [INFO]: Resource Name: BranchPublicInfo (AWS::CloudFormation::Stack)
Event Type: update
Reason: Embedded stack arn:aws:cloudformation:eu-central-1:675795832684:stack/amplify-XXXXX-main-XXXXX-apiXXXXX-R5I7DCBEDRZM-BranchPublicInfo-RO7LC76AQ9ON/97908130-5300-11ef-b9c9-06184733758b was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: Limit on the number of resources in a single stack operation exceeded

URL: https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/arn%3Aaws%3Acloudformation%3Aeu-central-1%3A675795832684%3Astack%2Famplify-XXXXX-main-XXXXX-apiXXXXX-R5I7DCBEDRZM%2Ffb232ec0-5182-11ef-976b-0257b97cb813/events

🛑 Resource is not in the state stackUpdateComplete

Name: BranchPublicInfo (AWS::CloudFormation::Stack), Event Type: update, Reason: Embedded stack arn:aws:cloudformation:eu-central-1:675795832684:stack/amplify-XXXXX-main-XXXXX-apiXXXXX-R5I7DCBEDRZM-BranchPublicInfo-RO7LC76AQ9ON/97908130-5300-11ef-b9c9-06184733758b was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: Limit on the number of resources in a single stack operation exceeded, IsCustomResource: false

And I can't understand what limit exactly.
There are no additional details in CloudFormation, the same text and status UPDATE_FAILED.
I don't understand what I violated, so I can think about it, improve it and fix it.
I have very few tables, about 10 and 3 functions. That's it, we can't move anywhere again, a dead end

chirpavel · 2024-08-09T10:35:31Z

How can I understand which limit I have violated?
What is the limit value?
How can I view or manually count the number of elements that fall under this limit?

There should be some hints in the logs.
This will reduce the number of requests and tasks, people will be able to do something themselves.

DougalW added the pending-triage label Sep 11, 2023

This was referenced Sep 11, 2023

API CustomResources.json 500 resources per template limit - question about splitting it up using nested stacks #658

Closed

Focus Areas for Q4 2023 aws-amplify/amplify-js#11833

Closed

ykethan transferred this issue from aws-amplify/amplify-cli Sep 11, 2023

ykethan added the transferred label Sep 11, 2023

dpilch added the theme:appsync-cloudformation-resource-limits label Sep 12, 2023

dpilch removed the pending-triage label Sep 12, 2023

AnilMaktala added the pending-response label Sep 13, 2023

chrisbonifacio closed this as not planned Won't fix, can't repro, duplicate, stale Sep 19, 2023

github-actions bot removed the pending-response label Sep 19, 2023

chrisbonifacio reopened this Sep 20, 2023

chrisbonifacio self-assigned this Sep 20, 2023

ykethan mentioned this issue Nov 29, 2023

EMFILE: too many open files, open #2100

Closed

2 tasks

dpilch added the bug Something isn't working label Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failure to build stack #1854

Failure to build stack #1854

DougalW commented Sep 11, 2023

DougalW commented Sep 11, 2023

DougalW commented Sep 11, 2023

DougalW commented Sep 11, 2023 •

edited

Loading

ykethan commented Sep 11, 2023

dpilch commented Sep 12, 2023

chrisbonifacio commented Sep 19, 2023

DougalW commented Sep 19, 2023 •

edited

Loading

DougalW commented Sep 20, 2023

chrisbonifacio commented Sep 20, 2023 •

edited

Loading

BBopanna commented Dec 31, 2023 •

edited

Loading

Amirbahal commented Jan 12, 2024

chirpavel commented Aug 9, 2024 •

edited

Loading

chirpavel commented Aug 9, 2024

Failure to build stack #1854

Failure to build stack #1854

Comments

DougalW commented Sep 11, 2023

How did you install the Amplify CLI?

If applicable, what version of Node.js are you using?

Amplify CLI Version

What operating system are you using?

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

Describe the bug

Expected behavior

Reproduction steps

Project Identifier

Log output

Additional information

Before submitting, please confirm:

DougalW commented Sep 11, 2023

DougalW commented Sep 11, 2023

DougalW commented Sep 11, 2023 • edited Loading

ykethan commented Sep 11, 2023

dpilch commented Sep 12, 2023

chrisbonifacio commented Sep 19, 2023

DougalW commented Sep 19, 2023 • edited Loading

DougalW commented Sep 20, 2023

chrisbonifacio commented Sep 20, 2023 • edited Loading

BBopanna commented Dec 31, 2023 • edited Loading

Amirbahal commented Jan 12, 2024

chirpavel commented Aug 9, 2024 • edited Loading

chirpavel commented Aug 9, 2024

DougalW commented Sep 11, 2023 •

edited

Loading

DougalW commented Sep 19, 2023 •

edited

Loading

chrisbonifacio commented Sep 20, 2023 •

edited

Loading

BBopanna commented Dec 31, 2023 •

edited

Loading

chirpavel commented Aug 9, 2024 •

edited

Loading