cloudformation deploy should handle throttling.

[fhewitt]

Bug description
We have a cloudformation which describe our DynamoDB tables: 6 tables, all with their global secondary indexes, scaling policies, scalable target + a shared access and scalling roles.

Trying to deploy the configuration we hit an error:
Unable to create alarms for scaling policy ReadAutoScalingPolicy-Session-Primary-dev due to reason: Rate exceeded (Service: AmazonCloudWatch; Status Code: 400; Error Code: Throttling; Request ID: 1f0a35b7-d542-11e7-83ca-0d260ce2adf0)

CloudFormation should handle itself the throttling automatically, just taking longer before completion.

Workaround
What is the temporary solution for now?

I've though about pushing table by table, handling myself the throttling, but this raised multiple issues:

1. References: I don't want to define a different scaling role for each table, but the scaling role must be defined in the same CloudFormation definition to be referenced.
2. This explode the number of stacks, increasing solution complexity and creating a maintenance issue.
3. This broke ability to ensure an atomic update, if the thirds stack fail, we got stuck with a broken state, partially updated and that's really not good.

Therefore, it doesn't seem to be a correct solution.

[joguSD]

The built-in retry handler should automatically retry Throttling exceptions. By default I believe it retries up to five times. Does this error always happen when you attempt to deploy?

@sanathkr Any ideas?

[fhewitt]

Yes. it was.

Version informaton, if this can be useful:
aws-cli/1.13.0 Python/2.7.9 Windows/8 botocore/1.8.3

The targetted region was ca-central-1.

Used command was:
aws cloudformation deploy --template-file dynamodb.cf.json --stack-name projectname-databases-dev --capabilities CAPABILITY_NAMED_IAM

Not sure if new version since the bug report.

[fhewitt]

Additional information. It seems to be specifically on the Scaling Policy insertion. We have a table with 5 GSI, all scalable, which imply 10 scaling policies and apparently it's too much for CloudFormation. So, much of the deploy seems to handle throttle, except the scaling policy insertion.

Updated the CLI today with
aws-cli/1.14.8 Python/2.7.9 Windows/8 botocore/1.8.12
And still have the issue.

[sanathkr]

So the deploy command doesn't do anything but call CloudFormation Create/ExecuteChangeSet. It doesn't orchestrate and call individual AWS services. So the throttling error you are getting, I think is directly received by CloudFormation service (and not the CLI). Is this correct?

Usually you would fix throttling issues by raising a support ticket with the concerned service (CloudWatch in this case) to increase the API throttling limits for your account.

Does this make sense?

[fhewitt]

Makes senses. It still an issue that should be fixed, but doesn't fall in the aws-cli. Therefore, I close the issue, thanks!