aws/request: Treat ConcurrentUpdateException as retryable

[radeksimko]

I have received the following error when trying to create two autoscaling targets at the same time:

ConcurrentUpdateException: You already have a pending update to an Auto Scaling resource.
			status code: 400, request id: 12bcfbd4-a8ed-11e7-ab84-f5312ac9a28e

[xibz]

@radeksimko - I don't think this should be retryable. The only real concern I have is that operations would be nondeterministic. What is your use case? Is it for the describe calls?

[radeksimko]

To be fair it would be best if the API was just capable of handling more than 1 request at a time 😜
Since that's not the reality there are IMO two ways of dealing with this:

1. Implement mutex on the API client side
2. Retry - which is what I'm proposing

Admittedly the first option may be a bit more "correct" - is that what you're suggesting?

[xibz]

@radeksimko - Yea, the first option I think is the better way of going about this. Is there any issues with the first approach?

[radeksimko]

My only issue is that it feels a bit dirty to do this. My feeling is that API maintainer (AWS in this case) is pushing the responsibility for dealing with the internal problem up to the user and it should be either hidden in the API (best case) or SDK.

As far as I'm aware the SDK doesn't have any concept of mutexes for any particular methods? Would there be any interest in adding those to the SDK or do you expect users of the SDK to implement this themselves?

[jasdel]

I don't think it would be good for the SDK to add murexes around these API operations. Mostly because it would only partially workaround the issue. In a distributed system environment a mutex within a single application instance won't be much help if two or more instances try to make the change close enough to each other.

If this is an issue with the service restrict any resource change to only a one at a time. I think this is feedback we should forward on to the service. Especially in the distributed environment, this issue should be handled either by the service allowing multiple concurrent changes, or the client application infrastructure limiting the changes to only a single one at a time.

I'm curious if there is anyway to identify that a resource change is pending. This might be a good place to have a waiter that will wait for a pending resource.

[xibz]

@jasdel - Agreed, I was thinking client side above the SDK as clarification. I did not see a waiter for this particular operation, unfortunately.

[radeksimko]

I'm curious if there is anyway to identify that a resource change is pending. This might be a good place to have a waiter that will wait for a pending resource.

That would be awesome.

In a distributed system environment a mutex within a single application instance won't be much help if two or more instances try to make the change close enough to each other.

Indeed - which is actually argument for retry 🙂 although I agree with @xibz that it's not a nice solution, to blindly retry.

[adriantodorov]

Hey @radeksimko , thank you for this. Our Jenkins pipeline is hanging because of this bug. is there a timeline for a new release for the plugin for provider "aws"?

[jasdel]

Thanks for creating this PR @radeksimko, but I don't think the SDK should retry for this API call. Since this is a API resource limit of the service API a waiter might be a better option, or for the application making the API call to manage the concurrent resource updates.

The AWS EC2 Forums is the best place to reach out to the Autoscaling and request the service support a higher rate of resource updates.