s3 sync: retry behavior; push last retry to bottom of queue

[buildbreakdo]

As understood, aws s3 sync behaviour retries N times then moves to the next file. Believe an improvement could be if the final retry was pushed to the end of the files to be processed queue.

Sitting in a hotel on a bad network, middle of an s3 sync, file failed to upload (network cut out), network came back as sync moved to the next file. Everything uploaded except that one file, had the final retry been at the bottom, would have no issues, looking to put the max amount of distance between error and final retry to create a window of time for potential self-resolution.

Additionally would be great if there was a rollup of what did not sync at the end, like X files uploaded; X files failed instead of scanning a list of -- in my case -- hundreds of console echos.

[stealthycoin]

So then you would be caught in an infinite loop as long as your network had cut out. Not acceptable for automated scripts using the cli. Unless I am misunderstanding your suggestion?

[buildbreakdo]

@stealthycoin Have 100 files, lets say default retry count is 3, at position 2 there is an error, current behaviour 3 retries occur back to back at position 2. Asking for the final retry (retry number three) to be pushed to the bottom of the sync queue after file 100 to create as large a window as possible for self-resolution of the error -- this would still be a final retry. Process is exactly the same except when the final retry for a file sync occurs. Would not be an infinite loop scenario.

[stealthycoin]

That would require a pretty big refactor, and would be very complicated to get right. I will mark this as a feature request.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[slavik57]

+1
Any news on this issue?
Is there a way to specify infinite retry?

[tim-finnigan]

Checking in as this issue hasn't received any activity in a long time. I didn't see this documentation on AWS CLI retries referenced yet so I'll link this here: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-retries.html

If the configurable retries don't address your use case then I believe this issue should be combined with #1092 as that relates to retries with s3 commands more broadly. Please let us know if there are any distinctions you'd like to make - otherwise I'll set this issue to auto-close.