s3 sync: basic progress display

[leezu]

aws s3 sync uses carriage return when printing progress to erase the line and start over, providing a live progress experience. This works fine in interactive session, but not for logging (see eg #2598).

Passing --no-progress disables all progress reporting.

Unfortunately, with --no-progress nothing is logged. It would good to have a --non-interactive-progress logging option, which simply logs each object transferred without the carriage return based live logging.

[justnance]

@leezu - Thank you for reporting this behavior. We've labeled this as enhancement pending further review.

[queglay]

Could I also suggest with the live logging progress output, considering the ability to change the rate of updates? Being able to do this can be usefull in a logging scenario if its not too verbose (I'm using gitlab ci). If we can alter the rate, that would be awesome. Perhaps --progress-update-rate could accept a number in seconds? or if set to 0 or -1, it would only produce a single line of output, equivalent to the suggestion of --non-interactive-output

Currently with gitlab ci in ansible running s3 sync in a shell will fill a full 500Kb log with download progress by default.

[Twipped]

To add to this, I'd really like to be able to do --only-show-errors --show-progress. This command line script I'm writing would benefit with progress output for the recursive copy operation, but I don't need the output of every file as it is fetched.

[tim-finnigan]

When you run aws s3 sync it prints a new line for each object it is uploading or downloading and shows the progress during that:

@leezu can you provide any more context for your use case? I’m trying to think of scenarios where a user would want to see each object’s transfer status but not the overall progress. I can see why someone might want the inverse of that as @Twipped suggested.

[leezu]

Hi @tim-finnigan, the usecase is to support non-interactive environments, such as AWS CloudWatch or CI tools such as Jenkins or Github Actions. Those environments don't handle carriage return well.

[tim-finnigan]

Thanks @leezu for following up. I see the reasoning behind what you're saying. To sum up this feature request, I think there is an ask for two new optional parameters that might be something like:

• --hide-progress : show object transfer statuses as they are logged but hide ongoing progress
• --show-progress : show ongoing progress but not individual object transfer statuses

[leezu]

Thank you @tim-finnigan. Yes, your proposal would address the issue.

[Scouter9001]

any update on this?

[tim-finnigan]

Hi @Scouter9001 we don't have any updates at this time, but we encourage everyone interested in this feature to 👍 the original issue to indicate their support. That will help the team better prioritize this relative to other feature requests.

[Keats]

There's the same issue with aws s3 cp. Eg I want to see the progress of the download of a huge file from kubectl but the carriage return causes issue. Having an option that just adds a new line for each update instead would be appreciated.