s3 sync: display progress?

[olalonde]

When using aws s3 sync s3://first-bucket s3://second-bucket, I'm seeing a bunch of logs but there's no way to tell how many objects are left or any sort of progress / ETA, which is really annoying since I have a lot of objects and have no idea whether my sync will be done in 10 minutes or 10 days.

Here are the logs that I'm seeing:

copy: s3://bucket1/...somefile... to s3://bucket2/...somefile....
copy: s3://bucket1/...somefile... to s3://bucket2/...somefile....
copy: s3://bucket1/...somefile... to s3://bucket2/...somefile....
copy: s3://bucket1/...somefile... to s3://bucket2/...somefile....
copy: s3://bucket1/...somefile... to s3://bucket2/...somefile....
etc.

Is there a flag to enable a progress bar or some kind of ETA? Otherwise, that would be a feature request.

[JordonPhillips]

We do display progress. It will show up like so:

Completed 9.8 MiB/ ~64.2 MiB (1.2 MiB/s) with ~543 file(s) remaining (calculating...)

Now due to the nature of how S3 works we can't get the full number of files or the full size ahead of time, so we use the tildes and "calculating" to indicate that until we have full information.

[olalonde]

I'm not seeing this with aws-cli/1.11.55 Python/2.7.12 Linux/4.9.13-moby botocore/1.5.18. Was it implemented recently?

$ docker run -it garland/aws-cli-docker sh -c "aws --version"
aws-cli/1.11.55 Python/2.7.12 Linux/4.9.13-moby botocore/1.5.18

[JordonPhillips]

You'll only see it when it's actually running, it won't show up in logs. This was added in 1.11.0

[olalonde]

Ah ok.. I'm guessing because it detects it's a non interactive shell.

[olalonde]

@JordonPhillips would it be possible to add a flag to show the progress even when stdout is not a tty?

[JordonPhillips]

So the reason why you won't see this in logs is that we use carriage returns to erase the line and start over. This lets us provide live progress without polluting output with thousands of lines of incremental progress text. Looking at the logs afterwards you won't see any of the progress updates because they all get wiped. If you're looking at it live then you should be seeing them regardless of the output destination. For example:

$ aws s3 cp s3://bucket/large-object . &> out.log & tail -f out.log

That runs the transfer in the background, redirecting all output to a log file. While that's going on it tails the log file so updates are shown. You will see the file change in real time, but at the end the file will be without any progress messages.

[olalonde]

I was tailing the docker logs and didn't see any progress message like you are saying.

$ docker run -d --name=s3 garland/aws-cli-docker sh -c "aws s3 cp s3://bucket/large-object ."
$ docker logs -f s3

[JordonPhillips]

Not sure what's going on, could you provide some debug logs?

[olalonde]

Maybe try start a sync inside docker and docker logs -f containername ?

[JordonPhillips]

If you append --debug to your command it'll print out other information that will help debugging this issue.

[JordonPhillips]

Closing due to inactivity. Please feel free to reopen once you've got those debug logs.

[here]

To add some detail to Jordan's comments,
This will appear at the bottom of the interactive terminal while the command is running.
The indicator does not display when using --dryrun as of aws-cli/1.15.71

[livni]

Is there a way to see only the "Completed" progress line, rather than logging each object?
AFAICT --quiet also mutes the "Completed" progress line

[jesse-houghton]

For those having issues with not being able to see completed status while tailing docker logs (i.e. not in an interactive shell) this should help

docker logs <container> --follow | sed --expression 's/\r/\n/g'

Should give you some log lines that look like

Completed 366.6 GiB/~366.8GiB (14.6MiB) with ~1000 file(s) remaining (calculating...)

Note: you may need to adjust the sed statement when using different versions of sed.