aws s3 sync: display and logging issues incorrect use of LF vs CRLF

[hansw96]

AWS version: aws-cli/1.11.122 Python/2.7.9 Windows/7 botocore/1.5.85
OS: Windows 7, and Windows Server 2012

Issue 1:
When running a sync in a command prompt (aws s3 sync C:\somedir\ s3://somebucket), the "completed" message does not get wiped out, but instead fills up the screen:

Completed 9.8 MiB/ ~64.2 MiB (1.2 MiB/s) with ~543 file(s) remaining (calculating...)
Completed 9.8 MiB/ ~64.2 MiB (1.2 MiB/s) with ~543 file(s) remaining (calculating...)
Completed 9.8 MiB/ ~64.2 MiB (1.2 MiB/s) with ~543 file(s) remaining (calculating...)
Completed 9.9 MiB/ ~64.2 MiB (1.2 MiB/s) with ~543 file(s) remaining (calculating...)

In issue #2598 you mention "we use carriage returns to erase the line and start over.". This does not work on Windows with the above AWS CLI version.

Issue 2:
When piping the output to a text file, multiple lines get bunched together.
aws s3 sync C:\somedir\ s3://somebucket > C:\temp\aws.log

When opening the text file in Notepad it looks like this:

Completed 15.1 KiB/574.4 KiB (46.1 KiB/s) with 7 file(s) remainingCompleted 15.2 KiB/574.4 KiB (46.5 KiB/s) with 7 file(s) remainingupload: C:\somedir\somefile to s3://somebucket/somedir/somefile

My guess is that the unix line ending (LF) is used for the "completed" messages, and the correct windows line ending (CRLF) for the "upload" messages.

Issue 3:
It would be nice to have an option to not log the "Completed" messages in the logfile, and only the actual uploaded files. There are many requests for this option: see issue #519

Thank you for your help!

[hansw96]

Re; Issue 1
Probably the line is too long and wraps around, and the trick of CR to start at the beginning of the line does not work.

[jamesls]

1. I've also confirmed this issue. Looks like \r is essentially ignored if the line wraps. This happens on both windows and linux:

import sys, time
s = time.sleep


sys.stdout.write("1 asdfasfdsfasfasdf")
sys.stdout.write("asdfsdfgafasdfasdfasdf" * 20)
sys.stdout.flush()
s(1)
sys.stdout.write("\r")
sys.stdout.write("2 asdfasfdsfasfasdf")
sys.stdout.write("asdfsdfgafasdfasdfasdf" * 20)
sys.stdout.flush()
s(1)
sys.stdout.write("\r")
sys.stdout.write("3 asdfasfdsfasfasdf")
sys.stdout.write("asdfsdfgafasdfasdfasdf" * 20)
sys.stdout.flush()
s(1)
sys.stdout.write("\r")
sys.stdout.write("4 asdfasfdsfasfasdf")
sys.stdout.write("asdfsdfgafasdfasdfasdf" * 20)
sys.stdout.flush()
s(1)
print("\n")

I'm not sure what we'd be able to do about that (that's just the behavior of \r) other than try not to line wrap and truncate long lines if necessary.

2. That output isn't really meant to be piped to other files. You could use the --debug log if you wanted detailed output that was designed to be written to a file.

3. I agree. Please upvote the feature request. It helps us prioritize which features to work on.

Let me know if you have any more questions.

[hansw96]

1. Truncating long lines would solve the problem!!

That output isn't really meant to be piped to other files...

I disagree, it is very common wanting to log the results of your backup... for monitoring purposes!

PS. The label "question" is really not appropriate. I did not ask a question how to do something. I reported bugs. So please apply a "feature-request", or "bug" label.

[justnance]

@hansw96 - Thanks for your feedback. I am relabeling this as a feature request for item:

3. I agree. Please upvote the feature request. It helps us prioritize which features to work on.

[wizofaus]

Upvote!

[jrd-nonlilly]

I disagree, it is very common wanting to log the results of your backup... for monitoring purposes!

I have to agree: I just stumbled upon this behavior as I'm trying to log aws s3 sync output. Very unfriendly.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[francisreyes-tfs]

why was this closed? this isn't a feature request, it's a bug.