s3 sync, s3 cp no space left on device errors handling

[vinogradovs]

aws s3 sync and aws s3 cp to local disk don't handle free space left on device errors

aws s3 sync puts IOError: [Errno 28] No space left on device errors but still tries to continue sync
aws s3 cp doesn't give any error, tries to continue coping and returns 0 exit code at the end

[jamesls]

Can you confirm what version of the CLI you're using (aws --version)?

[vinogradovs]

1.7.44,
Hm, it seems strange, but now exit code is 1 if the whole bucket is synced. I will check this on other machine later.
The other problem is that if there are a lot of files to sync, it is needed to wait until utility finishes to try syncing of all of them, and syncing can be going for hours or in some cases even days.

[jamesls]

Not sure I understand. Could you provide a script that demonstrates the issue. Here's what I tried. I'm seeing 0 rc's if the bucket is synced:

/tmp/foo $ aws s3 sync . s3://bucket/
upload: ./foo.txt to s3://bucket/foo.txt
/tmp/foo $ echo $?
0
/tmp/foo $ aws s3 sync . s3://bucket/
/tmp/foo $ echo $?
0

Note that if an error occurred during the syncing process a non zero will happen but we still keep trying to sync the remaining files for you. Here's some additional docs on the return codes used by the CLI: http://docs.aws.amazon.com/cli/latest/topic/return-codes.html#cli-aws-help-return-codes You can also see these docs by running aws help return-codes.

[chumaitoe]

I can confirm that aws s3 cp onto a device with insufficient space results in a partial copy and return code 0. aws-cli/1.7.44

[vinogradovs]

Execuse me for long waiting,

simple script:
aws s3 sync s3://mybucket/ /mnt/mount_point_with_no_space_left/test/
echo $#

The second one with cp
aws s3 cp --recursive s3://mybucket/ /mnt/mount_point_with_no_space_left/test/
echo $#
In both cases I get 0 exit code, but also I get errors like this:

download failed: s3://mybucket//test/setup.log to ../mnt/mount_point_with_no_space_left/test/setup.log Could not create directory s3://mybucket/ /mnt/mount_point_with_no_space_left/test/test5: [Errno 28] No space left on device: 's3://mybucket/ /mnt/mount_point_with_no_space_left/test/test5'
Completed 201 of 227 part(s) with 1 file(s) remaining

And if we have a bucket with billions of files we can wait for a very long time to get the result of 0 after several days, it seems more efficient to stop copy/sync in case of error with no space left.

[chumaitoe]

Did you get those errors on the console? I got nothing.

I think you mean "$?" where you wrote "$#" there.

[vinogradovs]

yes, It's a typo, I mean $?

[vinogradovs]

Strange, on other machine it works as expected, it seems the problem is local

[mtdowling]

Is this issue indeed local to your machine, or is there a reproducible test case you can provide that works on any/all machines? I'm trying to understand if this issue should remain open based on your last comment.

[chumaitoe]

In all my testing, copying a file onto a device with insufficient space (as opposed to no space) resulted in success return code with no complaint text - i.e., success behavior. So that seems like a test case to me..? Tried a couple of different machines, but not many - not sure what constitutes "different".

[alexaandru]

I also run into the "aws s3 cp" issue (not enough disk space, no error and exit code 0). Took me a while to figure out what the error was :) An error message would be very useful :) Cheers!

[jamesls]

Fixed via #1648.