aws s3 sync re-uploads files that are already in target location (Snowball Edge, Mac)

[genfhk]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone through the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
After syncing to Snowball Edge, we repeat the sync command to check that all files have been transferred. This finds additional files to transfer. However, based on the results of an aws s3 ls manifest run before the second sync (and a comparison of md5 checksums of source and destination files), the additional files had already been transferred. Sometimes we can run aws s3 sync multiple times and it will repeatedly sync the same subset of 'un-transferred' files.

SDK version number
AWS-CLI version: aws-cli/2.4.28 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off

Platform/OS/Hardware/Device
Mac OS 10.15.07

To Reproduce (observed behavior)
Uncertain - does not occur in all syncs

Expected behavior
Sync should only sync a given source file once if sync was successful

Logs/output
Get full traceback and error logs by adding --debug to the command.

2022-03-25 10:23:29,062 - MainThread - awscli.customizations.s3.filters - DEBUG - =/Volumes/XXXX-048/Video/744651/data/PreservationMasters/myh_744651_v01_pm.mkv.qctools.xml.gz final filtered status, should_include: True
2022-03-25 10:23:29,062 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: /Volumes/XXXX-048/Video/744651/data/PreservationMasters/myh_744651_v01_pm.mkv.qctools.xml.gz -> bucket-name/MPS-snowball/XXXX-048/Video/744651/data/PreservationMasters/myh_744651_v01_pm.mkv.qctools.xml.gz, file does not exist at destination

Additional context
Similar behavior reiererenced in closed issue: #843

[kdaily]

Hi @genfhk,

Thanks for writing in and sorry to hear you're having an issue. One thing to note is that Snowball Edge only supports AWS CLI version 1.16.14:

https://docs.aws.amazon.com/snowball/latest/developer-guide/using-adapter-cli.html

Currently, Snowball Edge devices support only version 1.16.14 and earlier of the AWS CLI. You can download and install this version of the AWS CLI from GitHub. Use the following procedure to perform this task.

I'm wondering if you still encounter the issue when using that supported version.

[tim-finnigan]

Since we haven't heard back in over a couple months I'm going to resolve this issue. As mentioned above, this may have to do with the Snowball Edge service pinning to an old version of the CLI. But please note that we have an open feature feature request here regarding that: aws/aws-sdk#216.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.