aws s3 sync re-uploads files that are already in target location

[ciank]

Using 1.3.21 of aws cli on Windows 7

Appears that having a folder name which starts with another folder name at the same level causes aws cli to believe that the files in that directory do not exist, and therefore re-uploads them. This conclusion based on running the following (with and without comparator):

aws s3 sync "C:\abc" "s3://my_bucket_name/abc"

c:\abc contains some files and 2 folders "xyz" and "xyz2". The first time it runs successfully and I have verified that everything has been copied to the s3 bucket. On every subsequent run all the files in c:\abc\xyz are re-uploaded erroneously.

I was puzzling as to what was going on (including investigation of #599). There is a thread about this in the aws forum https://forums.aws.amazon.com/thread.jspa?threadID=146851&tstart=0

[aivarsi]

I have the same problem and I added some extra debug info in Comparator class to see what's going on.

            LOG.debug("SRC: %s", src_file.compare_key)
            LOG.debug("DST: %s", dest_file.compare_key)

For the first some files the keys would be completely different, e.g.:

2014-08-04 11:45:19,746 - MainThread - awscli.customizations.s3.comparator - DEBUG - SRC: 1/2.jpg
2014-08-04 11:45:19,746 - MainThread - awscli.customizations.s3.comparator - DEBUG - DST: 10004/galleries/195761/Chrysanthemum.jpg
....
2014-08-04 11:45:19,748 - MainThread - awscli.customizations.s3.comparator - DEBUG - SRC: 1/Toward_the_Sea.jpg
2014-08-04 11:45:19,749 - MainThread - awscli.customizations.s3.comparator - DEBUG - DST: 10004/galleries/195761/Chrysanthemum.jpg
...
2014-08-04 11:45:19,754 - MainThread - awscli.customizations.s3.comparator - DEBUG - SRC: 1/catitems/fisheye.jpg
2014-08-04 11:45:19,756 - MainThread - awscli.customizations.s3.comparator - DEBUG - DST: 10004/galleries/195761/Chrysanthemum.jpg

Notice that the destination compare_key is equal in all cases.

Then after it actually reaches this file, it straightens out and starts comparing correctly:

2014-08-04 11:45:19,916 - MainThread - awscli.customizations.s3.comparator - DEBUG - SRC: 10004/galleries/195761/Chrysanthemum.jpg
2014-08-04 11:45:19,919 - MainThread - awscli.customizations.s3.comparator - DEBUG - DST: 10004/galleries/195761/Chrysanthemum.jpg
2014-08-04 11:45:19,919 - MainThread - awscli.customizations.s3.comparator - DEBUG - SRC: 10004/galleries/195761/Tulips.jpg
2014-08-04 11:45:19,920 - MainThread - awscli.customizations.s3.comparator - DEBUG - DST: 10004/galleries/195761/Tulips.jpg
2014-08-04 11:45:19,924 - MainThread - awscli.customizations.s3.comparator - DEBUG - SRC: 10004/galleries/195761/e091814ce7d3411d24c0b1ee0f6_prev.jpg
2014-08-04 11:45:19,924 - MainThread - awscli.customizations.s3.comparator - DEBUG - DST: 10004/galleries/195761/e091814ce7d3411d24c0b1ee0f6_prev.jpg
...

But later it would get out of sync again and repeat the pattern.

Let me know if you need more info.

[aivarsi]

Thank you @kyleknap! Very appreciated.

[kyleknap]

Fixed by pull request: #865

[ciank]

Thanks