syncing many large files between s3 buckets occasionally results in a copy failed due to a CompleteMultipartUpload operation: Access Denied

[hguturu]

I have been recently syncing (move-ing) several TB of data between two different S3 buckets. Each file in the "subfolder" is several 10s of GB (e.g. 50GB). I noticed that occasionally in the sync 1 file fails with the following error:

copy failed: {s3 source bucket location} to {s3 target bucket location} An error occurred (AccessDenied) when calling the CompleteMultipartUpload operation: Access Denied

The error is hard to reproduce, but I have observed it twice so far in the past two days. For reproducing, I have tried to delete the failed file from the target bucket and re-run the sync and it succeeds (I have never deleted everything and re-run the full sync since that is very time consuming). I am also unclear on what is actually failing since when I look at the byte count or verify the checksum they both match with the origin bucket.

The command I have been using is aws s3 sync {s3 source bucket location} to {s3 target bucket location}.

The awscli I have been been using is aws s3 --version:

• aws-cli/1.15.80 Python/2.7.14 Linux/4.14.94-89.73.amzn2.x86_64 botocore/1.10.79

This seems related to #635 since we also get our credentials from the instance metadata.

[JordonPhillips]

Hmm the refresh should be fairly aggressive, triggering like 15 minutes before the credentials expire. Do you manually revoke sessions at all? If it happens on a given file, does it repeatedly happen on that file or does it just happen the once?

[hguturu]

No I don't manually revoke or obtain any credentials since I let the cli handle that automatically with the instance meta data. AFAIK it doesn't happen to a specific file, just an arbitrary file from the set and unfortunately not reproduce-able (if I delete and re-sync that failed file). I haven't tried deleting and re-syncing the entire location since its several terabytes of data.

When I said I saw it twice, it meant I noticed while syncing two different locations between two buckets.

[justnance]

@hguturu - Thank you for your feedback. I've been working on a similar issue under #4251 and #4035. Please review and advise if the issue you are experiencing matches the same symptoms.

Thanks.

[hguturu]

@justnance Its unclear to me its the same issue as #4251 since that seems like a reproducible issue (i.e. always failing with large files). But it does share some similarity with my sync - although my sync was across two buckets in the same account, the two buckets were secured by the different KMS keys, like in #4251. My symptom was much more like #635, where the error only occurs occasionally (and is hard to reproduce).

[kdaily]

@hguturu, has this reoccurred for you? If not, I'm going to close it for now. Please feel free to reopen if it does. Thank you!

[hguturu]

Sounds good. Will re-open if it comes up.