Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not check md5 for downloading kms objects #1026

Merged
merged 2 commits into from
Nov 24, 2014

Conversation

kyleknap
Copy link
Contributor

Fixes #1014

The issue was that we rely on ETAG to perform MD5 checksums. However, you cannot reliably do that if it is encrypted by KMS.

I did some testing on all of the s3 encryption testing to see how MD5's checkout with the ETAG returned here is the summary

# Note comparator.py is the object that I am getting for all three cases.
$ md5 comparator.py
MD5 (comparator.py) = 7d62dba263afdd3f901128435ed146b7

# KMS encrypted
# Same md5 as unencrypted

$ aws s3api get-object --bucket mybucketfoo --key kmsencrypted temp.txt
{
    "AcceptRanges": "bytes", 
    "ContentType": "binary/octet-stream", 
    "LastModified": "Mon, 24 Nov 2014 20:27:51 GMT", 
    "ContentLength": 6194, 
    "ETag": "\"d12b33cb96e431763224615d904dc600\"", 
    "ServerSideEncryption": "aws:kms", 
    "SSEKMSKeyId": "arn:aws:kms:us-east-1:934212987125:key/c32756f3-bc40-4b64-951f-4785e2423b7d", 
    "Metadata": {}
}


# SSEC
# Different md5 as unencrypted

$ aws s3api get-object --bucket mybucketfoo --key ssecencrypted temp.txt --sse-customer-key fileb://sse_c.key --sse-customer-algorithm AES256
{
    "AcceptRanges": "bytes", 
    "ContentType": "binary/octet-stream", 
    "LastModified": "Mon, 24 Nov 2014 20:33:42 GMT", 
    "ContentLength": 6194, 
    "SSECustomerAlgorithm": "AES256", 
    "ETag": "\"3eb52a5967bba8ba6ab7035f94e59b14\"", 
    "SSECustomerKeyMD5": "F13HSk8VjVCCWFhcFJzuSg==", 
    "Metadata": {}
}


# SSE
# Same md5 as unencrypted

{
    "AcceptRanges": "bytes", 
    "ContentType": "application/x-python", 
    "LastModified": "Mon, 24 Nov 2014 20:39:05 GMT", 
    "ContentLength": 6194, 
    "ETag": "\"7d62dba263afdd3f901128435ed146b7\"", 
    "ServerSideEncryption": "AES256", 
    "Metadata": {}
}

Based on the results, the only encryption technique you can use the ETAG for is SSE. All other ones return ETAG's that you cannot use to compare to the MD5. Note that this object was not multipart uploaded/downloaded.

cc @jamesls @danielgtaylor

@kyleknap
Copy link
Contributor Author

Also when we decide to add support sse-c for the aws s3 command set. We will need to incorporate them in this md5 check. However note at them moment, you cannot download sse-c encrypted objects with aws s3 because it requires the encryption key to get the object.

@jamesls
Copy link
Member

jamesls commented Nov 24, 2014

:shipit: Looks good. Thanks for investigating and fixing.

kyleknap added a commit that referenced this pull request Nov 24, 2014
Do not check md5 for downloading kms objects
@kyleknap kyleknap merged commit 5291553 into aws:develop Nov 24, 2014
@quiver
Copy link
Contributor

quiver commented Nov 25, 2014

awesome @kyleknap !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

s3 cp cannot download kms-encrypted object
3 participants