Why not just use the ETag?

[stefansundin]

Because the ETags can only really be used for integrity checking for unencrypted objects (for encrypted objects, the ETag is probably still useful for Amazon who have access to the ciphertext). To add to that, it is hard to calculate the ETag for objects that resulted from a multipart upload. And we shouldn't have to pick between encrypting the data and being able to verify the integrity of the object.

So why go through all the trouble with the ETag when we can just attach a SHA256 checksum at the time of upload. Setting our own metadata means that we will rely on a value that we compute before the upload, and it's a value that we have full control over. We can pick any hash function that we want.

Anyway, this discussion is supposed to be about the ETags, so let's list some docs and existing work that may be interesting reading for anyone who asks themselves "why not use the ETag though?"

Official docs: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html

The entity tag represents a specific version of the object. The ETag reflects changes only to the contents of an object, not its metadata. The ETag may or may not be an MD5 digest of the object data. Whether or not it is depends on how the object was created and how it is encrypted as described below:

• Objects created through the AWS Management Console or by the PUT Object, POST Object, or Copy operation:
  • Objects encrypted by SSE-S3 or plaintext have ETags that are an MD5 digest of their data.
  • Objects encrypted by SSE-C or SSE-KMS have ETags that are not an MD5 digest of their object data.
• Objects created by either the Multipart Upload or Part Copy operation have ETags that are not MD5 digests, regardless of the method of encryption.

Type: String

Good explanation: https://teppen.io/2018/06/23/aws_s3_etags/

Program: https://github.com/peak/s3hash

aws-sdk-go-v2 PR: aws/aws-sdk-go-v2#1146

Please discuss more below!