How to use metadata to automatically compare the checksum

[stefansundin]

If the S3 object that you want to verify has a sha256sum metadata entry (or tag), then s3sha256sum will automatically compare that to the actual contents which it has just hashed. It will print OK or FAILED.

Example:

# Generate a ~10MB file with random data:
$ openssl rand -out 10mb.txt -base64 $(( 10 * 2**20 * 3/4 ))

# Calculate the SHA256 checksum: (your checksum will be different)
$ sha256sum 10mb.txt
e7b61bb2963b33e6acd645e20137b1fc35ba5978253cdbb7304c2359f726a6f1  10mb.txt

# Upload the file to S3 with the checksum as metadata:
$ aws s3 cp --metadata sha256sum=e7b61bb2963b33e6acd645e20137b1fc35ba5978253cdbb7304c2359f726a6f1 10mb.txt s3://bucketname/10mb.txt

# Use s3sha256sum to verify the integrity of the data on S3:
$ s3sha256sum s3://bucketname/10mb.txt
e7b61bb2963b33e6acd645e20137b1fc35ba5978253cdbb7304c2359f726a6f1  s3://bucketname/10mb.txt

OK (matches object metadata)

It's as simple as that.

You can use shrimp to easily attach the checksum (it has lots of other great features too). If there's a SHA256SUMS file in the working directory, it will try to look up the file you are uploading and automatically attach the checksum in the metadata.

The example is very similar:

# Generate a ~10MB file with random data:
$ openssl rand -out 10mb.txt -base64 $(( 10 * 2**20 * 3/4 ))

# Calculate the SHA256 checksum and save it in SHA256SUMS:
$ sha256sum 10mb.txt >> SHA256SUMS

# Upload the file to S3 using shrimp:
$ shrimp 10mb.txt s3://bucketname/10mb.txt

# Use s3sha256sum to verify the integrity of the data on S3:
$ s3sha256sum s3://bucketname/10mb.txt
e7b61bb2963b33e6acd645e20137b1fc35ba5978253cdbb7304c2359f726a6f1  s3://bucketname/10mb.txt

OK (matches object metadata)

[stefansundin]

With the launch of Additional Checksum Algorithms feature released in February 2022, s3sha256sum may not seem as important. With this feature, there is no longer a need to download the object to verify its consistency, which is especially great for objects stored in Glacier. Since AWS haven't published a good tool to verify these objects, I built s3verify to perform this task.

So which feature should you use? If you use shrimp to upload your files then I recommend that you use both --checksum-algorithm and --compute-checksum!

The biggest reason for this is because, for multi-part uploads, the checksum algorithm feature builds a hash of hashes, it is not possible to reconstruct a normal full-length checksum of the file using this information. It is handy to have the regular checksum value in case you want to be able to compare the checksum with a checksum collected by someone else without having the file itself handy. With the checksum algorithm feature, changing the part size in a multi-part upload will change the final checksum computed for the completed object.