How to resume hashing

[stefansundin]

If you interrupt the program with Ctrl-C, it will print the internal state of the hash function and print a command that will resume the process from that position in the object. This can be useful if it is prohibitively expensive (or time consuming) to restart a failed hashing from the beginning.

Here's an example:

$ s3sha256sum -verbose s3://bucketname/large-object.txt
Getting s3://bucketname/large-object.txt
^C
Interrupt received.
Aborted after 20.1 MiB (21052955 bytes).

To resume hashing from this position, run:
s3sha256sum -verbose -resume c2hhA6XKyU+eveeG9pM2A3OuJnpzBMCw4naMNqnXnbxumAehR2hpNHFDbVFYRFdKOXY2VDVtRjQ3NGZjRXVHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQT4b s3://bucketname/large-object.txt

Note: This value is the internal state of the hash function. It may not be compatible across versions of s3sha256sum or across Go versions.

As the output instructs, you can resume the hashing by re-running the command with the -resume switch. The program will use a range request to avoid downloading previous data.

$ s3sha256sum -verbose -resume c2hhA6XKyU+eveeG9pM2A3OuJnpzBMCw4naMNqnXnbxumAehR2hpNHFDbVFYRFdKOXY2VDVtRjQ3NGZjRXVHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQT4b s3://bucketname/large-object.txt
Resuming from position 20.1 MiB (21052955 bytes).

Getting s3://bucketname/large-object.txt

14d2c2bfee4a537bc803e063961d938ddd3904cd7db2306a6e79749a9aa5cd0c  s3://bucketname/large-object.txt

OK (matches object metadata)

If you're curious about what's inside of the base64 data, have a look at the Go source code here: https://github.com/golang/go/blob/go1.17/src/crypto/sha256/sha256.go#L50-L57

As the warning suggests, please do not assume that the value will work across versions of s3sha256sum or versions of Go.

There's also paranoid mode for the extra paranoid. If you enable it, then s3sha256sum will continually output the state and resume command on an interval.

For example, to have it output the state every second, use -paranoid 1s:

$ s3sha256sum -verbose -paranoid 1s s3://bucketname/large-object.txt
Getting s3://bucketname/large-object.txt
To resume hashing from 1.3 MiB (1405891 bytes), run: s3sha256sum -verbose -paranoid 1s -resume c2hhA16mK2ntpC/MyleckvBGhM67KPb/jGIVS0i5BvV9P4BJUnpEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFXPD s3://bucketname/large-object.txt
To resume hashing from 16.0 MiB (16774083 bytes), run: s3sha256sum -verbose -paranoid 1s -resume c2hhAyv1EPF0xyAWW8Cjwhf3J72X+otz71K5qV5ygflnv2mIQlFUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//PD s3://bucketname/large-object.txt
To resume hashing from 30.5 MiB (31962051 bytes), run: s3sha256sum -verbose -paranoid 1s -resume c2hhAzu1WBGEEWVQ3Zv2rxB7JVM3e52vYUJxu0vj+OkOWT+XRlVLAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB57PD s3://bucketname/large-object.txt
To resume hashing from 44.9 MiB (47100867 bytes), run: s3sha256sum -verbose -paranoid 1s -resume c2hhA79vRAswpdTNUYQymT6+W4SgxT/hYUgPtM4CjYqL0hg5RGp3AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACzrPD s3://bucketname/large-object.txt

14d2c2bfee4a537bc803e063961d938ddd3904cd7db2306a6e79749a9aa5cd0c  s3://bucketname/large-object.txt

OK (matches object metadata)

This should help you if you're afraid that something else will kill s3sha256sum and interrupt the process. As long as you have the console output somewhere, you can recover.

Lastly, if someone can think of a good way to compress the state to make the command a bit shorter, please let me know.