Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add blake3 and blake3_file hash functions #1860

Merged
merged 7 commits into from
Feb 11, 2024
Merged

Conversation

tgross35
Copy link
Contributor

Blake3 is a hash algorithm that is pretty popular when working with larger files, and is also (in theory) closer to SHA-3 in security than SHA-2. I believe it is used by sccache and some other larger build systems.

This adds blake3 and blake3_file functions. blake3_file is especially of note because it uses mmap and parallelization rather than reading the file into memory, so it should outperform sha256_file quite a bit (this is the same function used by the b3sum utility).

I need to rebase since I did this a while ago, but I may as well get your thoughts here.

@casey
Copy link
Owner

casey commented Jan 21, 2024

Love it, this is super cool.

These functions provide a faster alternative to `sha256` and
`sha256_file`. In particular, `blake3_file` uses memory map to avoid
reading the entire file into memory, as well as parallelization to
calculate the hash (this is consistent with the `b3sum` command line
tool).
@tgross35
Copy link
Contributor Author

Awesome! I rebased so this should be all set for a look.

@casey
Copy link
Owner

casey commented Feb 11, 2024

Nice, LGTM! I used blake3::hash instead of a hasher in in blake3(), since it's one and done and doesn't need incremental updates. I also changed uses of path.to_str to path.display(). Since paths are arbitrary bytes, they can't necessarily be converted to a string, and path.display returns something that represents a path but has a display implementation, so it can be printed. It looks like we made the same mistake in the sha256 functions, so I fixed those too.

@casey casey enabled auto-merge (squash) February 11, 2024 20:54
@casey
Copy link
Owner

casey commented Feb 11, 2024

I also added <sup>master</sup> annotations, so people know this hasn't made it into a release yet.

@casey casey merged commit dc952f9 into casey:master Feb 11, 2024
5 checks passed
@tgross35
Copy link
Contributor Author

Awesome, thanks!

@tgross35 tgross35 deleted the add-blake3 branch February 12, 2024 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants