add flag to 'ipfs add' to choose blake2 as hash function

[jbenet]

@whyrusleeping just thought that when we add blake2 support, prior versions wont be able to use those hashes. if we can throw in support in 0.4.0 it would help. Feature creep i know but breakingness :S

• https://blake2.net/
• https://github.com/codahale/blake2

(dat speed!)

[rht]

I'd vote for blake2b/s, but keep in mind that ipfs-inactive/archives#35 (comment).

For effective optimization, it'd be better to make a breakdown of the bottlenecks #1616 (comment) (this benchmark is old, but just to serve a point).

[jbenet]

@rht makes sense. i'd opt for 2b/s as well.

[rht]

Here is a heuristic benchmark:

ipfs add -q -r Godeps on https://github.com/ipfs/go-ipfs/tree/e8c753d2e52b9f6e47d7f13c6b6fcffd0888c548 with various hash functions:

• sha2-256: 90ms (total time spent on addAllAndPin: 363ms)
• sha2-512: 100ms
• blake2b-256: 63ms
• blake2b-512: 86ms
• sha3-256: 85ms
• sha3-512: 144ms
• sha1: 50ms

[whyrusleeping]

We should probably add support for this before ipfs 0.5.0

[jbenet]

would be nice

[aead]

Just for reference:
The "offical" golang x/crypto sub-repo now contains both - BLAKE2b and BLAKE2s

[whyrusleeping]

With the latest bitswap fixes, we can change the hash function up and have it work just fine. Lets actually get around to prioritizing this as an experimental feature

[whyrusleeping]

@Kubuxu do you think you could take on getting the blake2 multihash stuff implemented?

[Kubuxu]

For 0.4.5?

[whyrusleeping]

@Kubuxu hmm... I'd like to be able to have it in 0.4.5. But thats a decent hunk of work. If you think we can do it quickly enough then lets go for it, otherwise, its on 0.4.6

[Kubuxu]

I will use this issue to add things I need you to take a look at for blake2 support:

First one: multiformats/multicodec#25

[whyrusleeping]

The code for blake2 is now merged into master and will be in the 0.4.6 release. Its not yet available as an easily specifiable parameter to ipfs add, but that can be added soon.

[whyrusleeping]

Moving the implementation of a --hash-fn=blake2b-256 flag for add into the 0.4.7 milestone

[whyrusleeping]

Alright, having two releases in the wild that can understand this hash function is sufficient for deploying the change to master.

Lets go ahead and do this for 0.4.8 :) (it should be very simple)

[kevina]

@Kubuxu this is assigned to you. Do you want me to take it? Should be easy after #3743 is merged.

[Kubuxu]

It requires dependency propagation of go-multihash. So if you ware up to it, go for it.

[kevina]

@whyrusleeping the go-multihash dep is used nearly every where so it will be a difficult dependency to update and there are several outstanding P.R. for additional hashes. Do you still want to target this for Ipfs 0.4.8 or bump it?

[whyrusleeping]

@kevina we can probably bump this one to 0.4.9

I'm working right now on bubbling a dependency update, but it shouldnt be needed for blake2. I beleive that code is already merged into master here

[whyrusleeping]

The dep is for sure in master now. Lets get this into 0.4.10 :)

[kevina]

@whyrusleeping This sounds like an easy one for me to do, shall I?

[kevina]

With #3919 merged this is now possible.

[JayBrown]

Late question: is there a list of all the hashes supported by IPFS you can use with ipfs add --hash <hash>?

For example, an older one is here – https://multiformats.io/multihash/ – but blake2s-128 doesn't seem to be supported anymore. (At least it throws an error when using it.)

[kevina]

@JayBrown There is now, ipfs cid hashes.

[JayBrown]

@kevina Is that in a new version? My go-ipfs v0.4.17 doesn't have ipfs cid (yet).

EDIT: you'd probably also have to specify the CID version to print the respective list of hashes.

[Stebalien]

@jay we just released 0.4.18.

[dozykeys]

Good day @Stebalien

When i try to upload to ipfs, everything is successful, but whenever i try to use blake2b for the hashAlg,it fails with error, no hasher found for blake2b-208. I observed that just sha256 and sha512 are available.

IPFS Version : 0.63.5
Below is the full error

[nodemon] restarting due to changes...
[nodemon] starting node dozie2.js
Swarm listening on /ip4/127.0.0.1/tcp/4002/p2p/12D3KooWRnb6NYEcUaeciXQhCaERJnqcnbPGuQ9BZhQgjqqM64Lx
Swarm listening on /ip4/192.168.8.96/tcp/4002/p2p/12D3KooWRnb6NYEcUaeciXQhCaERJnqcnbPGuQ9BZhQgjqqM64Lx
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/p2p/12D3KooWRnb6NYEcUaeciXQhCaERJnqcnbPGuQ9BZhQgjqqM64Lx
file:///Users/macbook/Documents/node_apps/RC3-NFT/node_modules/ipfs-core-utils/src/multihashes.js:10
const LOAD_HASHER = (codeOrName) => Promise.reject(new Error(No hasher found for "${codeOrName}"))

Error: No hasher found for "blake2b-208"
at Multihashes.LOAD_HASHER [as _loadHasher] (file:///Users/macbook/Documents/node_apps/RC3-NFT/node_modules/ipfs-core-utils/src/multihashes.js:10:52)
at Multihashes.getHasher (file:///Users/macbook/Documents/node_apps/RC3-NFT/node_modules/ipfs-core-utils/src/multihashes.js:70:31)
at addAll (file:///Users/macbook/Documents/node_apps/RC3-NFT/node_modules/ipfs-core/src/components/add-all/index.js:97:30)
at addAll.next ()
at file:///Users/macbook/Documents/node_apps/RC3-NFT/node_modules/ipfs-core-utils/src/with-timeout-option.js:64:60
at AsyncGenerator.next ()
at last (/Users/macbook/Documents/node_apps/RC3-NFT/node_modules/it-last/index.js:13:20)
at IPFS.add (file:///Users/macbook/Documents/node_apps/RC3-NFT/node_modules/ipfs-core/src/components/add.js:14:26)
at /Users/macbook/Documents/node_apps/RC3-NFT/dozie2.js:64:23

[Jorropo]

@dozykeys you are showing a js-ipfs stack trace not a Kubo one.
You should open a new issue on js-ipfs's repo but
It's probable js-ipfs "just" don't support blake2.

[dozykeys]

okay thank @Jorropo but can i also use kubo on nodejs?

[Stebalien]

No. Kubo is written in go.

[dozykeys]

@Stebalien thanks.