Reuse of createHash initializations

[ppKrauss]

Reuse is a fundamental issue...

const crypto = require('crypto')
const H = crypto.createHash('md5')  // NEED REUSE THIS INITIALIZATION
console.log( H.update('some data to hash').digest('hex') ) // first ok...
console.log( H.update('some2').digest('hex') )  // ugly "Error: Digest already called"!

---

The problem that I am trying to solve: to reuse initialization.

The desired behavior: no error.

... No alternative solutions, but please, show me if there are some elegant one.

[sam-github]

Could you describe what it is you need, and why?

Your example isn't helpful. I don't know if you want openssl style digest reuse, where after _final() the only call that can be made is an init, after which new data can be digested, or whether you want to continue on hashing new data as if the digest had not been extracted.

The latter would require calling EVP_MD_CTX_copy() every time before getting the digest value, so would be prohibitively expensive. It would be possible to expose a .copy() method on digests, if that's what you want, and you have a motivating use-case.

[ppKrauss]

Hi @sam-github ,

Your example isn't helpful

any loop needs reuse:

const crypto = require('crypto')
const H = crypto.createHash('md5')  // NEED REUSE THIS INITIALIZATION

for (let a of  ["Ana Santos", "José Silva", "Bryan Lee", "Maria Silva", "Vitor Santos"]) 
    // ... or suppose looping over 10000 items
    console.log( a,  H.update(a).digest("hex") )  // ugly error

But the only valid (no error) syntax option is:

const H = require('crypto').createHash
for (let a of  ["Ana Santos", "José Silva", "Bryan Lee", "Maria Silva", "Vitor Santos"]) 
    console.log( a,  H('md5').update(a).digest("hex") )
    // of course, no error, but no reuse.

Please, show an example of reuse using the cited .copy().

---

Important: there are many crypto.getHashes() algorithms... This question is about only one, sha256, there are 2,493 pageviws of users that did not understand "Digest already called" error... This error not make sense: because reuse make sense.

[sam-github]

This still does not answer my question, you provide code, but no explanation of how you want the code to work.

I don't know if you want openssl style digest reuse, where after _final() the only call that can be made is an init, after which new data can be digested, or whether you want to continue on hashing new data as if the digest had not been extracted.

Specifically, will iteration 2 output the hash of "José Silva", or of "Ana SantosJosé Silva"?

[ppKrauss]

@sam-github sorry, I edited with complete example... But really not understand what you show as solution.

[sam-github]

Specifically, will iteration 2 output the hash of "José Silva", or of "Ana SantosJosé Silva"?

It appears you want iteration 2 to output the hash of "José Silva", in other words a call to .digest() will reset the internal state back to the initial just-constructed state, is this correct?

But really not understand what you show as solution.

I offered no solution. Until now, I wasn't even sure what the problem was.

I'll try to find some time to benchmark initialization to see what kind of performance gains are possible.

Providing your own benchmarks might help motivate someone to pick this up, or you could PR the change. It seems reasonable to me.

[bnoordhuis]

a call to .digest() will reset the internal state back to the initial just-constructed state

It would be kind of odd (in a 'no parity' sense) to have that work for hashes but not e.g. HMACs.

(You could theoretically make it work for HMACs by retaining the secret but I don't think we should.)

[sam-github]

Differing from HMAC would be a definite downside.

I notice that the openssl APIs have the ability to reuse the HASH context. I assume this is for use-cases where a HASH with some params is created, then used over and over to digest large numbers of high-throughput blocks/records/whatever, and avoid having to do a digest-init for each one.

That OpenSSL does it suggests it might be useful, but whether it is relevant to any node users, or even noticable if it was done is an open question. I keep hoping someone will come up with a use-case backed by benchmarks showing that this would have a significant impact for them, but I haven't seen it yet.

[tniessen]

I did a quick'n'dirty benchmark to see how much time createHash() needs in the following snippet (data is a buffer):

let hash = crypto.createHash('sha256');
hash.update(data);
hash.digest();

data	createHash('sha256')
32 B	52%
256 B	40%
1 KB	25%
2 KB	16%
4 KB	9%
8 KB	5%
16 KB	3%
64 KB	0.1%

No surprises so far, the initialization time only matters for small inputs, and probably even less when dealing with strings instead of Buffers. Three questions arise:

1. How many applications need to compute large numbers of cryptographic hash values of small inputs?
2. What is the primary cause for the time spent, is it node or OpenSSL? (Likely node / v8.)
3. Is there another way to solve the problem? (e.g. pooling)

[ppKrauss]

Thanks @tniessen, perfect (!), now will be possible do finish this issue... after some discussion.

The main justification is not performance, but something like "syntax usability enhancement", as I commented before:
"... this question is about only one of many, the sha256, there are 2,650 pageviws of users that did not understand 'Digest already called' error... This error not make sense: because reuse make sense".

About performence: ok, the gain is 0 or, sometimes, greater tham zero. Is good, any gain will be a sum, a better justification, little better cost-benefit.

[tniessen]

@ppKrauss You have expressed your personal opinion quite strongly, both here and on StackOverflow. Please keep the discussion productive with constructive arguments instead of repeating that things are "ugly" and that "reuse makes sense".

If the only benefit is "syntax usability enhancement", then consider the downsides:

• It would differ from the way other crypto APIs behave, see Reuse of createHash initializations #25857 (comment).
• As @sam-github explained, the semantics are not immediately obvious and will lead to confusion, see Reuse of createHash initializations #25857 (comment).
• Node.js is primarily used for asynchronous applications. And I am 100% sure that people will incorrectly use Hash objects in asynchronous code as soon as we allow reuse. I assume very few applications need to compute many hashes of small input values in a synchronous way.

[AshDee]

Regarding this thread.

In my code, I try to create hashes by reading different files by using createReadStream. But It is only possible to read one file. If I try to read the second file I get the error Digest already called.

From the documents, I came to know that digest can be used only once.

So, is there a workaround to read each file and generate corresponding hashes for each file?

As @sam-github had mentioned in his previous comment-

It appears you want iteration 2 to output the hash of "José Silva", in other words, a call to .digest() will reset the internal state back to the initial just-constructed state, is this correct? -

I would like to have something like this

[tniessen]

In my code, I try to create hashes by reading different files by using createReadStream. But It is only possible to read one file. If I try to read the second file I get the error Digest already called.

@AshDee I don't understand, why don't you just create a new instance using createHash for each file? The additional overhead is minimal compared to IO delays.

[AshDee]

In my code, I try to create hashes by reading different files by using createReadStream. But It is only possible to read one file. If I try to read the second file I get the error Digest already called.

@AshDee I don't understand, why don't you just create a new instance using createHash for each file? The additional overhead is minimal compared to IO delays.

Thank You @tniessen .I'll try it out...

[bnoordhuis]

I'm closing this out in favor of #29903.

[hippyaki]

Hi @sam-github ,

Your example isn't helpful

any loop needs reuse:

const crypto = require('crypto')
const H = crypto.createHash('md5')  // NEED REUSE THIS INITIALIZATION

for (let a of  ["Ana Santos", "José Silva", "Bryan Lee", "Maria Silva", "Vitor Santos"]) 
    // ... or suppose looping over 10000 items
    console.log( a,  H.update(a).digest("hex") )  // ugly error

But the only valid (no error) syntax option is:

const H = require('crypto').createHash
for (let a of  ["Ana Santos", "José Silva", "Bryan Lee", "Maria Silva", "Vitor Santos"]) 
    console.log( a,  H('md5').update(a).digest("hex") )
    // of course, no error, but no reuse.

Please, show an example of reuse using the cited .copy().

Important: there are many crypto.getHashes() algorithms... This question is about only one, sha256, there are 2,493 pageviws of users that did not understand "Digest already called" error... This error not make sense: because reuse make sense.

@ppKrauss Love you mate. Awesome solution with that. Works GREAT for me.