Feat: settable hasher for MiMC

[AlexandreBelling]

Description

The PR ports the changes made in gnark-crypto in feat: hash registry with statestorer and implements an equivalent [StateStorer] interface which is hash function agnostic. All hash functions can potentially implement the interface as it returns an array of [frontend.Variable] to represent the state which should work in all scenarios.

Type of change

• New feature (non-breaking change which adds functionality)

How has this been tested?

• TestStateStorer: the test attempts to use [State] and [SetState] to reproduce the same hash between two different hasher.

How has this been benchmarked?

Not benchmarked

Checklist:

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I did not modify files generated from templates
• golangci-lint does not output errors locally
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[ivokub]

Suggested edit:

diff --git a/go.mod b/go.mod
index c75380e1..da78b441 100644
--- a/go.mod
+++ b/go.mod
@@ -9,7 +9,7 @@ require (
 	github.com/blang/semver/v4 v4.0.0
 	github.com/consensys/bavard v0.1.22
 	github.com/consensys/compress v0.2.5
-	github.com/consensys/gnark-crypto v0.14.1-0.20241010154951-6638408a49f3
+	github.com/consensys/gnark-crypto v0.14.1-0.20241211083239-be3c2bbb1724
 	github.com/fxamacker/cbor/v2 v2.7.0
 	github.com/google/go-cmp v0.6.0
 	github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8
diff --git a/go.sum b/go.sum
index 1960733d..fc75f3a8 100644
--- a/go.sum
+++ b/go.sum
@@ -61,8 +61,8 @@ github.com/consensys/bavard v0.1.22 h1:Uw2CGvbXSZWhqK59X0VG/zOjpTFuOMcPLStrp1ihI
 github.com/consensys/bavard v0.1.22/go.mod h1:k/zVjHHC4B+PQy1Pg7fgvG3ALicQw540Crag8qx+dZs=
 github.com/consensys/compress v0.2.5 h1:gJr1hKzbOD36JFsF1AN8lfXz1yevnJi1YolffY19Ntk=
 github.com/consensys/compress v0.2.5/go.mod h1:pyM+ZXiNUh7/0+AUjUf9RKUM6vSH7T/fsn5LLS0j1Tk=
-github.com/consensys/gnark-crypto v0.14.1-0.20241010154951-6638408a49f3 h1:jVatckGR1s3OHs4QnGsppX+w2P3eedlWxi7ZFq56rjA=
-github.com/consensys/gnark-crypto v0.14.1-0.20241010154951-6638408a49f3/go.mod h1:F/hJyWBcTr1sWeifAKfEN3aVb3G4U5zheEC8IbWQun4=
+github.com/consensys/gnark-crypto v0.14.1-0.20241211083239-be3c2bbb1724 h1:lfTzZSy3FG2z5qFfRihDHmuolUvyEBWW8gsrjlZJQ/I=
+github.com/consensys/gnark-crypto v0.14.1-0.20241211083239-be3c2bbb1724/go.mod h1:ePFa23CZLMRMHxQpY5nMaiAZ3yuEIayaB8ElEvlwLEs=
 github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
 github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
 github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
diff --git a/std/hash/mimc/mimc_test.go b/std/hash/mimc/mimc_test.go
index a3e48112..6914a733 100644
--- a/std/hash/mimc/mimc_test.go
+++ b/std/hash/mimc/mimc_test.go
@@ -17,6 +17,7 @@ limitations under the License.
 package mimc
 
 import (
+	"crypto/rand"
 	"errors"
 	"fmt"
 	"math/big"
@@ -165,3 +166,58 @@ func TestStateStoreMiMC(t *testing.T) {
 			test.WithCurves(curve))
 	}
 }
+
+type recoveredStateTestCircuit struct {
+	State    []frontend.Variable
+	Input    frontend.Variable
+	Expected frontend.Variable `gnark:",public"`
+}
+
+func (c *recoveredStateTestCircuit) Define(api frontend.API) error {
+	h, err := NewMiMC(api)
+	if err != nil {
+		return fmt.Errorf("initialize hash: %w", err)
+	}
+	if err = h.SetState(c.State); err != nil {
+		return fmt.Errorf("set state: %w", err)
+	}
+	h.Write(c.Input)
+	res := h.Sum()
+	api.AssertIsEqual(res, c.Expected)
+	return nil
+}
+
+func TestHasherFromState(t *testing.T) {
+	assert := test.NewAssert(t)
+
+	hashes := map[ecc.ID]hash.Hash{
+		ecc.BN254:     hash.MIMC_BN254,
+		ecc.BLS12_381: hash.MIMC_BLS12_381,
+		ecc.BLS12_377: hash.MIMC_BLS12_377,
+		ecc.BW6_761:   hash.MIMC_BW6_761,
+		ecc.BW6_633:   hash.MIMC_BW6_633,
+		ecc.BLS24_315: hash.MIMC_BLS24_315,
+		ecc.BLS24_317: hash.MIMC_BLS24_317,
+	}
+
+	for cc, hh := range hashes {
+		hasher := hh.New()
+		ss, ok := hasher.(hash.StateStorer)
+		assert.True(ok)
+		_, err := ss.Write([]byte("hello world"))
+		assert.NoError(err)
+		state := ss.State()
+		nbBytes := cc.ScalarField().BitLen() / 8
+		buf := make([]byte, nbBytes)
+		_, err = rand.Read(buf)
+		assert.NoError(err)
+		ss.Write(buf)
+		expected := ss.Sum(nil)
+		bstate := new(big.Int).SetBytes(state)
+		binput := new(big.Int).SetBytes(buf)
+		assert.CheckCircuit(
+			&recoveredStateTestCircuit{State: make([]frontend.Variable, 1)},
+			test.WithValidAssignment(&recoveredStateTestCircuit{State: []frontend.Variable{bstate}, Input: binput, Expected: expected}),
+			test.WithCurves(cc))
+	}
+}

[ivokub]

Looks good to me.

I would also add a test which checks that if we store the state of native hash in gnark-crypto and initialize from the state in circuit, then the results are the same. If it makes sense to you then I have implemented the test. You can apply the suggested edit.

At first the CI would probably fail, I think you would also need to merge master to have all the other changes included as code generation has changed a bit.

[ivokub]

@AlexandreBelling - does it seem good? I can wrap up the PR myself (adding the suggested edit and rebasing on master to make CI work)

[ivokub]

LGTM!

[ivokub]

It seems there are more issues when updating the dependencies - we also changed in gnark-crypto for the underlying type of the small field elements to be array of uint32 (not array of uint64 as for large field elements), and this breaks more in the constraint system.

It will take a bit more to fix.

[AlexandreBelling]

I am wondering why this has an impact on the current PR. I can take a look into this tomorrow.

[ivokub]

I am wondering why this has an impact on the current PR. I can take a look into this tomorrow.

So, to test the consistency of restoring hasher from state between gnark and gnark-crypto I added a test where we create state from gnark-crypto and restore it in-circuit in gnark. To do that I had to update gnark-crypto dependency, but it had introduced optimization for small field generation which now uses array of uint32 instead of uint64 to represent small field element.

Additionally, in gnark we have locally a tinyfield (0x2f) code-generated code for some sanity checks as we can test edge cases for all possible inputs, and it was now also generated as [1]uint32. And it breaks some assumptions we have here and there (e.g. to avoid generics and code generation we use [6]uint64 for representing any coefficient value for terms).

We're already working on it on gnark-crypto side (Consensys/gnark-crypto#577) and when that is fixed then we can update the dependency here. There were also some other API changes for field code generation which we need to update in gnark to make tinyfield generation work again, but it is a very small change.

So to make it short - your PR is all good, it is just the other changes introduced in gnark-crypto which breaks stuff in gnark. And we need to resolve them to make CI happy.

[ivokub]

I am wondering why this has an impact on the current PR. I can take a look into this tomorrow.

So, to test the consistency of restoring hasher from state between gnark and gnark-crypto I added a test where we create state from gnark-crypto and restore it in-circuit in gnark. To do that I had to update gnark-crypto dependency, but it had introduced optimization for small field generation which now uses array of uint32 instead of uint64 to represent small field element.

Additionally, in gnark we have locally a tinyfield (0x2f) code-generated code for some sanity checks as we can test edge cases for all possible inputs, and it was now also generated as [1]uint32. And it breaks some assumptions we have here and there (e.g. to avoid generics and code generation we use [6]uint64 for representing any coefficient value for terms).

We're already working on it on gnark-crypto side (Consensys/gnark-crypto#577) and when that is fixed then we can update the dependency here. There were also some other API changes for field code generation which we need to update in gnark to make tinyfield generation work again, but it is a very small change.

So to make it short - your PR is all good, it is just the other changes introduced in gnark-crypto which breaks stuff in gnark. And we need to resolve them to make CI happy.

Alright - fixed all the stuff in other PRs and merged back here. Seems to work nicely now. I'll go ahead and merge.