Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: more elegant solution to mem dump using unsafe and generics #501

Closed
wants to merge 4 commits into from
Closed

perf: more elegant solution to mem dump using unsafe and generics #501

wants to merge 4 commits into from

Conversation

gbotrel
Copy link
Collaborator

@gbotrel gbotrel commented May 3, 2024 •
edited
Loading

  • refactor: expose pedersen basis slices
  • feat: more elegant solution to mem dump using unsafe and generics

Motivation; enable other gnark objects (like groth16.ProvingKey) to be written and read very fast (though not portable). Limit code duplication between curves, fields, proof systems, etc.

  • Adds utils/unsafe with WriteSlice[S ~[]E, E any](w io.Writer, s S) error ... ;
  • kzg.UnsafeToBytes(...) []byte is now WriteDump(w io.Writer) error

🔥 Benchmarks for kzg SRS dump 🔥 --> (I mean using unsafe is a cheat, we just benchmark the non-unsafe method + real IO)

benchmark                                      old ns/op      new ns/op      delta
BenchmarkSerializeSRS/WriteTo-10               1227334792     1200767042     -2.16%
BenchmarkSerializeSRS/WriteRawTo-10            1291239625     1275224792     -1.24%
BenchmarkSerializeSRS/UnsafeToBytes-10         288414938      49002102       -83.01%
BenchmarkDeserializeSRS/UnsafeReadFrom-10      1589721583     1607936042     +1.15%
BenchmarkDeserializeSRS/UnsafeFromBytes-10     283540219      81233306       -71.35%

benchmark                                      old allocs     new allocs     delta
BenchmarkSerializeSRS/WriteTo-10               540            540            +0.00%
BenchmarkSerializeSRS/WriteRawTo-10            542            541            -0.18%
BenchmarkSerializeSRS/UnsafeToBytes-10         522            512            -1.92%
BenchmarkDeserializeSRS/UnsafeReadFrom-10      1046           1047           +0.10%
BenchmarkDeserializeSRS/UnsafeFromBytes-10     1029           1031           +0.19%

benchmark                                      old bytes      new bytes      delta
BenchmarkSerializeSRS/WriteTo-10               2147544696     2147544696     +0.00%
BenchmarkSerializeSRS/WriteRawTo-10            4295028640     4295028632     -0.00%
BenchmarkSerializeSRS/UnsafeToBytes-10         1610821246     67176202       -95.83%
BenchmarkDeserializeSRS/UnsafeReadFrom-10      1627598752     1627598848     +0.00%
BenchmarkDeserializeSRS/UnsafeFromBytes-10     1610820720     1610820744     +0.00%

@gbotrel gbotrel requested review from ivokub and Tabaie May 3, 2024 01:38
@github-actions GitHub Actions
Copy link

github-actions bot commented May 3, 2024

📦 github.com/consensys/gnark-crypto/kzg

kzg/kzg.go:40:10: cannot use &kzg_bn254.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bn254/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bn254/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:42:10: cannot use &kzg_bls12377.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls12-377/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls12-377/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:44:10: cannot use &kzg_bls12378.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls12-378/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls12-378/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:46:10: cannot use &kzg_bls12381.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls12-381/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls12-381/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:48:10: cannot use &kzg_bls24315.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls24-315/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls24-315/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:50:10: cannot use &kzg_bls24317.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls24-317/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls24-317/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:52:10: cannot use &kzg_bw6761.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bw6-761/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bw6-761/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:54:10: cannot use &kzg_bw6633.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bw6-633/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bw6-633/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:56:10: cannot use &kzg_bw6756.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bw6-756/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bw6-756/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)

@gbotrel gbotrel mentioned this pull request May 3, 2024
Merged
ivokub
ivokub approved these changes May 3, 2024
View reviewed changes
Copy link
Collaborator

@ivokub ivokub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Imo we can add a simple guard in case the slice to write is empty to have better errors, but I think normally shouldn't happen.

utils/unsafe/dump_slice.go
return err
}

data := unsafe.Slice((*byte)(unsafe.Pointer(&s[0])), size*len(s))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add guard in case len(s) == 0.

@gbotrel gbotrel deleted the branch refactor/utils_side_effect May 4, 2024 00:44
@gbotrel gbotrel closed this May 4, 2024
@gbotrel gbotrel mentioned this pull request May 4, 2024
Merged
gbotrel added a commit that referenced this pull request May 4, 2024
@gbotrel
perf: unsafe.WriteSlice (#501) (#503)
564b6f7 
* refactor: expose pedersen basis slices

* refactor: move test util package into isolated one

* feat: more elegant solution to mem dump using unsafe and generics

* fix: add BinaryDumper interface in kzg root package

* feat: add guard for len == 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ivokub ivokub ivokub approved these changes

@Tabaie Tabaie Awaiting requested review from Tabaie

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@gbotrel @ivokub