Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/scroll mem reduction #274

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Feat/scroll mem reduction #274

wants to merge 7 commits into from

Conversation

CPerezz
Copy link
Member

@CPerezz CPerezz commented Feb 9, 2024

Results so far

In main.

Start: Proof Gen for K=21 GWC, plonk_api test
End: Proof Gen for K=21 GWC, plonk_api test ..................................129.805s
Start: Proof Gen for K=21 SHPLONK, plonk_api test
End: Proof Gen for K=21 SHPLONK, plonk_api test ..............................128.257s

Dhat output for K = 21 GWC

dhat: Total:     49,691,685,732 bytes in 24,022 blocks
dhat: At t-gmax: 9,932,123,677 bytes in 142 blocks
dhat: At t-end:  5,744 bytes in 3 blocks

Dhat output for K = 21 SHPLONK

dhat: Total:     51,002,396,268 bytes in 24,416 blocks
dhat: At t-gmax: 9,932,123,677 bytes in 142 blocks
dhat: At t-end:  5,744 bytes in 3 blocks

In feat/scroll_mem_opt

Start:   Proof Gen for K=21 GWC, `plonk_api` test
End:     Proof Gen for K=21 GWC, `plonk_api` test ..................................196.833s
Start:   Proof Gen for K=21 SHPLONK, `plonk_api` test
End:     Proof Gen for K=21 SHPLONK, `plonk_api` test ..............................198.223s

Dhat output for K = 21 GWC

dhat: Total:     67,626,846,484 bytes in 6,325,485 blocks
dhat: At t-gmax: 5,066,733,341 bytes in 160 blocks
dhat: At t-end:  5,744 bytes in 3 blocks

Dhat output for K = 21 SHPLONK

dhat: Total:     68,937,557,020 bytes in 6,325,879 blocks
dhat: At t-gmax: 5,771,389,277 bytes in 291 blocks
dhat: At t-end:  5,744 bytes in 3 blocks

@CPerezz
Copy link
Member Author

CPerezz commented Feb 9, 2024

Current status:

Testing plonk_api test and failing for both IPA, GWC and SHPLONK.
It's weird as I checked the diff against taikoxyz#6 and axiom-crypto#17 and they're identical except from some changes in generics I made.

Next things to analyze:

  • I haven't added any bugs in the plonk_api test by mistake when adding the profiling features.
  • Start to look deeper on whether permutation or lookup can be the cause (by removing them from the test CS).
  • If none of them are, then we will have a bigger issue as the error is related to the change but only us have it. None of the other forks do. Which makes it quite a bone to chew 😞

EDIT: Had the brilliant idea of running all tests.

test plonk::circuit::tests::iter_product ... ok
test plonk::circuit::tests::iter_sum ... ok
test plonk::tests::add_trivial_to_inv0_rational ... ok
test plonk::tests::sub_rational_from_inv0_rational ... ok
test plonk::tests::mul_rational_by_inv0_rational ... ok
test plonk::tests::add_rational_to_inv0_rational ... ok
test plonk::tests::sub_trivial_from_inv0_rational ... ok
test arithmetic::test_lagrange_interpolate ... ok
test poly::domain::tests::test_coeff_to_extended_part ... ok
test poly::domain::tests::test_rotate ... ok
test poly::domain::tests::test_l_i ... ok
test plonk::proptests::operation_commutativity ... ok
test poly::domain::tests::test_lagrange_vecs_to_extended ... FAILED
test poly::kzg::commitment::test::test_parameter_serialisation_roundtrip ... ok
test poly::kzg::commitment::test::test_commit_lagrange ... ok
test poly::ipa::msm::tests::msm_arithmetic ... ok
test poly::multiopen_test::test::test_roundtrip_gwc ... ok
test poly::multiopen_test::test::test_roundtrip_shplonk ... ok
test poly::ipa::commitment::test::test_commit_lagrange_epaffine ... ok
test poly::ipa::commitment::test::test_commit_lagrange_eqaffine ... ok
test poly::multiopen_test::test::test_roundtrip_ipa_keccak ... ok
test poly::multiopen_test::test::test_roundtrip_ipa ... ok
test poly::ipa::commitment::test::test_opening_proof ... ok
test poly::domain::tests::bench_lagrange_vecs_to_extended ... ok
test poly::kzg::multiopen::shplonk::proptests::test_intermediate_sets ... ok
test plonk::circuit::compress_selectors::tests::test_selector_combination ... ok
test poly::domain::tests::bench_coeff_to_extended_parts ... ok

Seems we have a possible candidate.

@CPerezz
Copy link
Member Author

CPerezz commented Feb 9, 2024

Although plonk_api test now passes. One thing to notice is that none of Scroll, Axiom nor Taiko have the shuffle API. Hence they simply got ridd of it and did not need to adapt it to the new Evaluator.

So now is time to integrate the shuffle API with the new Evaluator stack and see if we can make it work.
Ideally, this shouldn't be too hard as considering that the shuffle API isn't expected to be a gigantic argument with lots of instantiations, the idea is that we can simply do FFT/IFFT of the entire thing at once and merge it later with $g(x)$ later to construct $h(x)$

This incorporates the work done in
scroll-tech#28 in order to lower the
memory consumption significantly trading off for some performance.

A much more deep analysis can be found here: axiom-crypto#17
This remodels the `plonk-api` test such that is now easy to extract
meterings for `heap` and `perf` profiling.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant