[memory optimization] lookup prover memory analysis

[lispc]

During the memory profiling of proving of zkevm-circuit, we found that there seems to be some memory optimization viable in plookup prover in halo2.

commit_permuted of Argument costs no less than 70% of the total memory used by our proving process. More details can be found in the SVG graph link below. In this function, the unpermuted_cosets is calculated and saved in memory, which is later used in Committed::construct . So my question is, is it possible not to precompute & save unpermuted_cosets ? Can they be calculated on the fly in Committed::construct so we can avoid creating lots of temporary rotation polynomial to save memory?

I wonder whether my observation is right. Thank you!

Details: a flamegraph SVG made from heaptrack (we are testing a 16-degree circuit, the peak memory is about 50GB)

[str4d]

Oof, that's chonky 😄 We haven't exactly tested the lookup argument ourselves with large degrees, so this is useful feedback! I'll have a look into this.

Side-note: Halo 2 does not use plookup. Our lookup argument is documented here: https://zcash.github.io/halo2/design/proving-system/lookup.html

[lispc]

Thank you! @str4d

We also did some profiling of CPU. Here is a snapshot from "The profiling graph for proof (in flamegraph)" in zkevm's cpu profiling results. It seems lots of time is used in Domain::rotate_extended. I wonder maybe COW can be applied in the rotation to speed up? Just an idea. Thanks for any more comments

[str4d]

The rotate_extended time is already known by us, and we have a plan to fix it; doing so is blocked on refactoring how parallelizing works in halo2. Not sure if we actually opened an issue for it though; I'll do so today.