use partial parent cache and other methods to reduce the mem usage for sdr

[dtynn]

tested on a machine

• AMD Ryzen 9 3900X 12-Core Processor
• 62.9GiB physical mem
• 10GiB swap on ssd
• 8T hdd (7200rpm) for staged file, sealed file & cache dir

with

• 13:32:51 ~ 20:11:44 for seal_pre_commit_phase1
• 20:11:44 ~ 23:04:04 for seal_pre_commit_phase2

TODOs:

• patch bellman to serialize circuit synthesizing & calculating: bellman pr #81
• use mixed-base-layer to see if we can reduce some more mem for seal_pre_commit_phase1
• copying base layer to exp cost ~9min in the last 4 layers, with a high ratio of SWAPIN, which I think, can be optimized

see log here

[porcuquine]

For env var configuration, please use the mechanism in settings.rs for consistency and to consolidate in one place developers and users can keep track of.

[dignifiedquire]

@dtynn what are the numbers you are seeing on that setup with the current implementation, so we have a comparison.

[dtynn]

@dtynn what are the numbers you are seeing on that setup with the current implementation, so we have a comparison.

@dignifiedquire
considering for seal_pre_commit_phase1 only:

62.9GiB physical + 10GiB swap, both full used VS 56GiB (parent cache) + 64GiB, with 50% ~ 80% more time cost for each layer

if we use the official implementation without full cached parent indexes(FIL_PROOFS_MAXIMIZE_CACHING=0), mem usage reduces, but the time would be much longer (don't remember the exact numbers)

update

for official implementation without full cached parent indexes(FIL_PROOFS_MAXIMIZE_CACHING=0), used 2h11m for layer1
benchy binary compiled with

CC=clang RUSTFLAGS="-C target-cpu=native -C target-feature=+sse4.1,+sse4.2,+avx,+avx2,+sse2,+sha,+adx," cargo build --release -p fil-proofs-tooling --bin benchy

[dignifiedquire]

@dtynn for the numbers you reported, what value for partial_num did you use?

[dtynn]

@dtynn for the numbers you reported, what value for partial_num did you use?

@dignifiedquire
128

[dignifiedquire]

Thanks a lot for this @dtynn! I have used this code to build #1163 which has similar properties, but better integrated into the machinery.