Cache and stream drgprah

[dignifiedquire]

General Idea:

How to cache the drgraph

• precalculate all parents:
  • regular: [[u64; 13], NODES]
  • reversed: [[u64; 13], NODES]
• store on disk: offsets into the data, eg parent * NODE_SIZE
  • regular: [[u64; 13]; NODES]
  • reversed: [[u64; 13]; NODES]

To read, memmap either regular.cache or reversed.cache, and read in chunks of 13 the required offsets.
This will be a continous read, and memmap + OS should be able to make this efficient for us, optimal for available memory.

• Estimated size: 3.25GiB for each cache file at a sector size of 1Gib
• This is of course strongly dependent on the amount of base and expander parents. (assumed to be 5 + 8 = 13 above).

[dignifiedquire]

When using sector sizes of up to 128GiB we can get away with storing u32s for the actual parents.

[porcuquine]

Correct. I suggest we bake in the assumption of sector sizes not exceeding 128GiB for an initial solution.

Note that we need to add 1 for the 'immediate predecessor', which is an extra DRG parent.

We also need to double the expander parents. However, we can then cut this in half if we are clever about not storing padding nodes. We can probably do that without too much difficulty with some small overhead. Exactly how best to do that will be clearer once we implement padding avoidance in circuits.