Splitstore: next steps

[vyzo]

Now that we have shipped the first version of splitstore, we are looking for ways to improve it with the long term goal of making it the default in the future.

Here are some steps that we plan to take to improve it further:

• Merge splitstore: compact hotstore prior to garbage collection #5778
• Adjust options for the coldstore badger so that it doesn't memmap and put pressure in the memory subsystem whenever another node is syncing off a splitstore node.
• Kill compactFull; it was originally written with an eye towards garbage collection, but it hasn't proven very useful so far; all our nodes all use simple compaction (Splitstore optimizations #5808)
• WalkSnapshot is marking as live more objects that we need, as it effectively walks the chain all the way to genesis; we really don't need to keep old blocks in the hot range. It is also very slow (precisely because it walks all the way to genesis). So the plan is to replace WalkSnapshot with an optimized variant that only marks reachable state roots and not old blocks and stops once we are out of finality range (Splitstore optimizations #5808)
• Tracking store: use nil valued keys embedding the write tipset in a compound key <write_tipset>:<cid> as this allows us to do very efficient range scans.

[vyzo]

These have been addressed with splitstore v1.