beacon: Cull beacon db memory (passivation)

[jamescowens]

This pull request implements passivation of beacon elements in the beacon db while leaving the leveldb backing store untouched. This was already anticipated in the version of the beacon db that was in 5.2.x, but required some further adjustments to implement.

Some comments on the theory behind this and the implementation...

In general, we have the problem in Gridcoin of certain contracts being linked by protocol logic. Beacon contracts are an example. There is a relatively complex process for beacons that involve several different paths through possible beacon states. The most typical example is

Advertisement (pending) -> Activation (verification) -> Renewal -> Renewal -> ....-> etc.

Each CPID's series of beacons can be thought of as a "chainlet". This was already implemented in the BeaconRegistry/BeaconDB for 5.2.

Without getting into specifics, the beacon registry for renewals and reversions needs to be able to pull up the "prior" beacon. All beacons that are not the original beacon in a beacon chain" have a previous beacon hash entry, which is a form of a one-way linked list by hash (as opposed to CBlockIndex, which is a two-way linked list by pointer). We don't need the forward direction directly in the beacon object, only the reverse linkage, and unlike the the complexity done in CBlockIndex to turn the hash links in CDiskBlockIndex into pointers for absolute speed, we can just use the hashes and map finds, which are slower but ok, because this is not used as much.

The beacon contract fundamentals are recorded in the beacon db IN THE ORDER they were actually encountered originally in the blockchain and applied. This is the only forward state we need and is used in initialization of the beacon state from leveldb at wallet startup to avoid having to replay all of the contracts themselves, which is very heavyweight.

For normal wallet running conditions though, we do not need all of those beacon records to remain in memory. The m_historical map nominally would contain an entry for every beacon state encountered for every beacon since 180 days from the head OR from the Fern hard fork point of 2053000, whichever is earlier. (This is true up to and including 5.3.0.0.) It is obvious to see that this grows without bound over time as the back of the window is effectively pinned at the Fern hard fork point, although the size of the beacon maps is very small compared to the block map and block index.

The passivation in this PR essentially removes from memory historical records that are not currently being referred to by either m_beacons (the active map) or m_pending (the pending map). It is a primitive "minimum working set" algorithm, and quite frankly, all we need to do the job here. The find function for the beacon db are extended from the normal find function to first look in memory for the provided hash, and if it is not found, transparently load it from leveldb into the map and then provide the iterator or value. This allows functions, such as renewals, which need to pull up the prior beacon record to match the key/verify to work properly with a prior record that was passivated. As a renewal is processed, it will implicitly load back into memory the prior record. The find and insert will set a flag m_needs_passivation to true if they had to load an element into memory that was already passivated before. This will allow a scheduled passivation function to repassivate the element later if it is no longer needed.

During initialization of the BeaconRegistry/BeaconDB during wallet startup, passivation is called on each historical record pointed to by a newly loaded record's previous beacon hash to minimize the memory footprint on load. (Unfortunately, a major downside in the beacon db implementation from a memory use perspective is the need as discussed earlier to preserve the exact order of the contract application as originally encoded in the blockchain. This requires ordering the records by a record sequence number key rather than the native hash as in memory. This necessitates the use of a temporary map ordered by record sequence number populated by an leveldb iterator, which is then used to populate the m_historical map. This temporarily uses more memory but is unavoidable to preserve leveldb performance, especially using regular disks - a leveldb iterator in natural key order is MUCH faster than an iterative find in leveldb.) The m_needs_passivation flag is set after initialization to run passivation across the entire database on the next scheduled run to catch records that could not be passivated on initial load because they were originally pinned by the beacon workflow.

There is a scheduled job using the global scheduling instance that now calls the database wide passivation every 5 minutes. If the m_needs_passivation is false, it will exit without doing anything. If true, it will run passivate_db(), which will iterate through the beacon_db and call passivate on each record, then set the flag to false.

[jamescowens]

So far this runs well in both mainnet and testnet, but fails the deep reorg test. See the attached log. The auditsnapshotaccruals at 2196000 (before the new newbie fix height) is supposed to have mismatches and it does. When reorganized back to the original height these mismatches should disappear. They don't.

See the included log:
reorg_run_1-20210322-failure.log

[jamescowens]

Ah... the failure is a false positive. It fails on master this deep too. I just realized this reorg doesn't rebuild the accruals after it comes back, which is actually done by a 5.3 client ON FIRST RUN if the accrual version doesn't match. This will not be done on a deep reorg and back. A reorg this deep is unnatural anyway. I will retest it with a shallower one.

[jamescowens]

Ok. I did a shallower reorg from 2204007 to 2199000 and back and it looks good. I am calling it a possible success, because the auditsnapshotaccruals run after the reorg but before the reorg back shows all matches but one CPID not present. The auditsnapshotaccruals run after the reorg back matches the original auditsnapshotaccruals perfectly.

reorg_run_2-20210322-possible_success.log

[div72]

ACK d036b82.