refactor state_history's logs to be mutex free

[spoonincode]

Refactor ship log to no longer hold a mutex that can potentially block the main thread for extended (or even indefinite) time. Resolves #274. Coupled with AntelopeIO/leap#236 resolves AntelopeIO/leap#20.

The thread safety mostly comes from the new random_access_file that is similarish to cfile but provides mutex free access to a file (at least, mutex free from the standpoint of our application -- there may serialization in the kernel). Additionally, the ship log is modified to be append only: previously the ship log would truncate away forked out blocks when applying a fork, now the forked out blocks remain in the log but are no longer "reachable" -- the index doesn't refer to them (but need to be careful when transversing the log in reverse, such as rebuilding the index; this may arguably make this a ship log "format change")

[ericpassmore]

Note:start
group: STABILITY
category: INTERNALS
summary: Refactor ship log to no longer hold a mutex that can potentially block the main thread for extended (or even indefinite) time.
Note:end

[linh2931]

Since this is a rewrite of SHiP, we can go over old SHiP issues to make sure they are not regressed if still applicable, and add more unit and integration tests.

[spoonincode]

This test applies a fork that decreases the resulting head block number, and then closes the log and reopens it. It was modified/removed because the new log implementation does not like this (to be specific, it fails on reopening the log file because the index contains more blocks than the log contains). Since this is now valid in savanna the log will need to be improved.

For the simple single-log case this is quite a trivial fix: truncate the index as appropriate -- nothing in the ship connection thread could be using the index. For the partitioned log case things get a wee bit more tricky.

[spoonincode]

There has been a fairly substantial modification of the logic in b98fe82 to ensure that the log is left in a sane state when a fork with a block number less than the previous head block is applied (this isn't strictly a problem when "closes the log and reopens it" as previously discussed). The change will require careful review.

For the single-log case it is as trivial as expected: truncate the index and then tweak some state of the log a little (arguably simpler now).

For partitioned logs, the approach is to "unrotate" the logs as needed: if the applied block occurs before the first block in the head log, the head log is removed and the newest retained log is restored as the head log. A new unit test has been enhanced to probe in this area more. I've also restored more of this section of the ship_log test that was removed previously.

#221 comes along for the ride in the improvement because now random_access_file and state_history_log are moveable (1876cc8 and 1eccd39)

[linh2931]

Since this is a rewrite of SHiP, we can go over old SHiP issues to make sure they are not regressed if still applicable, and add more unit and integration tests.

After I reviewed

Since this is a rewrite of SHiP, we can go over old SHiP issues to make sure they are not regressed if still applicable, and add more unit and integration tests.

tests/ship_log.cpp has a large number of new unit tests. We can just go through old SHiP issues and add more integration tests to reflect production usages.