Track sealing processes across lotus-miner restarts

[magik6k]

Summary of changes here:

• Worker calls are now async
  • When called worker methods
    • Generate and persist a CallID (basically a random UUID)
    • Start the work in a goroutine
    • Return the CallID to manager
    • After work is finished call Return[...] with the CallID and the result on the Manager
• Manager is now tracking work and calls across restarts
  • When a method is called on the manager, it generates WorkID (tuple of task type (like PreCommit1) and params)
  • If we see that this WorkID is not running we persist it in work statestore (set work state as started)
  • We schedule the task as before, after the scheduler allocates us to a worker, we execute the worker call
  • We connect the CallID we got from the worker to the WorkID, and set the work state to running
  • After the worker returns, we translate the CallID to WorkID, and pipe the result to the correct place, mark the Work as done, and the Call as returned
• Manager <-> Worker RPCs are now using HTTP instead of websockets, not having to rely on a single TCP connection serviving for days is much more robust

[whyrusleeping]

Alright, so i left a number of comments, and wasnt able to do a good job reviewing worker_local.go, but overall this LGTM.

[arajasek]

Okay, I'll call this "approved" after having spent a couple more hours on it, but that basically means "I think I understand how this is supposed to work". Ish.

[Kubuxu]

SGWM, but if there is a subtle bug here, I've for sure missed it.

[magik6k]

Closes #3453
Closes #3585
Closes #3597
Closes #3672
Closes #3744
Closes #3814
Closes #3815
Closes #3825
Closes #4262

[cryptowhizzard]

Seems my issue is not resolved yet ( on the latest master build ).

2020-11-10T13:42:13.729 INFO filecoin_proofs::api > generate_piece_commitment:start
2020-11-10T13:42:14.093 INFO filecoin_proofs::api > generate_piece_commitment:finish
2020-11-10T13:42:14.098 INFO filecoin_proofs::api > generate_piece_commitment:start
2020-11-10T13:42:14.173 INFO filecoin_proofs::api > generate_piece_commitment:finish
2020-11-10T13:42:14.180 INFO filecoin_proofs::api > generate_piece_commitment:start
2020-11-10T13:42:17.729Z ^[[34mINFO^[[0m miner miner/miner.go:382 Time delta between now and our mining base: 47s (nulls: 0)
2020-11-10T13:42:18.176Z ^[[34mINFO^[[0m rpc go-jsonrpc@v0.1.2-0.20201008195726-68c6a2704e49/client.go:346 rpc output message buffer {"n": 4}
2020-11-10T13:42:18.367Z ^[[34mINFO^[[0m miner miner/miner.go:382 Time delta between now and our mining base: 48s (nulls: 1)
2020-11-10T13:42:18.419Z ^[[34mINFO^[[0m rpc go-jsonrpc@v0.1.2-0.20201008195726-68c6a2704e49/client.go:346 rpc output message buffer {"n": 5}
2020-11-10T13:43:13.667Z ^[[31mERROR^[[0m rpc go-jsonrpc@v0.1.2-0.20201008195726-68c6a2704e49/websocket.go:654 Connection timeout {"remote": "127.0.0.1:1234"}
2020-11-10T13:43:13.667Z ^[[31mERROR^[[0m storageminer storage/wdpost_run.go:751 handler: websocket connection closed
2020-11-10T13:43:13.667Z ^[[33mWARN^[[0m storageminer storage/wdpost_sched.go:106 window post scheduler notifs channel closed
2020-11-10T13:43:13.667Z ^[[31mERROR^[[0m events events/events_called.go:382 event diff fn failed: handler: websocket connection closed
2020-11-10T13:43:13.667Z ^[[33mWARN^[[0m events events/events.go:97 listenHeadChanges quit
2020-11-10T13:43:13.667Z ^[[33mWARN^[[0m rpc go-jsonrpc@v0.1.2-0.20201008195726-68c6a2704e49/websocket.go:289 failed to send request {"method": "xrpc.cancel", "id": 2361, "error": "write tcp 127.0.0.1:38560->127.0.0.1:1234: use of closed network connection"}
2020-11-10T13:43:13.669Z ^[[31mERROR^[[0m storageminer storage/wdpost_sched.go:94 ChainNotify error: RPC client error: sendRequest failed: websocket routine exiting
2020-11-10T13:43:13.669Z ^[[31mERROR^[[0m events events/events_called.go:478 getting parent tipset in checkNewCalls: RPC client error: sendRequest failed: websocket routine exiting
2020-11-10T13:43:13.669Z ^[[33mWARN^[[0m events events/events.go:155 headChange failed: RPC client error: sendRequest failed: websocket routine exiting
2020-11-10T13:43:13.669Z ^[[31mERROR^[[0m events events/events_called.go:382 event diff fn failed: RPC client error: sendRequest failed: websocket routine exiting
2020-11-10T13:43:13.669Z ^[[31mERROR^[[0m events events/events_called.go:478 getting parent tipset in checkNewCalls: RPC client error: sendRequest failed: websocket routine exiting
2020-11-10T13:43:13.669Z ^[[33mWARN^[[0m events events/events.go:155 headChange failed: RPC client error: sendRequest failed: websocket routine exiting
2020-11-10T13:43:13.669Z ^[[33mWARN^[[0m events events/events.go:97 listenHeadChanges quit
2020-11-10T13:43:13.669Z ^[[33mWARN^[[0m rpc go-jsonrpc@v0.1.2-0.20201008195726-68c6a2704e49/websocket.go:289 failed to send request {"method": "xrpc.cancel", "id": 2349, "error": "write tcp 127.0.0.1:38560->127.0.0.1:1234: use of closed network connection"}
2020-11-10T13:43:14.669Z ^[[34mINFO^[[0m events events/events.go:104 restarting listenHeadChanges
2020-11-10T13:43:14.669Z ^[[31mERROR^[[0m events events/events.go:95 listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting
2020-11-10T13:43:14.669Z ^[[34mINFO^[[0m events events/events.go:104 restarting listenHeadChanges
2020-11-10T13:43:14.669Z ^[[31mERROR^[[0m events events/events.go:95 listen head changes errored: listenHeadChanges ChainNotify call failed: RPC client error: sendRequest failed: websocket routine exiting