fix: fsm: shutdown removed sectors FSMs

[magik6k]

Related Issues

Fixes #9968

Proposed Changes

• Don't log stuff to removed sectors logs
• Shutdown statemachines for removed sectors
• Add an envvar to keep removed sector FSMs running - LOTUS_KEEP_REMOVED_FSM_ACTIVE, this makes it possible to move sectors out of the Removed state with update-state in the extremely unlikely case when that's needed.

Additional Info

Could use testing

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
  • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green

[rjan90]

Can confirm that the PR does not log stuff for removed sectors anymore.

Before the PR, if one tried to remove an already removed sector, i.e lotus-miner sectors remove --really-do-it 49, that sectors log would end up adding a state machine error in the log:

12.     2023-03-05 10:16:49 +0100 CET:  [event;sealing.SectorRemove]    {"User":{}}
13.     2023-03-05 10:16:49 +0100 CET:  [event;sealing.SectorRemoved]   {"User":{}}
14.     2023-03-05 10:16:49 +0100 CET:  [error;<nil>]   state machine error: %!s(<nil>)
15.     2023-03-05 10:17:08 +0100 CET:  [event;sealing.SectorRemove]    {"User":{}}
16.     2023-03-05 10:17:08 +0100 CET:  [error;<nil>]   state machine error: %!s(<nil>)

After cherry-picking the PR, when one tries to remove an already removed sector, it just error with normal shutdown of the state machine, and no entires are added to the log:

lotus-miner sectors remove --really-do-it 49
ERROR: normal shutdown of state machine

[ZenGround0]

Im seeing the splitstore test failures that I thought I fixed with #10366, investigating

--edit--
PR CI is just out of date, after rebase tests pass locally