AR Fixes: task leader handling, restoring, state updating, AR.Destroy deadlocks

[schmichael]

Replacement of #4775 rebased on master post-#4792 merge.

Fixes:

• Fix panic on restore due to nil prev alloc watcher ee06875
• Fix panic when copying a nil TaskHandle 942f1d7
• Port task leader tests from deprecated alloc runner and get them passing (with -race)
• Refactor AR task killing into a single method that kills the leader first
• Fix deadlock in AR.Destroy (see note below)
• Documented and tested state database behavior to make restoring more robust.

Notable Code Change: AllocRunner.Run

Instead of go ar.Run() you should now call ar.Run() as AllocRunner.Run now creates its own goroutine. This allows it to synchronously mark that it has run so that when AR.Destroy is called it knows whether or not to wait for ar.Run() to exit.

Notable Test Change: TestClient Cleanup

Made TestClientConfig and TestClient return a cleanup func to clean up its state and alloc dirs after a test finishes.

[dadgar]

Reviewed alloc_runner and it LGTM. Didn't look at all the files

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.