Prevent state file corruption and check state file sanity on save

[schmichael]

2 changes to try to address #1367

• Asserts persisted state files are nonzero
• Prevents interleaved state file persist calls which could cause [feature] gracefully handle corrupt startup state #1367

Here's the potential series of events that the lock is designed to guard against:

g1 - r.dirtyCh <- struct{}{}
g2 - r.syncStatus()
g1 - writes temporary file inside of persistState
g2 - truncates temporary file prior to writing to it inside of persistState
g1 - os.Renames truncated temporary file to state file
<process killed>

Using random temporary file names would also work but then we'd have to clean up temporary files leaked during unclean shutdowns.

[dadgar]

LGTM

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.