Do not initiate MMP writes while pool is suspended

[ofaaland]

Description

If the pool is suspended, skip the MMP write. At the next cycle of the mmp thread's main loop (which is after a timed wait) check again. In this case the thread will be sleeping most of the time, even though it's polling rather than waiting on the spa_suspend_cv.

I chose to do this rather than waiting on spa_suspend_cv so that the MMP thread does not need to start taking the spa_suspend_lock on every cycle.

Motivation and Context

While the pool is suspended on host A, it may be imported on host B.
If host A continued to write MMP blocks, it would be blindly
overwriting MMP blocks written by host B, and the blocks written by
host A would have outdated txg information.

How Has This Been Tested?

Ran ztest many times. Imported test pools by hand and triggered suspension via zinject; monitored writes via multihost_history and via zdb -lu -VVV.

I did not add a new test to verify this behavior change because the test methods I came up with were not clean.

The best test would be to set failmode to WAIT, import a pool, trigger suspension using zinject, verify that MMP writes stopped via zdb or via multihost_history, and then resume the pool with zpool clear or simply export it. However currently zpool clear and zpool export fail for different reasons. I also attempted modifying ztest to trigger a suspension internally, but it was not clear it would be reliable as ztest does not always tolerate suspension well and a test might pass because ztest crashed, which would not validate mmp.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.
• Change has been approved by a ZFS on Linux member.

[codecov]

Codecov Report

Merging #7182 into master will decrease coverage by 0.07%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7182      +/-   ##
==========================================
- Coverage   76.39%   76.32%   -0.08%     
==========================================
  Files         327      327              
  Lines      103768   103765       -3     
==========================================
- Hits        79278    79203      -75     
- Misses      24490    24562      +72

Flag	Coverage Δ
#kernel	76.06% <100%> (+0.11%)	⬆️
#user	65.62% <100%> (-0.27%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e921f65...61cb7c4. Read the comment docs.

[behlendorf]

LGTM.