Verify dRAID empty sectors

[behlendorf]

Motivation and Context

This issue was discovered when investigating a checksum error
which was reported against the wrong leaf vdev when manually
testing corrupting a vdev by overwriting it with dd.

Description

Verify that all empty sectors are zero filled before using them to
calculate parity. Failure to do so can result in incorrect parity
columns being generated and written to disk if the contents of an
empty sector are non-zero. This was possible because the checksum
only protects the data portions of the buffer, not the empty sector
padding.

This issue has been addressed by updating raidz_parity_verify() to
check that all dRAID empty sectors are zero filled. Any sectors
which are non-zero will be fixed, repair IO issued, and a checksum
error logged. They can then be safely used to verify the parity.

This specific type of damage is unlikely to occur since it requires
a disk to have silently returned bad data, for an empty sector, while
performing a scrub. However, if a pool were to have been damaged
in this way, scrubbing the pool with this change applied will repair
both the empty sector and parity columns as long as the data columns
are intact as verify by the checksum. Checksum errors will be reported
in the zpool status output for any repairs which are made.

How Has This Been Tested?

Manually constructed a pool, selectively damaged an empty sector,
scrubbed the pool, and verified via instrumentation that:

1. the empty sector we detected and repaired, and
2. the parity columns were verified by the scrub, and
3. checksum errors were only reported for the correct vdev.

The following steps can be used.

truncate -s 1G /var/tmp/vdev{1..9}
zpool create tank draid /var/tmp/vdev*

# Damage an empty sector and verify it's non-zero.
dd if=/dev/urandom of=/var/tmp/vdev1 bs=512 count=1 seek=8192 conv=notrunc
dd if=/var/tmp/vdev1 bs=512 count=1 skip=8192 | hexdump

# Scrub the pool, verify the empty sector was repaired and checksum errors reported correctly
zpool scrub tank
dd if=/var/tmp/vdev1 bs=512 count=1 skip=8192 | hexdump
zpool status -v tank

It also happens to be the case the existing zpool_events_error.ksh
test case reliably hits this issue when extended to verify dRAID pools.
The blocks are reliably laid out on disk such that we're highly likely
to overwrite an empty sector, and the test case strictly checks for an
exact number of checksum errors to be reported against specified
vdevs.

Furthermore, the existing redundancy_draid* tests provide additional
coverage for these code path. I've run the "redundancy" test group 100
times and observed no failures. Between these tests case, and the manual
testing which was done, I don't think there's a need to add a new test
case.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[bwatkinson]

This works well with the update to FreeBSD's abd_allocate_zero_scatter() since that now calls bzero() and is used when allocating the dRAID map for writes. This is also make sure that any previous FreeBSD dRAID writes that were not zero'ing out the skip sectors will do so during a scrub.

[mmaybee]

Looks good!