Fixes in persistent L2ARC

[gamanakis]

Motivation and Context

persist_l2arc_008.ksh test failing occasionally.
Closes #11887.

Description

In l2arc_add_vdev() first decide whether the device is eligibile for
L2ARC rebuild or whole device trim and then add it to the list of cache
devices. Otherwise l2arc_feed_thread() might already start writing on
the device invalidating previous content as l2ad_hand = l2ad_start.
However l2arc_rebuild_vdev() needs the device present in the cache
device list to figure out its l2arc_dev_t. Fix this by moving most of
l2arc_rebuild_vdev() in a new function l2arc_rebuild_dev() which does
not need to search in the cache device list.

In contrast to l2arc_add_vdev() we do not have to worry about
l2arc_feed_thread() invalidating previous content when onlining a
cache device. The device parameters (l2ad*) are not cleared when
offlining the device and writing new buffers will not invalidate
all previous content. In worst case only buffers that have not had
their log block written to the device will be lost.

Retire persist_l2arc_00{4,5,8} tests since they cover code already
covered by the remaining ones. Test persist_l2arc_006 is renamed to
persist_l2arc_004 and persist_l2arc_007 is renamed to persist_l2arc_005.

Fix a typo in persist_l2arc_004, and remove an assertion that is not
always true from l2arc_arcstats_pos. Also update an assertion in
persist_l2arc_005 and explain why in a comment.

How Has This Been Tested?

Locally all l2arc tests in ZTS pass.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have updated tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

Makes sense, thanks for the fix! I also notice we've added l2arc_005_pos and l2arc_007_pos to the "maybe" exception list in zts-report.py. I'm not sure exactly what those occasional failures were, but I wonder if this might help with them.

[gamanakis]

I am going to remove tests persist_l2arc_00{4,5,8} since they cover code which is already tested by the remaining tests.
Test persist_l2arc_006 renamed to persist_l2arc_004, and
persist_l2arc_007 renamed to persist_l2arc_005

The failure in persist_l2arc_007 is absolutely justified, I understand now why. I removed the failing assertion and put in a comment.

00441a1: contains the changes above.