VSHA-536 Zero md Superblocks

[rustydb]

Summary and Scope

• Relates to: VSHA-536 Support virtio #56

Issue Type

• Bugfix Pull Request

This change ensures the RAIDs are eradicated and made unrecognizable by erasing their superblocks in order to resolve sync timing problems between Vshasta and metal.

The new logic explicitly stops the md devices, wipes their magic bits, and then eradicates the md superblocks on each disk.

During testing of VSHA-536 there was some fiddling with how the RAIDs were wiped to account for some peculiarities with the timings of how virtio synced and updated the kernel. The changes had been tested on metal without any observed problems, but in my recent series of tests some fatal inconsistencies were observed. The partprobe was revealing md handles, this caused mdadm to restart/resume RAIDs that had been "nuked" and this in turn caused partitioning to fail.

This change also includes some minor fixes:

• The wipefs command for sd/nvme devices was not getting piped to the log file.
• The info printed when manually sourcing /lib/metal-md-lib.sh in a dracut shell is now left justified and aligned by colon.
• The extra /sbin/metal-md-scan call in /sbin/metal-md-disks is removed, it is no longer important shouldn't be invoked every loop that calls /sbin/metal-md-disks.
• metal-kdump.sh no longer invokes /sbin/metal-md-scan under root=kdump because that script is already invoked by the initqueue (see metal-genrules.sh)
• All initqueue calls to metal-md-scan have been changed to --unique and --onetime to ensure they never have an opportunity to run forever (as witnessed during a kdump test of the LiveCD)

A note about the dependency on mdraid-cleanup:

It turns out relying on mdraid-cleanup was a bad idea. The mdraid-cleanup script only stops RAIDs, it does not remove any superblock (or remove the RAIDs for that matter). This means that there is a (small) possibility that the RAID and its members still exist when the partprobe command fires. The window of time that this issue can occur is very small, and varies. VShasta has not hit this error in the 10-20 deployments it has done in the past 3-4 days, my 50+ boots I tested didn't hit this, but the past 10 NCN boots I just attempted hit this almost every time.

Prerequisites

• I have included documentation in my PR (or it is not required)
• I tested this on internal system (if yes, please include results or a description of the test)
• I tested this on a vshasta system (if yes, please include results or a description of the test)

Idempotency

Risks and Mitigations