VSHA-536 Zero md Superblocks (#58)

* VSHA-536 Zero `md` Superblocks

This change ensures the RAIDs are eradicated and made unrecognizable
by erasing their superblocks in order to resolve sync timing problems
between Vshasta and metal.

The new logic explicitly stops the `md` devices, wipes their magic bits,
and then eradicates the `md` superblocks on each disk.

During testing of VSHA-536 there was some fiddling with how the RAIDs
were wiped to account for some peculiarities with the timings of how
`virtio` synced and updated the kernel. The changes had been tested on
metal without any observed problems, but in my recent series of tests some
fatal inconsistencies were observed. The `partprobe` was revealing `md`
handles, this caused `mdadm` to restart/resume RAIDs that had been
"nuked" and this in turn caused partitioning to fail.

This change also includes some minor fixes:
- The `wipefs` command for sd/nvme devices was not getting piped to the
  log file.
- The info printed when manually sourcing `/lib/metal-md-lib.sh` in a
  dracut shell is now left justified and aligned by colon.
- The extra `/sbin/metal-md-scan` call in `/sbin/metal-md-disks` is
  removed, it is no longer important shouldn't be invoked every loop
  that calls `/sbin/metal-md-disks`.
- `metal-kdump.sh` no longer invokes `/sbin/metal-md-scan` under
  `root=kdump` because that script is already invoked by the initqueue
  (see `metal-genrules.sh`)
- All initqueue calls to `metal-md-scan` have been changed to `--unique`
  and `--onetime` to ensure they never have an opportunity to run
  forever (as witnessed during a kdump test of the LiveCD)

A note about the dependency on `mdraid-cleanup`:

It turns out relying on `mdraid-cleanup` was a bad idea. The
`mdraid-cleanup` script only stops RAIDs, it does not remove any
superblock (or remove the RAIDs for that matter). This means that
there is a (small) possibility that the RAID and its members still exist
when the `partprobe` command fires. The window of time that this issue
can occur is very small, and varies. VShasta has not hit this error in
the 10-20 deployments it has done in the past 3-4 days.

* Pretty print

Make all URLs printed by dracut-metal-mdsquash contain a commit hash.

Remove the verbose `mount` and `umount` for pretterier output.

Update the `README.adoc` file with a better/verbose explanation of the
wipe process.

90metalmdsquash/metal-genrules.sh

@@ -29,7 +29,7 @@ command -v getarg > /dev/null 2>&1 || . /lib/dracut-lib.sh
 
 case "$(getarg root)" in 
     kdump)
-        /sbin/initqueue --settled /sbin/metal-md-scan
+        /sbin/initqueue --settled --onetime --unique /sbin/metal-md-scan
 
         # Ensure nothing else in this script is invoked in this case.
         exit 0
@@ -59,7 +59,7 @@ case "${metal_uri_scheme:-}" in
         ;;
     '')
         # Boot from block device.
-        /sbin/initqueue --settled /sbin/metal-md-scan
+        /sbin/initqueue --settled --onetime --unique /sbin/metal-md-scan
         ;;
     *)
         warn "Unknown driver $metal_server; metal.server ignored/discarded"

90metalmdsquash/metal-kdump.sh

@@ -33,8 +33,7 @@ command -v _overlayFS_path_spec > /dev/null 2>&1 || . /lib/metal-lib.sh
 
 case "$(getarg root)" in 
     kdump)
-        /sbin/initqueue --settled /sbin/metal-md-scan
-
+
         # Ensure nothing else in this script is invoked in this case.
         exit 0
         ;;

90metalmdsquash/metal-lib.sh

@@ -76,10 +76,26 @@ _load_dracut_dep
 export METAL_LOG_DIR='/var/log/metal'
 mkdir -p $METAL_LOG_DIR
 
+##############################################################################
+# constant: METAL_HASH
+# constant: METAL_DOCS_URL
+#
+# This is the VCS hash for commit that produced this library, it is auto-filled
+# when this module is built into an OS package.
+# This is useful for printing URLs to documentation that are relevant to the
+# library running in an initramFS.
+METAL_HASH='@@metal-hash@@'
+if [[ ${METAL_HASH} =~ 'metal-hash' ]]; then
+    # Default to main if this is running directly out of the repo.
+    METAL_HASH='main'
+fi
+export METAL_HASH
+export METAL_DOCS_URL=https://github.com/Cray-HPE/dracut-metal-mdsquash/tree/${METAL_HASH}
+
 ##############################################################################
 # constant: METAL_DONE_FILE_PAVED
 #
-# This file path present a file that the wipe function creates when it is 
+# This file path present a file that the wipe function creates when it is
 # invoked. The existence of the file implies the wipe code as been invoked,
 # the contents of the file can be interpretted to determine what the wipe
 # function actually did (see func metal_paved).
@@ -169,7 +185,7 @@ metal_die() {
     fi
     type die
     echo >&2 "metal_die: $*"
-    echo >&2 "GitHub/Docs: https://github.com/Cray-HPE/dracut-metal-mdsquash"
+    echo >&2 "GitHub/Docs: ${METAL_DOCS_URL}/README.adoc"
     sleep 30 # Leaving time (30seconds) for console/log buffers to catch up.
     if [ "$_reset" = 1 ]; then
 

90metalmdsquash/metal-md-disks.sh

@@ -39,12 +39,12 @@ disks_exist || exit 1
 # Now that disks exist it's worthwhile to load the libraries.
 command -v pave > /dev/null 2>&1 || . /lib/metal-md-lib.sh
 
-# Wipe; this returns early if a wipe was already done.
-pave
-
 # Check for existing RAIDs
 /sbin/metal-md-scan
 
+# Wipe; this returns early if a wipe was already done.
+pave
+
 # At this point this module is required; a disk must be created or the system has nothing to boot.
 # Die if no viable disks are found; otherwise continue to disk creation functions.
 if [ ! -f /tmp/metalsqfsdisk.done ] && [ "${metal_nowipe}" -eq 0 ]; then
@@ -65,7 +65,7 @@ if [ ! -f /tmp/metalsqfsdisk.done ] && [ "${metal_nowipe}" -eq 0 ]; then
         metal_die "No disks were found for the OS that were [$metal_disk_small] (in bytes) or larger, all were too small or had filesystems present!"
         exit 1
     else
-        echo >&2 "Found the following disks for the main RAID array (qty. [$metal_disks]): [${md_disks[*]}]"
+        echo >&2 "Found the following disk(s) for the main RAID array (qty. [$metal_disks]): [${md_disks[*]}]"
     fi
 fi
 

90metalmdsquash/metal-md-lib.sh

@@ -71,7 +71,7 @@ boot_drive_authority=${boot_fallback#*=}
 [ -z "$boot_drive_authority" ] && boot_drive_authority=BOOTRAID
 case $boot_drive_scheme in
     PATH | path | UUID | uuid | LABEL | label)
-        info "bootloader will be located on ${boot_drive_scheme}=${boot_drive_authority}"
+        printf '%-12s: %s\n' 'bootloader' "${boot_drive_scheme}=${boot_drive_authority}"
         ;;
     '')
         # no-op; drive disabled
@@ -88,7 +88,7 @@ esac
 [ -z "${sqfs_drive_authority}" ] && sqfs_drive_scheme=SQFSRAID
 case $sqfs_drive_scheme in
     PATH | path | UUID | uuid | LABEL | label)
-        info "SquashFS file is on ${sqfs_drive_scheme}=${sqfs_drive_authority}"
+        printf '%-12s: %s\n' 'squashFS' "${sqfs_drive_scheme}=${sqfs_drive_authority}"
         ;;
     *)
         metal_die "Unsupported sqfs-drive-scheme ${sqfs_drive_scheme}\nSupported schemes: PATH, UUID, and LABEL"
@@ -100,7 +100,7 @@ oval_drive_scheme=${metal_overlay%%=*}
 oval_drive_authority=${metal_overlay#*=}
 case "$oval_drive_scheme" in
     PATH | path | UUID | uuid | LABEL | label)
-        info "Overlay is on ${oval_drive_scheme}=${oval_drive_authority}"
+        printf '%-12s: %s\n' 'overlay' "${oval_drive_scheme}=${oval_drive_authority}"
         ;;
     '')
         # no-op; disabled
@@ -218,10 +218,10 @@ add_overlayfs() {
     [ -f /tmp/metalovalimg.done ] && return
     local mpoint=/metal/ovaldisk
     mkdir -pv ${mpoint}
-    if ! mount -v -n -t xfs /dev/md/ROOT "$mpoint"; then
+    if ! mount -n -t xfs /dev/md/ROOT "$mpoint"; then
 
         # try shasta-1.3 formatting or die.
-        mount -v -n -t ext4 /dev/md/ROOT "$mpoint" \
+        mount -n -t ext4 /dev/md/ROOT "$mpoint" \
             || metal_die "Failed to mount ${oval_drive_authority} as xfs or ext4"
     fi
 
@@ -233,7 +233,7 @@ add_overlayfs() {
         "${mpoint}/${live_dir}/${metal_overlayfs_id}" \
         "${mpoint}/${live_dir}/${metal_overlayfs_id}/../ovlwork"
     echo 1 > /tmp/metalovalimg.done && info 'OverlayFS is ready ...'
-    umount -v ${mpoint}
+    umount ${mpoint}
 }
 
 ###############################################################################
@@ -261,13 +261,13 @@ fetch_sqfs() {
 
         # Mount read-only to prevent harm to the device; we literally just need to pull the files off it.
         mkdir -vp /tmp/source
-        mount -v  -n -o ro -L "$metal_local_url_authority" /tmp/source || metal_die "Failed to mount $metal_local_url_authority from $metal_server"
+        mount -n -o ro -L "$metal_local_url_authority" /tmp/source || metal_die "Failed to mount $metal_local_url_authority from $metal_server"
         (
             set -e
             cd "$1"
             cp -pv "/tmp/source/${metal_local_dir#//}/${squashfs_file}" . && echo "copied ${squashfs_file} ... " > debug_log
         ) || warn 'Failed to copy ; may retry'
-        umount -v /tmp/source
+        umount /tmp/source
     fi
     if [ -f "$1/${squashfs_file}" ]; then
         echo 1 > /tmp/metalsqfsimg.done
@@ -316,10 +316,10 @@ add_sqfs() {
         info "URI host ${uri_host} responds ... "
     fi
     mkdir -pv $sqfs_store
-    if mount -v -n -t xfs /dev/md/SQFS $sqfs_store; then
+    if mount -n -t xfs /dev/md/SQFS $sqfs_store; then
         mkdir -pv "$sqfs_store/$live_dir"
         fetch_sqfs "$sqfs_store/$live_dir" || metal_die 'Failed to fetch squashFS into squashFS storage!'
-        umount -v $sqfs_store
+        umount $sqfs_store
     else
         # No RAID mount, issue warning, delete mount-point and return
         metal_die "Failed to mount /dev/md/SQFS at $sqfs_store"
@@ -348,7 +348,7 @@ pave() {
     # the original run.
     if [ -f "$METAL_DONE_FILE_PAVED" ]; then
         echo "${FUNCNAME[0]} already done" >>"$log"
-        echo "wipe done file already exists ($METAL_DONE_FILE_PAVED); not wiping disks"
+        echo "wipe 'done file' already exists ($METAL_DONE_FILE_PAVED); not wiping disks"
         return 0
     fi
     {
@@ -361,17 +361,19 @@ pave() {
     } >>"$log" 2>&1
 
     if [ "$metal_nowipe" -ne 0 ]; then
-        echo "${FUNCNAME[0]} skipped: metal.no-wipe=${metal_nowipe}" >>$log
         warn 'local storage device wipe [ safeguard: ENABLED  ]'
-        warn 'local storage devices will not be wiped.'
+        warn "local storage devices will not be wiped (${METAL_DOCS_URL}#metalno-wipe)"
         echo 0 > "$METAL_DONE_FILE_PAVED" && return 0
     else
         warn 'local storage device wipe [ safeguard: DISABLED ]'
+        warn "local storage devices WILL be wiped (${METAL_DOCS_URL}#metalno-wipe)"
     fi
-    warn 'local storage device wipe commencing (USB devices are ignored) ...'
+    warn 'local storage device wipe commencing ...'
+    warn "local storage device wipe ignores USB devices and block devices smaller than [$metal_ignore_threshold] bytes."
 
     warn 'nothing can be done to stop this except one one thing ...'
-    warn "... power this node off within the next [$metal_wipe_delay] seconds to prevent any and all operations ..."
+    warn "power this node off within the next [$metal_wipe_delay] second(s) to cancel."
+    warn "NOTE: this delay can be adjusted, see: ${METAL_DOCS_URL}#metalwipe-delay"
     while [ "${metal_wipe_delay}" -ge 0 ]; do
         [ "${metal_wipe_delay}" = 2 ] && unit='second' || unit='seconds'
         sleep 1 && local metal_wipe_delay=$((${metal_wipe_delay} - 1)) && echo "${metal_wipe_delay} $unit"
@@ -402,28 +404,27 @@ pave() {
     doomed_raids="$(lsblk -l -o NAME,TYPE | grep raid | sort -u | awk '{print "/dev/"$1}' | tr '\n' ' ' | sed 's/ *$//')"
     warn "local storage device wipe is targeting the following RAID(s): [$doomed_raids]"
     for doomed_raid in $doomed_raids; do
-        wipefs --all --force "$doomed_raid" >>"$log" 2>&1
+        {
+        wipefs --all --force "$doomed_raid"
+        mdadm --stop "$doomed_raid"
+        } >>"$log" 2>&1
     done
 
     # 3. NUKE BLOCKs
     # Wipe each selected disk and its partitions.
     doomed_disks="$(lsblk -b -d -l -o NAME,SUBSYSTEMS,SIZE | grep -E '('"$metal_subsystems"')' | grep -v -E '('"$metal_subsystems_ignore"')' | sort -u | awk '{print ($3 > '$metal_ignore_threshold') ? "/dev/"$1 : ""}' | tr '\n' ' ' | sed 's/ *$//')"
     warn "local storage device wipe is targeting the following block devices: [$doomed_disks]"
     for doomed_disk in $doomed_disks; do
-        wipefs --all --force "$doomed_disk"*
+        {
+            mdadm --zero-superblock "$doomed_disk"*
+            wipefs --all --force "$doomed_disk"*
+        } >>"$log" 2>&1
     done
 
-    # 4. Cleanup mdadm
-    # Now that the signatures and volume groups are wiped/gone, mdraid-cleanup can mop up and left
-    # over /dev/md handles.
-    {
-        lsblk
-        mdraid-cleanup
-        lsblk
-    } >>"$log" 2>&1
+    _trip_udev
 
-    # 5. Notify the kernel of the partition changes
-    # NOTE: This could be done in the same loop that we wipe devices, however mileage has varied.
+    # 4. Notify the kernel of the partition changes
+    # NOTE: This could be done in the same loop where we wipe devices, however mileage has varied.
     #       Running this as a standalone step has had better results.
     for doomed_disk in $doomed_disks; do
         {
@@ -433,8 +434,6 @@ pave() {
         } >>"$log" 2>&1
     done
 
-    _trip_udev
-
     warn 'local storage disk wipe complete' && echo 1 > "$METAL_DONE_FILE_PAVED"
     {
         mount -v

Makefile

@@ -46,7 +46,7 @@ rpm_package_source:
 	tar --transform 'flags=r;s,^,/${NAME}-${VERSION}/,' --exclude .git --exclude dist -cvjf $(SOURCE_PATH) .
 
 rpm_build_source:
-	BUILD_METADATA=$(BUILD_METADATA) rpmbuild --nodeps -ts $(SOURCE_PATH) --define "_topdir $(BUILD_DIR)"
+	rpmbuild --nodeps -ts $(SOURCE_PATH) --define "_topdir $(BUILD_DIR)"
 
 rpm_build:
-	BUILD_METADATA=$(BUILD_METADATA) rpmbuild --nodeps -ba $(SPEC_FILE) --define "_topdir $(BUILD_DIR)"
+	rpmbuild --nodeps -ba $(SPEC_FILE) --define "_topdir $(BUILD_DIR)"