turning OFF encryption for my backup edge case

[tinker2much]

I'm an enthusiastic user of both your sdm (for creating/burning/encrypting images) and RonR's image-backup (part of Image File Utilities - see here) (for creating and incrementally updating backup images). The images created by image-backup are normal images suitable for being burned by Raspberry Pi Imager.

My desired workflow is:

1. create and burn (and sometimes encrypt) images using sdm
2. go on about my business adding and updating and changing stuff
3. back them up periodically using IFU
4. if recreating an image from scratch, use sdm
5. if restoring from backups, use Imager.

My problem is that neither image-backup nor Imager currently have support for encrypted rootfs. I have worked on some hacks to image-backup to allow it to back up the files from an encrypted rootfs and write them (unencrypted) to a normal pi image. So far so good, a step on the way.
BUT, If I were to restore such images, even if they burned correctly, the software or settings are appropriate for the previously encrypted roots, while the re-burned files and rootfs are NOT encrypted.

My initial desire is just for enough knowledge to go into the backed up image and fiddle it's configuration so as to TURN OFF that expectation of encryption, so when I burn the image the system is functional, though un-encrypted. Then, if desired, I can use sdm-cryptconfig to actually re-encrypt the rootfs and files. Not optimal, but it's a start (Until everyone fully supports encrypted rootfs.)

In glancing through sdm-cryptconfig, I see the following things getting set for encryption:

1. update the root statement in cmdline.txt
2. update /etc/fstab
3. update /etc/crypttab

I imagine (perhaps optimistically), that I just have to reverse these changes to accomplish what I want. What am I missing?

[gitbls]

I think you've got it.

1. Restore root= statement to have a PARTUUID=uuid target
2. Replace the /dev/mapper/$cryptdevice (where $cryptdevice is cryptroot unless you've changed it) with the correct PARTUUiD
3. Comment out the line in /etc/crypttab

LMK if you run into anything that sdm can help with. For instance, would it be helpful for you if sdm saved pre-crypt cmdline.txt and fstab?

[tinker2much]

what about
/etc/cryptsetup-initramfs/conf-hook

[gitbls]

That file is part of the package cryptsetup-initramfs. On my system it looks like it has no useful content in it (everything commented out).

Is that the case on your system as well? If so, you can ignore it, since it does nothing. If yours has uncommented content, please post it.

[tinker2much]

Also commented here, thanks. I hadn't noticed there were no live lines at all.

[tinker2much]

I think this is all I need to do to the image before burning it back. It seems to do what I want on some test files taken from a system encrypted by sdm-cryptconfig. (See below, encrypted and non-encrypted versions)

# script to unconfigure encryption on an image that had been
#   encrypted by sdm-cryptconfig and backed up by image-backup

# usage: encryptoff fstab cmdline.txt crypttab
#   the originals will be saved as .encr
FSTAB=$1
CMDLINE=$2
CRYPTTAB=$3

# get the boot partition's PARTUUID from fstab and replace 
#   the 01 with a 02 to make a root partition PARTUUID
PARTUUID_B=`grep boot $FSTAB | cut -f1 -d " "`
PARTUUID_P=${PARTUUID_B%01}02

# replace the "/dev/mapper/cryptroot" in fstab with the real rootfs PARTUUID
sed -i.encr "s#/dev/mapper/cryptroot#$PARTUUID_P#" $FSTAB 

# replace the "/dev/mapper/cryptroot" in cmdline.txt with the real rootfs PARTUUID
# ALSO remove the "cryptdevice=/dev/sda2:cryptroot" phrase
sed -i.encr -e "s#/dev/mapper/cryptroot#$PARTUUID_P#" -e "s# cryptdevice=.*:cryptroot##" $CMDLINE

# comment out the cryptroot line in /etc/crypttab
sed -i.encr 's/^cryptroot/#cryptroot/' $CRYPTTAB

fstab: 

proc            /proc           proc    defaults          0       0
PARTUUID=6ac188ed-01  /boot/firmware  vfat    defaults          0       2
/dev/mapper/cryptroot /              ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

proc            /proc           proc    defaults          0       0
PARTUUID=6ac188ed-01  /boot/firmware  vfat    defaults          0       2
PARTUUID=6ac188ed-02 /              ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

cmdline.txt: 

video=HDMI-A-1=1920x1080@60D console=serial0,115200 console=tty1 root=/dev/mapper/cryptroot rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US vc4.force_hotplug=1 rw cryptdevice=/dev/sda2:cryptroot quiet

video=HDMI-A-1=1920x1080@60D console=serial0,115200 console=tty1 root=PARTUUID=6ac188ed-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US vc4.force_hotplug=1 rw quiet

Also, what about the "rw" in cmdline.txt?

crypttab:

# <target name>	<source device>		<key file>	<options>
cryptroot	/dev/sda2  none  luks

# <target name>	<source device>		<key file>	<options>
#cryptroot	/dev/sda2  none  luks

[gitbls]

If you want to be rigorous, seems like you should remove it. I would, since it didn't have it before.

[tinker2much]

New version, also removing "rw":

pi@pi5:~/Documents/encryptfix $ cat encryptoff 
# script to unconfigure encryption on an image that had been
#   encrypted by sdm-cryptconfig and backed up by image-backup

# usage: encryptoff fstab cmdline.txt crypttab
#   the originals will be saved as .encr
FSTAB=$1
CMDLINE=$2
CRYPTTAB=$3

# get the boot partition's PARTUUID from fstab and replace 
#   the 01 with a 02 to make a root partition PARTUUID
PARTUUID_B=`grep boot $FSTAB | cut -f1 -d " "`
PARTUUID_P=${PARTUUID_B%01}02

# replace the "/dev/mapper/cryptroot" in fstab with the real rootfs PARTUUID
sed -i.encr "s#/dev/mapper/cryptroot#$PARTUUID_P#" $FSTAB 

# replace the "/dev/mapper/cryptroot" in cmdline.txt with the real rootfs PARTUUID
# ALSO remove the "cryptdevice=/dev/sda2:cryptroot" phrase
# ALSO remove the "rw" phrase
sed -i.encr -e "s#/dev/mapper/cryptroot#$PARTUUID_P#" -e "s# cryptdevice=.*:cryptroot##" -e "s# rw##" $CMDLINE

# comment out the cryptroot line in /etc/crypttab
sed -i.encr 's/^cryptroot/#cryptroot/' $CRYPTTAB

Just the cmdline.txt, before and after:

pi@pi5:~/Documents/encryptfix $ cat cmdline.txt.encr cmdline.txt 

video=HDMI-A-1=1920x1080@60D console=serial0,115200 console=tty1 root=/dev/mapper/cryptroot rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US vc4.force_hotplug=1 rw cryptdevice=/dev/sda2:cryptroot quiet

video=HDMI-A-1=1920x1080@60D console=serial0,115200 console=tty1 root=PARTUUID=6ac188ed-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US vc4.force_hotplug=1 quiet

[gitbls]

The sed -i.encr is both amusing and interesting. I know what i, n, r, and e are, but what are . and c?

[tinker2much]

-i is usually just for in-place updating the file, but any characters after that mean that there will be a backup made of the original with those chars as the suffix. So this is really saying "backup the original with the suffix ".encr"

       -i[SUFFIX], --in-place[=SUFFIX]

              edit files in place (makes backup if SUFFIX supplied)

... and I just got why that's amusing. Finally. I'm also amused.

[gitbls]

DOH! Although I read the man page, I didn't read the text for the -i switch, obviously. 😊

[tinker2much]

I had been figuring that I would run this encryptoff script as a fix on the unecncrypted-but-configured-for-encryption backup image, vis sdm --explore, because how could I change them on the actually encrypted live system without messing that up? Now, I wonder if I could run it on the live encrypted system just before backing it up? Then the backup would be ready to burn, and I could un-un-encrypt it (!) via a mirror-image script called encrypton? This would work if all of these config files are really just used at boot, and are just sitting there while running, and therefore could be changed, and changed back, on the live system, without consequence?

[gitbls]

I like your original approach better, but maybe I just misunderstood it. I generally try to not muck with the config for my running systems, primarily because...they're running and happy, so why tempt fate?

[tinker2much]

I'm less accustomed to doing things to an image, than I am doing things to a running system. But I take your point about not messing with the thing that's working.

[tinker2much]

I've finished making a test system for encryption on an sd card (my live pi5 is on an ssd).
I backed it up, encrypted it via sdm, then compared the running system to the backup using RonR's image-compare. There were 500+ differences noted, but most were either unrelated (things I had changed outside of encryption) or seemed innocuous (like the man files for newly installed encryption tools). I pruned the list to just over 100 items - could you please peruse the list to see if you think I need to touch anything else in the "encryptoff" script?
image-comparex-pruned.txt

[tinker2much]

See the man page for rsync and the --itemize-changes option

[gitbls]

That said, looking through the list, all except for one appear to be crypt-related. That one is: etc/rc.local

What changed in that?

[gitbls]

--itemize-changes...Thanks! I needed to learn something new today!

[tinker2much]

I don't see anything changed in /etc/rc.local and it doesn't mention anything about encryption.

[gitbls]

Then I think you're good to go.

[tinker2much]

I have my backup image available on a network drive. I run sudo sdm --explore . I intend to run my encryptoff script, pointing it to /etc/fstab, /boot/firmware/cmdline.txt and /etc/crypttab. But there's nothing at /boot/firmware. No files at all. What am I doing? The cmdline.txt file at just /boot is the new "what you want is not here" model, not the previous link, so that doesn't help.

[gitbls]

Curious. What do you get from this?

sudo sdm --mount /path/to/img
df
ls -l /boot
ls -l /boot/firmware

And then type 'exit' to leave the mount command

[tinker2much]

I assume you DIDN'T mean /boot, but did mean just boot, which off the current directory of /mnt/sdm yields /mnt/sdm/boot?

pi@pi5:/mnt/backup $ sudo sdm --mount pi5-bookworm64.img
* IMG 'pi5-bookworm64.img' mounted on /mnt/sdm
** BE VERY CAREFUL!! **
** Precede all path references by '/mnt/sdm' or you WILL modify your running system **
** Use 'exit' to Exit the bash shell and unmount the IMG
root@pi5:/mnt/sdm# ls
bin   dev  home        initrd.img.old  lib64	   media  opt	root  sbin  sys  usr  vmlinuz
boot  etc  initrd.img  lib	       lost+found  mnt	  proc	run   srv   tmp  var  vmlinuz.old
root@pi5:/mnt/sdm# ls -l boot
total 105052
-rw-r--r-- 1 root root       92 Jan 29 11:40 cmdline.txt
-rw-r--r-- 1 root root   230365 Oct  5 06:33 config-6.1.0-rpi4-rpi-2712
-rw-r--r-- 1 root root   230343 Oct  5 06:33 config-6.1.0-rpi4-rpi-v8
-rw-r--r-- 1 root root   230511 Jan 25 13:52 config-6.1.0-rpi8-rpi-2712
-rw-r--r-- 1 root root   230500 Jan 25 13:52 config-6.1.0-rpi8-rpi-v8
-rw-r--r-- 1 root root       91 Jan 29 11:40 config.txt
drwxr-xr-x 2 root root     4096 Dec 31  1969 firmware
-rw-r--r-- 1 root root 17894407 Jan 30 07:55 initrd.img-6.1.0-rpi4-rpi-2712
-rw-r--r-- 1 root root 17891442 Jan 30 07:54 initrd.img-6.1.0-rpi4-rpi-v8
-rw-r--r-- 1 root root 17922430 Jan 30 07:53 initrd.img-6.1.0-rpi8-rpi-2712
-rw-r--r-- 1 root root 17923018 Jan 30 07:53 initrd.img-6.1.0-rpi8-rpi-v8
lrwxrwxrwx 1 root root       17 Oct  9 22:38 overlays -> firmware/overlays
-rw-r--r-- 1 root root       83 Oct  5 06:33 System.map-6.1.0-rpi4-rpi-2712
-rw-r--r-- 1 root root       83 Oct  5 06:33 System.map-6.1.0-rpi4-rpi-v8
-rw-r--r-- 1 root root       83 Jan 25 13:52 System.map-6.1.0-rpi8-rpi-2712
-rw-r--r-- 1 root root       83 Jan 25 13:52 System.map-6.1.0-rpi8-rpi-v8
-rw-r--r-- 1 root root  8728960 Oct  5 06:33 vmlinuz-6.1.0-rpi4-rpi-2712
-rw-r--r-- 1 root root  8726865 Oct  5 06:33 vmlinuz-6.1.0-rpi4-rpi-v8
-rw-r--r-- 1 root root  8753764 Jan 25 13:52 vmlinuz-6.1.0-rpi8-rpi-2712
-rw-r--r-- 1 root root  8751247 Jan 25 13:52 vmlinuz-6.1.0-rpi8-rpi-v8
root@pi5:/mnt/sdm# ls -l boot/firmware
total 0
root@pi5:/mnt/sdm# 

[gitbls]

You're right. I meant /mnt/sdm/boot and /mnt/sdm/boot/firmware.

[tinker2much]

I'm very puzzled. I also tried to mount the boot partition from the backup image using RonR's image-mount, and I get nada, nothing at all. Not even what I saw using your --mount command.

Does this mean the image is bad? Which could certainly be true, given my hacking on the image-backup software...

[tinker2much]

When I use his image-mount to mount other bookworm backup images (before encryption) they all show 300+ files. So I assume my backup hacking is lacking.

[tinker2much]

I'm the guy spinning plates on sticks - I think the script is spinning well, but oops the backup is tipping, wait, I bet the restore won't work...

[gitbls]

I guess what I'd do at this point is recreate the scenario and after each step look at /boot and /boot/firmware to see where it's going awry. Once you know where to look, it shouldn't take too long to sort out how you duffed it.

Some days I bounce from one duff to the next 😂

[tinker2much]

Well. I'm puzzled, and I'm pleased. I tried debugging the backup process a little more, but I couldn't see anything wrong (I'm not the best at debugging other people's code). Despite earlier problems seeing boot or boot/firmware on the backed up image, I just said WTH and burned it to a usb-stick using imager, plugged it into a handy pi4 and it booted, though I got an error, as expected, because though it's actually unecncrypted it was still configured for encryption. Shut it down, plugged it into my pi desktop machine, both bootfs and rootfs automounted off /media/pi, thus giving me access to fstab, cmdline.txt, and crypttab. I pointed my encryptoff script at those locations, it ran and did its thing. Shut down, slugged it back into the pi4, and it booted fully, though with a few complaints, and I'm writing this from that machine. I took some screenshots as it was booting, so I'll go back and see what it was complaining about. But getting a desktop and networking and a browser going seems like it must be close to good.

[gitbls]

Sounds like great progress! Your proof of concept works, now you just need to codify it so you can check the box and life is good 😎

[tinker2much]

The error at top of screen makes me wonder whether there's some remnant of encrypted settings left, even if things seem to work:

The error near the bottom seems less likely to be related to encryption:

... and I don't know what to make of it:

pi@pi5:~/bin $ systemctl status NetworkManager-wait-online.service
× NetworkManager-wait-online.service - Network Manager Wait Online
     Loaded: loaded (/lib/systemd/system/NetworkManager-wait-online.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-01-31 11:23:08 CST; 2h 27min ago
       Docs: man:nm-online(1)
   Main PID: 800 (code=exited, status=1/FAILURE)
        CPU: 195ms

Jan 30 19:10:28 pi5 systemd[1]: Starting NetworkManager-wait-online.service - Network Manager Wait Online...
Jan 31 11:23:08 pi5 systemd[1]: NetworkManager-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 11:23:08 pi5 systemd[1]: NetworkManager-wait-online.service: Failed with result 'exit-code'.
Jan 31 11:23:08 pi5 systemd[1]: Failed to start NetworkManager-wait-online.service - Network Manager Wait Online.

[gitbls]

If I understand correctly, you use image- backup to make a copy of the running system, while it's unencrypted. Correct?

Then, you copy the resulting IMG (or disk, doesn't matter for purposes of this) to something that you want to boot from.

After that you want to enable that resulting disk to boot without encryption.

So, you removed all the stuff you mentioned above. Perfect.

Did I get all that correctly?

I missed the part about wanting to boot that disk. To make that work without the cryptsetup message you're seeing, you need to rebuild initramfs, but before that remove the crypt stuff that gets added to initramfs:

• /etc/kernel/postinst.d/initramfs-rebuild
• /etc/initramfs-tools/hooks/luks_hooks
• /etc/initramfs-tools/modules (although this is probably OK to leave)

After that, do sudo update-initramfs -u to rebuild initramfs

I think that should solve it.

Sorry I missed the boot implications. I've been deep in a twisty maze of passages and going a bit 🥴

[tinker2much]

I'm very glad that I've got some end-to-end way to use my customary backup and restore tools on an encrypted rootfs - even though it's still a little twisty itself. I need to be able to restore my now-encrypted pi5 desktop machine, where lots of things change all the time, in case the media fries out or if I otherwise lose it. Namely, I can encrypt an existing pi using sdm-cryptconfig, continue doing incremental backups to an unencrypted existing backup image using a (slightly) hacked image-backup script, burn the backup image to a device, mount it on a running pi, run my encryptoff script to make the franken-thing think it's not encrypted and at least runnable, and then I can boot it. If I need to re-encrypt the result, back to sdm-cryptconfig, and Voila! I'm back in business.

Thank you for your help and review while I figured this out, even though it was never sdm's issue at all, just follow-on stuff after sdm did its encrypting completely correctly. I am tired enough at the moment that I'll hold off on implementing your latest suggestions for a completely clean encryption-removal solution. What I've got at least works, and it only seems to add a 20 second or so wait during the boot.

To be clear, this is a different workflow than creating or re-creating initial images, or knocking out single-purpose machines, where sdm itself is my preferred solution. And most of my pi's WON'T be encrypted, so the whole create/backup/restore process there is as simple as always.

[gitbls]

I'm very glad that I've got some end-to-end way to use my customary backup and restore tools on an encrypted rootfs - even though it's still a little twisty itself. I need to be able to restore my now-encrypted pi5 desktop machine, where lots of things change all the time, in case the media fries out or if I otherwise lose it. Namely, I can encrypt an existing pi using sdm-cryptconfig, continue doing incremental backups to an unencrypted existing backup image using a (slightly) hacked image-backup script, burn the backup image to a device, mount it on a running pi, run my encryptoff script to make the franken-thing think it's not encrypted and at least runnable, and then I can boot it. If I need to re-encrypt the result, back to sdm-cryptconfig, and Voila! I'm back in business.

Excellent!

Thank you for your help and review while I figured this out, even though it was never sdm's issue at all, just follow-on stuff after sdm did its encrypting completely correctly. I am tired enough at the moment that I'll hold off on implementing your latest suggestions for a completely clean encryption-removal solution. What I've got at least works, and it only seems to add a 20 second or so wait during the boot.

Absolutely! Glad to help.

To be clear, this is a different workflow than creating or re-creating initial images, or knocking out single-purpose machines, where sdm itself is my preferred solution. And most of my pi's WON'T be encrypted, so the whole create/backup/restore process there is as simple as always.

👍