Unable to get sdm to customize my image

[framps]

Not sure what I did wrong but my image doesn't do any customization when I boot it.

That's what I did:

sudo sdm --customize 2023-12-11-raspios-bookworm-armhf-lite.img --plugin user:"adduser=pi|prompt" --plugin disables:piwiz --plugin L10n:host --plugin network:"wpa=/path/to/wpa_supplicant.conf" --restart
sudo sdm --burn /dev/sde --hostname mypi1 --expand-root 2023-12-11-raspios-bookworm-armhf-lite.img 

When the image starts I'm prompted for keyboard layout and username and password 😢

According the customization messages localization and user definition should have been done.

 Plugin user: Keys/values found:
   adduser: pi
   prompt: 
% Plugin user: User 'pi' home directory '/home/pi' already exists
* Plugin user: Complete Phase 0
> Run Plugin 'disables' (/mnt/sdm/usr/local/sdm/plugins/disables) Phase 0 with arguments: 
  'piwiz'
* Plugin disables: Start Phase 0
> Plugin disables: Keys/values found:
   piwiz: 
* Plugin disables: Complete Phase 0
> Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host'
* Plugin L10n: Start Phase 0
> Plugin L10n: Keys/values found:
   host: 
> Plugin L10n: Read L10n configuration from host system
> Plugin L10n: Load Localization (L10N) settings from running system
> Plugin L10n:   Keymap:       de
> Plugin L10n:   Locale:       en_US.UTF-8
> Plugin L10n:   Timezone:     Europe/Berlin
> Plugin L10n:   WiFi Country: 
> Plugin L10n: Save L10n configuration

Would be great to get a hint what I'm doing wrong.

[gitbls]

Can you redo the burn command and add the switches --expand-root and --regen-ssh-host-keys? (only need to redo the burn and not the customize).

If that doesn't do it, please post the complete /etc/sdm/history from in the burned /dev/sde. You can grab that by doing sudo sdm --explore /dev/sde and scp-ing the file somewhere (or cut/paste of course).

[framps]

I just noticed ssh was not enabled even it's documented it's enabled all the time. I also noticed the hostname was updated to mypi1.

systemd complained about umnet condition check for !/etc/sshd/sshd/sshd_not_to_be_run. I deleted the file and then was able to start sshd to copy the history file to my desktop.

history.log

[gitbls]

Thanks for the history file. I see you redid the burn with the switches I suggested. Did that fix getting the keyboard layout and add user prompts?

When you say "the hostnme was updated to mypi1", you're referring to the Pi, which is correct, since that's what you asked it to do. Am I understanding your comment correctly?

At the moment, sdm assumes that if you use the network plugin you'll enable ssh yourself. I see now that this might not be a valid assumption and will look into refining it.

That said, your invocation `--plugin network:wpa=/path/to/wpa_supplicant.conf is incorrect because:

2023-12-17 21:12:04 ? Plugin network: wpa file '/path/to/wpa_supplicant.conf' not found
2023-12-17 21:12:04 ? Plugin 'network' exited with failure

Also note that even if the file /path/to/wpa_supplicant.conf did exist, it wouldn't configure your network on Bookworm, since you're using NetworkManager.

It appears from the log that this error (non-existent wpa_supplicant.conf) did stop the customize (there's more that sdm does after this) but appears that you overlooked it. But, since you included the network plugin, sdm (unwisely) decided it didn't need to enable ssh for you.

If you redo the customize with the exact same command line but excluding the network plugin, ssh will be enabled and everything should work. 🤞 (just kidding, these are the only issues that I see)

[framps]

Thanks for the history file. I see you redid the burn with the switches I suggested. Did that fix getting the keyboard layout and add user prompts?

No. Sorry - I forgot to mention this.

When you say "the hostnme was updated to mypi1", you're referring to the Pi, which is correct, since that's what you asked it to do. Am I understanding your comment correctly?

The system booted from the image prepared by sdm used host mypi1. That's what I requested with the customization.

If you redo the customize with the exact same command line but excluding the network plugin, ssh will be enabled and everything should work.

The customization now took much more time and I saw much more messages. Even the boot of the image now takes much more time and I saw a lot of messages which were created by sdm. But now I'm no longer able to login 😭 .

[gitbls]

OK, if the customization took longer then it probably ran to completion. As before, I need to see the /etc/sdm/history file from the SSD/SD Card. If you can't login, you can access it on your ubuntu system with sdm --explore /dev/sdd (or whatever your drive is).

[framps]

ssh is now enabled. But user pi cannot log in 😞

[gitbls]

As above, I need to see the history file from the disk you burned and booted.

Thx

[framps]

I cannot scp the history file any more because I cannot login any more. I have to shutdown RPi and use a SD card reader to read and provide the history file you ask for. Give me a second ...

[gitbls]

OK. on the inability to login, presumably the password you entered when prompted for it during customize was entered correctly WRT case, etc, and you're using that password when you try to login.

Also, when you've got the IMG open with sdm --customize you can easily change the pi password using passwd pi . Then exit the customize and reboot the system. You should be able to login then.

[framps]

I just plugged in the SD card with a card reader to my desktop.

history.log

[framps]

--plugin user:"adduser=pi|prompt"

That's what I used in customization. I was never prompted for a password.

[gitbls]

never? Not even on the first time you customized?

You should have gotten a prompt for a password the first time.

If you did, then you have uncovered a buglet in the user plugin, in that it doesn't honor the prompt when you re-customize an IMG.

Please restart with a fresh IMG, do the customize, burn and boot. LMK how that goes.

[framps]

Just to make sure I got you: That's what you want me to do:
``
sudo sdm --customize 2023-12-11-raspios-bookworm-armhf-lite.img --plugin user:"adduser=pi|prompt" --plugin disables:piwiz --plugin L10n:host --restart
sudo sdm --burn /dev/sde --hostname mypi1 --expand-root 2023-12-11-raspios-bookworm-armhf-lite.img

When should I've been prompted for the pi pwd? During customization or during boot of the image?

[gitbls]

That should do it, yes. For testing you could put the password on the customize command instead of prompt, I can look at that when I get back home in abt an hour.

[framps]

It's a couple of minutes before midnight here in Germany. I'm now going to sleep. You're 6 hours behind me 😃

I will work on this tomorrow.

[gitbls]

OK. JFYI I started with a "fresh" IMG, did this customization:

sudo sdm --customize 2023-12-05-raspios-bookworm-arm64-lite.img --plugin user:"adduser=xx|prompt" \
   --plugin disables:piwiz --plugin L10n:host --restart

Then burned it:

sudo sdm --burn /dev/sdd --hostname pt --expand-root --regen-ssh-host-keys 2023-12-05-raspios-bookworm-arm64-lite.img 

The system booted with no asking for username/localization/etc, was on the network via Ethernet, and I was able to login with the password that I provided when I customized the IMG.

So, I'm 99.999% sure that if you do the same your disk will boot and you'll be able to login with the username/password you provide.

PS I think Seattle is 9hrs behind Germany. 😲

[framps]

So, I'm 99.999% sure that if you do the same your disk will boot and you'll be able to login with the username/password you provide.

I now booted the system successfully with a fresh image 😄 .Apologize for the confusion. Looks like the image got in a bad state because of my multiple attempts to create the image. I never grabbed a fresh image.

PS I think Seattle is 9hrs behind Germany

You're right 👍 . When I read WA (I skipped Seattle) I thought you're located in Washington DC 😢 : I frankly didn't know there exists a state WA in the US 😟

As far as I understand it's possible to provide new plugins. Do you think it's worth to create one for raspiBackup? Actually the raspiBackup target audience are home users of a Raspberry and they will not use sdm.

[gitbls]

Oh, excellent! No worries...for me, confusion that people have often creates a learning experience, and my goal is to help people get jump-started with sdm. I was SO excited when I realized what it could do in helping build customized IMGs quickly and I'm happy when others are able to use it as well.

If you run into any other questions/problems, please ask!

As far as your question/suggestion about raspibackup, I'm not familiar with it, as I have a simple backup strategy that works for me. That said, many of the plugins that I built for sdm (such as btwifiset, imon, knockd, and pistrong) were done not because I thought they'd be really popular, but because it makes my systems easier to rebuild. You may be right that raspiBackup users wouldn't use sdm, but you never know...

If you personally use raspiBackup on enough Pis that you are/will/might use sdm to configure, then it might be useful for you to have such a plugin for your own use. If you do end up building one, I'd be happy to help you with anything you run into while building it, and we could certainly discuss whether it should be included in sdm. You could also provide the plugin on the raspiBackup github page in addition to, or instead of, including it in sdm. Up to you!

WA state is a pretty cool place. We have lots of high tech (MSFT, AMZN, etc), lots of green, and fresh air. Definitely a different place than DC! 🤣

[framps]

I use raspiBackup on 3 Raspberries and update them whenever a new version is available. I don't have a mass deployment scenario where sdm will help which I expect 99.999% of the users of raspiBackup will have also.

As far as I can see and with your feedback there will be not much users install raspiBackup during image deployment. Maybe I will create one just because I have some spare time 😆 . I noticed you provide a lot of doc and samples 👍 Do you have any doc about the different phases (0,1, postplugin) and what's the state of the image in this phase?

[gitbls]

This document describes the various phases: https://github.com/gitbls/sdm/blob/master/Docs/Operating-Details.md
and this one has some useful information about programming plugins: https://github.com/gitbls/sdm/blob/master/Docs/Programming-Plugins-and-Custom-Phase-Scripts.md

After you read those, you might want to have a look through a couple of plugins, such as btwifiset, knockd, and system for some additional insights.

Of course I'll appreciate any feedback you have on these docs from the perspective of someone writing an installer plugin,

Thx!