[20.09] 3-minute delay at boot waiting for wpa_supplicant to restart; may also result in failure to configure the wireless interface #107341
Labels
0.kind: bug
Something is broken
6.topic: nixos
Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS
Comments
|
Relevant bits of the log |
veprbl
added
the
6.topic: nixos
|
I think I have an idea but I don't know why it's hanging. Can you try setting this? |
|
It looks like there is a deadlock: udev is waiting for the The most likely cause is dhcpcd wanting systemd-udev-settle. |
|
It's not related to wpa_supplicant in particular. It looks like having any service waiting for udev-settle will hang a |
5 tasks
5 tasks
rnhmjoj
added a commit
that referenced
this issue
Feb 20, 2021
systemd-udev-settle is a terrible hack[1] and should never[2] ever[3] used, seriously it's very bad. It was used as a stop-gap solution for issue #39069, but thanks to PR #79532 it can be removed now. [1]: systemd/systemd#7293 (comment) [2]: #73095 [3]: #107341
rnhmjoj
added a commit
to rnhmjoj/nixpkgs
that referenced
this issue
Feb 21, 2021
It's a dull and boring day, it's cold outside and I'm stuck at home: let me tell you the story of systemd-vconsole-setup. In the beginnings of NixOS[1], systemd-vconsole-setup was a powerful sysinit.target unit, installed and running at boot to set up fonts keyboard layouts and even colors of the virtual consoles. If needed, the service would also be restarted after a configuration change, consoles were happy and everything was good, well, almost. Since the service had no way to specify the dependency "ttys are ready", modesetting could sometimes happen *after* systemd-vconsole-setup had started, leaving the console in a broken state. So abbradar worked around that by putting a systemd-udev-settle `After=`. In the meanwhile, probably realizing their mistake, systemd added a shiny udev rule to start the systemd-udev-settle at the right time[2]. However, the rule bypassed systemd by directly running the binary `systemd-udev-settle`, and the service - though still installed - fell into disuse. Two years would pass before a good samaritan, seeing the poor jobless systemd-udev-settle service, decided to give it the coup de grâs[3] by unlisting it from the installed units. This, combined with another bug, caused quite a commotion[4] in NixOS; to see why remember the fact that `WantedBy=` in upstream units doesn't work[5], so it had to be added manually in cc54211, but while systemd removed it, the NixOS unit continued to install and restart the service, making a lot of fuss when switching configuration. After at least thee different tentative fixes, deedrah realised[6] what the root cause was and fpletz put the final nail[7] in the coffin of systemd-udev-settle. The service would never see the light of a boot again, NixOS would not restart it all the time but thanks to udev consoles would still get their pretty fonts and playful colors. The En.. ..no, wait! You should ask what came of systemd-udev-settle, first. And why is the service even around if udev is doing all the work? Udev-settle, like the deceitful snake that he is, laid hidden for years. He looks innocuous doesn't it? A little hack. Only until it leaves his den and a poor user[8] drops dead. Obviously, it serves no purpose, as the service is not part of the boot process anymore, so let's remove it for good! About the service, it may not be useful at boot, but it can be started to pick up changes in vconsole.conf and set the consoles accordingly. But wait, this doesn't work anymore: the service is never started at boot (remember f76d2aa), so switch-to-configuration.pl will not restart it. Fortunately it can be repaired: here I install a new unit which does *nothing* on start, but restarts the real service when reloaded. This perfectly reproduces the original behavior, hopefully without the original bugs too. The End? [1]: NixOS@cc54211 [2]: systemd/systemd@f6ba867#diff-84849fddcef81458f69725dc18c6614aade5c4f41a032b6908ebcf1ee6740636 [3]: systemd/systemd@8125e8d [4]: https://web.archive.org/web/20180603130107/https://github.com/NixOS/nixpkgs/issues/22470 [5]: NixOS#81138 [6]: https://web.archive.org/web/20180603130107/https://github.com/NixOS/nixpkgs/issues/22470#issuecomment-330930456 [7]: NixOS@f76d2aa [8]: NixOS#107341
|
No, unless systemd-udev-settle is removed from every service, there is still the possibility of a deadlock like this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Since #104722 (backported to 20.09 as 645b8a3), the boot process hangs for 3 minutes on this message:
Even a minimal configuration with just this:
as the only network-relevant config displays the same behavior.
I got the same delay + message while booting a recent nixos-minimal-20.09 iso.
In addition to the delay, with
networking.useNetworkd = true;, the wireless interface is never configured at all it seems.I haven't yet tried with nixos-unstable.
Reverting 645b8a3 fixes the issue.
Ping @rnhmjoj @doronbehar
The text was updated successfully, but these errors were encountered: