NetworkManager's nm-online kills nixos-rebuild #180175

tbidne · 2022-07-04T22:45:25Z

Describe the bug

The systemd service NetworkManager-wait-online.service can prevent nixos-rebuild from succeeding:

warning: the following units failed: NetworkManager-wait-online.service

× NetworkManager-wait-online.service - Network Manager Wait Online
     Loaded: loaded (/etc/systemd/system/NetworkManager-wait-online.service; enabled; vendor preset: enabled)
    Drop-In: /nix/store/k5yq51spcggip2h6aq1y0bydkpr4zahc-system-units/NetworkManager-wait-online.service.d
             └─overrides.conf
     Active: failed (Result: exit-code) since Tue 2022-07-05 10:18:52 NZST; 36ms ago
       Docs: man:nm-online(1)
    Process: 1258376 ExecStart=/nix/store/b4yhg54s70i0v0k1qnnv8vnja6018yrh-networkmanager-1.38.2/bin/nm-online -s -q (code=exited, status=1/FAILURE)
   Main PID: 1258376 (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 22ms

Jul 05 10:17:52 nixos systemd[1]: Starting Network Manager Wait Online...
Jul 05 10:18:52 nixos systemd[1]: NetworkManager-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jul 05 10:18:52 nixos systemd[1]: NetworkManager-wait-online.service: Failed with result 'exit-code'.

This service runs nm-online -s -q, and the nm-online man page says:

-s | --wait-for-startup
           Wait for NetworkManager startup to complete, rather than waiting for network connectivity specifically. Startup is
           considered complete once NetworkManager has activated (or attempted to activate) every auto-activate connection
           which is available given the current network state. This corresponds to the moment when NetworkManager logs "startup
           complete". This mode is generally only useful at boot time. After startup has completed, nm-online -s will just
           return immediately, regardless of the current network state.

           There are various ways to affect when startup complete is reached. For details see NetworkManager-wait-
           online.service(8).

This corresponds to the moment when NetworkManager logs "startup complete". This mode is generally only useful at boot time.

I am not familiar with this tool, but my experience is that after my laptop has been up for some time (e.g. days), nm-online will often return an error code rather than correctly determine the network is up, thus killing any future nixos-rebuild commands.

Steps To Reproduce

Steps to reproduce the behavior:

Get your machine in a state where nm-online -s -q does not return success (not sure how to do this on demand).
Witness nixos-rebuild failure.

Expected behavior

nixos-rebuild should not fail due to an erroneous network check.

Additional context

This is tricky as it is not a nix issue per se but rather an issue with a presumably flaky systemd service. It is easy enough to disable this service manually:

systemd.services.NetworkManager-wait-online.enable = false;

And perhaps this is the best solution. But a number of my coworkers all ran into this issue independently, so I thought it merited an issue for discoverability, if nothing else. My gut reaction is that a flaky check should probably not be required by default, but I don't know enough about this service's importance/fragility to say.

This issue was noticed only recently, both on nixos-unstable and nixos-22.05.

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.47, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Thanks!

The text was updated successfully, but these errors were encountered:

tbidne · 2022-07-05T00:36:02Z

Possibly it relates to this? #178046

Edit: Nevermind, that commit is not in nixos-22.05.

nixos-discourse · 2022-07-06T22:29:24Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-disable-networkmanager-wait-online-service-in-the-configuration-file/19963/4

lorenz · 2022-07-16T10:56:52Z

This is related to udev not initializing devices. NetworkManager never completes startup because a WireGuard interface is never initialized by udev. A workaround is just putting the affected device into networking.networkmanager.unmanaged.

DeskworkTrickster · 2022-08-09T18:09:34Z

I'm wondering if there is any solution to this. Since this triggers for me always. So, I'm not even sure if my nixos-rebuild switch does actually complete.

As mentioned in the initial description of the issue:

This corresponds to the moment when NetworkManager logs "startup complete". This mode is generally only useful at boot time.

This never seems to be the case, since the actual message is "Started Network Manager.".
Apart from that I'm wondering why it relies on a log message string, when the status of NetworkManager.service would be way less prone to errors.

EDIT: no idea why, but it went away ...
Everything works as expected again.

oati · 2022-08-22T13:45:18Z

I think I'm running into this.

systemd-networkd-wait-online.service

Aug 22 08:38:49 erin-laptop systemd[1]: Starting Wait for Network to be Configured...
Aug 22 08:40:49 erin-laptop systemd-networkd-wait-online[26249]: Timeout occurred while waiting for network connectivity.
Aug 22 08:40:49 erin-laptop systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Aug 22 08:40:49 erin-laptop systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Aug 22 08:40:49 erin-laptop systemd[1]: Failed to start Wait for Network to be Configured.

It only happens after I've connected my USB-C dock with an ethernet connection at least once after boot. (Note: I'm running tmpfs on root, so my system should "forget" everything about my dock on reboot.)

I'm also running systemd-networkd, not NetworkManager.

See NixOS/nixpkgs#180175

NorfairKing · 2022-08-31T12:19:38Z

@Stale don't you dare!

Attempt to fix nm-online-service from stalling on tailscale interface. See NixOS/nixpkgs#180175

See NixOS/nixpkgs#180175

pjones · 2022-10-10T17:58:10Z

@ncfavier Any chance you can help with this?

ncfavier · 2022-10-10T20:59:09Z

I don't use NetworkManager so I wouldn't know, but in the case of systemd-networkd there are relevant options under systemd.network.wait-online: anyInterface and ignoredInterfaces. I recommend at least setting the former to true on laptops.

So, I'm not even sure if my nixos-rebuild switch does actually complete.

Warning about failed units is pretty much the last thing that the activation script does, so it's probably fine (but the failure should be fixed, of course).

lorenz · 2022-10-10T21:14:38Z

BTW I've "fixed" this by setting

# udev 250 doesn't reliably reinitialize devices after restart
systemd.services.systemd-udevd.restartIfChanged = false;

But this is really an upstream systemd bug.

Temporarily fixed by disabling nm-wait-online NixOS/nixpkgs#180175

blaggacao · 2022-11-07T18:59:27Z

My Ubuntu has:

❯ systemctl cat NetworkManager-wait-online.service
# /lib/systemd/system/NetworkManager-wait-online.service
[Unit]
Description=Network Manager Wait Online
Documentation=man:nm-online(1)
Requires=NetworkManager.service
After=NetworkManager.service
Before=network-online.target

[Service]
# `nm-online -s` waits until the point when NetworkManager logs
# "startup complete". That is when startup actions are settled and
# devices and profiles reached a conclusive activated or deactivated
# state. It depends on which profiles are configured to autoconnect and
# also depends on profile settings like ipv4.may-fail/ipv6.may-fail,
# which affect when a profile is considered fully activated.
# Check NetworkManager logs to find out why wait-online takes a certain
# time.

Type=oneshot
ExecStart=/usr/bin/nm-online -s -q
RemainAfterExit=yes

# Set $NM_ONLINE_TIMEOUT variable for timeout in seconds.
# Edit with `systemctl edit NetworkManager-wait-online`.
#
# Note, this timeout should commonly not be reached. If your boot
# gets delayed too long, then the solution is usually not to decrease
# the timeout, but to fix your setup so that the connected state
# gets reached earlier.
Environment=NM_ONLINE_TIMEOUT=60

[Install]

My latest NixOS (22.05) config has:

> nix-repl> c.config.systemd.services.NetworkManager-wait-online
{ after = [ ... ]; aliases = [ ... ]; before = [ ... ]; bindsTo = [ ... ]; confinement = { ... }; conflicts = [ ... ]; description = ""; documentation = [ ... ]; enable = false; environment = { ... }; jobScripts = [ ... ]; onFailure = [ ... ]; partOf = [ ... ]; path = [ ... ]; postStart = ""; postStop = ""; preStart = ""; preStop = ""; reload = ""; reloadIfChanged = false; reloadTriggers = [ ... ]; requiredBy = [ ... ]; requires = [ ... ]; requisite = [ ... ]; restartIfChanged = true; restartTriggers = [ ... ]; runner = error: attribute 'ExecStart' missing

       at /nix/store/6dgpkrc0gxlndr4j2524ihlsr8209ph7-source/nixos/modules/testing/service-runner.nix:65:9:

           64|       my $cmd = <<END_CMD;
           65|       ${service.serviceConfig.ExecStart}
             |         ^
           66|       END_CMD
«derivation

based on this definition:

    systemd.services.NetworkManager-wait-online = {
      wantedBy = [ "network-online.target" ];
    };

Where is this ExecStart coming from in your configurations?

Also:

nixpkgs on  fix/teamviewer-service-deps [$]
❯ rg 'nm-online'
[ nothing ]

bjornfor · 2022-11-07T19:43:36Z

@blaggacao: The reference to nm-online comes from upstream service unit NetworkManager-wait-online.service, not from nixpkgs itself.

domenkozar · 2023-01-10T13:02:18Z

I'd vote for disabling this service until we can make it reliable. It's doing no good currently.

pinpox · 2023-03-17T08:40:43Z

I've been tripping over this bug for quite some time now and it is annoying for users. As mentioned above, the error can be worked around with:

systemd.services.NetworkManager-wait-online.enable = lib.mkForce false;
systemd.services.systemd-networkd-wait-online.enable = lib.mkForce false;

I was concernd if there might be other dependencies or services that require this to be enabled, so I grepped through nixpkgs for both. These are the mentions:

NetworkManager-wait-online.service

modules/networkmanager The actual definition of this service
pkgs.hqplayerd can probably be removed

systemd-networkd-wait-online.service

modules/system/boot/networkd Service devinition
Used in various nixos tests, which should not be relevant to normal operation of the system (?)
- nixos/tests/systemd-networkd-dhcpserver-static-leases.nix
- nixos/tests/kea.nix
- nixos/tests/systemd-networkd.nix
- nixos/tests/systemd-bpf.nix
- nixos/tests/systemd-networkd-dhcpserver.nix

TL;DR

Upon first look the usage of these two services seem minimal to me and they are causing more problems that doing good. Agreeing with @domenkozar's proposal, I'd vote to disable them per default. If this is agreed upon, I can submit a PR

matthiasbeyer · 2023-03-17T09:10:59Z

I've been running with that service disabled for 6 months now and have not experienced a single issue. Don't count my voice too heavily, though 😉 ! 👍

lorenz · 2023-03-17T12:05:08Z

If we're going to work around this I'd still prefer systemd.services.systemd-udevd.restartIfChanged = false; as the other workaround just masks the issue while udev's still half-broken.

pinpox · 2023-03-22T10:24:07Z

If we're going to work around this I'd still prefer systemd.services.systemd-udevd.restartIfChanged = false; as the other workaround just masks the issue while udev's still half-broken.

For some reason, that didn't work for me. On rebuild it said "not restarting service"

supermarin · 2023-04-10T22:33:40Z

Yep confirming what @pinpox said:

updating GRUB 2 menu...
NOT restarting the following changed units: systemd-udevd.service

Still seeing the issue on HEAD. Disabling wait-online like mentioned previously fixes nixos-rebuild.

systemd.services.NetworkManager-wait-online.enable = lib.mkForce false;
systemd.services.systemd-networkd-wait-online.enable = lib.mkForce false;

AkechiShiro · 2024-09-28T13:52:55Z

Is this only fixed when using tailscaled, for users not using tailscaled, they should apply which of the dozens workaround shared here ? Is this really a good solution for new users? Shouldn't something be also added in NetworkManager's module ?

Quoting previous comments :
#180175 (comment)

supermarin · 2024-09-28T14:09:04Z

@AkechiShiro have you observed this without using Tailscale?

As far as I understand the issue is tied down to the combination of Tailscale firing up before NetworkManager and depending on it being online, ending up in a deadlock essentially

pinpox · 2024-09-28T14:15:33Z

The same issue happened to me with wireguard in the past. I'll try the suggested fix

AkechiShiro · 2024-09-28T14:18:58Z

@supermarin I have yes and I believe I'm not the only one

supermarin · 2024-09-28T14:25:40Z

Mind posting your configuration.nix in a gist?
We should be able to create a minimal flake and reproduce this with build-vm. Maybe cut out the VM's network access on purpose.

We should reopen this issue then if this happens outside of just Tailscale & NM.

EDIT: do you use wireguard? Tailscale uses wireguard so that could be the lower common denominator

AkechiShiro · 2024-09-28T15:55:29Z

I do use Wireguard yes, so it could be I have the same issue as @pinpox

Regarding my configuration.nix, it is all over the place currently, a minimal flake would be great but I can't provide my whole config at the moment, if I have time I'll try in a VM with a single Wireguard interface setup.

Atemu · 2024-09-28T16:18:22Z

Can anyone confirm whether they still experience this issue without using tailscaled?

sgraf812 · 2024-09-30T10:29:31Z

As recently as two years ago, I had this issue without knowingly using tailscaled. (Posted to #59603.)
Since then, I'm using the workaround from the OP and have lived happily ever since.

The problem is perhaps that tailscaled is not the only service that activates NetworkManager-wait-online.service, so it doesn't make sense to fix this issue in tailscaled.

Perhaps this issue has since been fixed by an unrelated patch upstream (however that would happen), but I'm not optimistic. Multiple people (for example #180175 (comment), linking to #182449) have pointed that this is probably an upstream bug in systemd/udev. Let me try to get rid of the workaround and see if it is still an issue.

AkechiShiro · 2024-09-30T12:48:48Z

Thanks @sgraf812 I may try to reach out to systemd/udev maintainers about issue #182449 hopefully we can nail this issue down once and for all

AkechiShiro · 2024-09-30T13:37:53Z

We did receive an answer from Poettering but I don't know how to properly answer back, if anyone more experienced could pitch in : systemd/systemd#34585

The wait will only be enabled on machines with NetworkManager enabled. Closes #180175 (cherry picked from commit 0d822cc)

The wait will only be enabled on machines with NetworkManager enabled. Closes NixOS#180175 (cherry picked from commit 0d822cc)

nixos-discourse · 2024-10-15T04:31:57Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rebuild-error-failed-to-start-network-manager-wait-online/41977/5

Atemu · 2024-10-16T11:15:43Z

Just ran into this again with tailscaled.

supermarin · 2024-10-16T16:49:37Z

@Atemu ran into it as well, but already had a running system with tailscale up. Manually shut down tailscale, and rebuild succeeded (and tailscale was back up). Tested with several rebuilds and restarts, works ok so far

Atemu · 2024-10-16T17:04:55Z

An idea I had on this is that we could perhaps hack around this using a systemd unit that Conflicts the units which appear to block nm-online.target such as tailscaled. It'd be Required by the nm-online target and we'd then have another target which wants all those units and is wanted by the network-online target but After the nm-online target.

NixOS/nixpkgs#180175

inmaldrerah · 2024-10-24T07:21:28Z

Maybe this would help? (I closed because earlier I thought it was obsolete)

And I've made a small flake version for testing purpose: https://github.com/inmaldrerah/nixos-extensions

At least last time I used this, I didn't have to systemd stop tailscaled beforehand when tailscale was updated.

supermarin · 2024-10-24T18:15:51Z

I can try testing with it.
Think the repro steps are:

Have a system with Tailscale and NM-wait-online
Disable nm-wait-online with the hack above
Put nm-wait-online back in

On the next rebuild it should get stuck. I'll try this later today and report back if it reproduces consistently. Note: the Tailscale fix was backported to 24.05 so need to pin nixpkgs to a commit prior to that

Atemu · 2024-10-25T06:05:14Z

Note that this issue occurs even with the supposed fix.

….service timing out on generation activation This is caused by nmcli waiting on the mullvad wiregaurd network device to be up, which never happens. The workaround is adding the device, wg0-mullvad, to the unmanaged devices list. Upstream issue is NixOS/nixpkgs#180175 .

JonnieCache · 2024-12-04T13:09:33Z

Also still experiencing this with tailscaled on nixpkgs unstable bc947f541ae55e999ffdb4013441347d83b00feb (I think that's the right sha.)

If I stop the tailscaled unit I can then rebuild OK, and the service comes back up again.

supermarin · 2024-12-04T18:52:29Z

Yeah just ran into it as well :(
@inmaldrerah mind posting what's the best way to test your flake?
I looked into it quickly but not 100% clear on how to plug in switch-to-configuration-ng.

NixOS/nixpkgs#180175

Munksgaard · 2024-12-13T12:42:15Z

I'm still running into this problem. However, I'm not using NetworkManager, but systemd-networkd. Adding the following line to my config seemed to fix things:

systemd.services.tailscaled.after = ["systemd-networkd-wait-online.service"];

Perhaps we should also add that line to the tailscale service definition?

inmaldrerah · 2024-12-18T01:52:07Z

Yeah just ran into it as well :( @inmaldrerah mind posting what's the best way to test your flake? I looked into it quickly but not 100% clear on how to plug in switch-to-configuration-ng.

That flake can be used by making an overlay over nixpkgs, ~~disable system.switch.enable,~~ and enable system.switch.enableNg in the configuration.nix.

Now I haven't been using my flake for a while, but still keep ~~system.switch.enable = false and~~ system.switch.enableNg = true, and I am no longer experiencing this problem on the unstable branch (until a73246e2eef4c6ed172979932bc80e1404ba2d56), for some reason.

Edit: I checked the code of nixos/modules/system/activation/switchable-system.nix and found that I don't have to disable system.switch.enable now, and system.switch.enableNg = system.switch.enable (= true) is now default.

nixos-discourse · 2024-12-27T09:33:49Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/error-network-wait-online-service/57902/3

tbidne added the 0.kind: bug Something is broken label Jul 4, 2022

veprbl added the 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS label Jul 5, 2022

lorenz mentioned this issue Jul 22, 2022

udev gets stuck after reload not initializing devices #182449

Open

jammus added a commit to jammus/dotfiles that referenced this issue Aug 23, 2022

Prevent issue with nm-online breaking rebuild

f1853e4

See NixOS/nixpkgs#180175

svrana added a commit to svrana/nix-home that referenced this issue Oct 5, 2022

Instruct nm not to manage tailscale0

fc03621

Attempt to fix nm-online-service from stalling on tailscale interface. See NixOS/nixpkgs#180175

svrana added a commit to svrana/nix-home that referenced this issue Oct 5, 2022

Disable nm wait-online service for now

66cdfc6

See NixOS/nixpkgs#180175

jack-michaud added a commit to jack-michaud/nix that referenced this issue Oct 15, 2022

fix: rebuild timeout from nm-wait-online

1d1c8d0

Temporarily fixed by disabling nm-wait-online NixOS/nixpkgs#180175

This was referenced Nov 4, 2022

nixos: losing network after "big" nixos-rebuild switch #198267

Open

mod: add reasonable timeout to nm-wait or users get stuck #199885

Open

sgraf812 mentioned this issue Dec 18, 2022

NetworkManager-wait-online.service is broken #59603

Closed

gorschu added a commit to gorschu/nix-config that referenced this issue Mar 18, 2023

Work around bug NixOS/nixpkgs#180175

2a348d3

archer-65 added a commit to archer-65/nix-dotfiles that referenced this issue Mar 21, 2023

network: see NixOS/nixpkgs#180175

8385f5a

kalbasit added a commit to kalbasit/soxincfg that referenced this issue Apr 3, 2023

profiles/workstation/nixos: workaround NixOS/nixpkgs#180175

a556c68

fpletz closed this as completed in #344678 Sep 28, 2024

fpletz closed this as completed in 0d822cc Sep 28, 2024

github-actions bot pushed a commit that referenced this issue Oct 5, 2024

tailscaled: after NetworkManager-wait-online

ffc03d2

The wait will only be enabled on machines with NetworkManager enabled. Closes #180175 (cherry picked from commit 0d822cc)

presto8 pushed a commit to presto8/nixpkgs that referenced this issue Oct 9, 2024

tailscaled: after NetworkManager-wait-online

6b98de2

The wait will only be enabled on machines with NetworkManager enabled. Closes NixOS#180175 (cherry picked from commit 0d822cc)

Atemu reopened this Oct 16, 2024

srid added a commit to srid/nixos-config that referenced this issue Oct 22, 2024

Workaround a NetworkManager issue on switch

6574f72

NixOS/nixpkgs#180175

huang12zheng pushed a commit to huang12zheng/nixos-config that referenced this issue Dec 5, 2024

Workaround a NetworkManager issue on switch

a447084

NixOS/nixpkgs#180175

NetworkManager's nm-online kills nixos-rebuild #180175

NetworkManager's nm-online kills nixos-rebuild #180175

Comments

tbidne commented Jul 4, 2022

Describe the bug

Steps To Reproduce

Expected behavior

Additional context

Notify maintainers

Metadata

tbidne commented Jul 5, 2022 • edited Loading

nixos-discourse commented Jul 6, 2022

lorenz commented Jul 16, 2022

DeskworkTrickster commented Aug 9, 2022 • edited Loading

oati commented Aug 22, 2022 • edited Loading

NorfairKing commented Aug 31, 2022

pjones commented Oct 10, 2022

ncfavier commented Oct 10, 2022

lorenz commented Oct 10, 2022

blaggacao commented Nov 7, 2022 • edited Loading

bjornfor commented Nov 7, 2022

domenkozar commented Jan 10, 2023

pinpox commented Mar 17, 2023

NetworkManager-wait-online.service

systemd-networkd-wait-online.service

TL;DR

matthiasbeyer commented Mar 17, 2023

lorenz commented Mar 17, 2023

pinpox commented Mar 22, 2023

supermarin commented Apr 10, 2023

AkechiShiro commented Sep 28, 2024 • edited Loading

supermarin commented Sep 28, 2024

pinpox commented Sep 28, 2024

AkechiShiro commented Sep 28, 2024

supermarin commented Sep 28, 2024 • edited Loading

AkechiShiro commented Sep 28, 2024 • edited Loading

Atemu commented Sep 28, 2024

sgraf812 commented Sep 30, 2024 • edited Loading

AkechiShiro commented Sep 30, 2024

AkechiShiro commented Sep 30, 2024

nixos-discourse commented Oct 15, 2024

Atemu commented Oct 16, 2024

supermarin commented Oct 16, 2024

Atemu commented Oct 16, 2024

inmaldrerah commented Oct 24, 2024

supermarin commented Oct 24, 2024

Atemu commented Oct 25, 2024

JonnieCache commented Dec 4, 2024

supermarin commented Dec 4, 2024

Munksgaard commented Dec 13, 2024

inmaldrerah commented Dec 18, 2024 • edited Loading

nixos-discourse commented Dec 27, 2024

tbidne commented Jul 5, 2022 •

edited

Loading

DeskworkTrickster commented Aug 9, 2022 •

edited

Loading

oati commented Aug 22, 2022 •

edited

Loading

blaggacao commented Nov 7, 2022 •

edited

Loading

AkechiShiro commented Sep 28, 2024 •

edited

Loading

supermarin commented Sep 28, 2024 •

edited

Loading

AkechiShiro commented Sep 28, 2024 •

edited

Loading

sgraf812 commented Sep 30, 2024 •

edited

Loading

inmaldrerah commented Dec 18, 2024 •

edited

Loading