ConditionNeedsUpdate= (in systemd units) will never trigger

[travier]

See original issue in coreos/fedora-coreos-tracker#1538.

Previous PRs and issues:

• deploy: Delete .updated file from /etc and /var on new deployments #1631
• ConditionNeedsUpdate needs /usr to have newer modification time than /etc #1628
• Question: modification time for /usr and /etc coreos/rpm-ostree#1415
• https://bugzilla.gnome.org/show_bug.cgi?id=752950

---

Describe the bug

As we keep timestamp of files at 0 (UNIX Epoch) in our ostree commits (https://ostreedev.github.io/ostree/repo/#content-objects), this condition in systemd unit will never trigger.

A quick grep on my system show the following potentially impacted services:

$ grep -Rni ConditionNeedsUpdate= /usr/lib/systemd
/usr/lib/systemd/system/ldconfig.service:14:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-hwdb-update.service:17:ConditionNeedsUpdate=/etc
/usr/lib/systemd/system/systemd-journal-catalog-update.service:17:ConditionNeedsUpdate=/var
/usr/lib/systemd/system/systemd-sysusers.service:20:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-update-done.service:17:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-update-done.service:18:ConditionNeedsUpdate=|/var

# Symlinks to the services above:
/usr/lib/systemd/system/sysinit.target.wants/ldconfig.service:14:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-hwdb-update.service:17:ConditionNeedsUpdate=/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-journal-catalog-update.service:17:ConditionNeedsUpdate=/var
/usr/lib/systemd/system/sysinit.target.wants/systemd-sysusers.service:20:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-update-done.service:17:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-update-done.service:18:ConditionNeedsUpdate=|/var

More investigation is needed to see if that's an issue or just a quirk.

Reproduction steps

N/A

Expected behavior

ConditionNeedsUpdate= "works"

Actual behavior

ConditionNeedsUpdate= "doesn't work"

System details

N/A

Butane or Ignition config

N/A

Additional information

See https://bugzilla.redhat.com/show_bug.cgi?id=2230187 (moved to https://issues.redhat.com/browse/RHEL-3624)

[travier]

I've just found:

ostree/src/libostree/ostree-sysroot-deploy.c

Line 3403 in 4451949

/* Ensure that the new deployment does not have /etc/.updated or

and on a freshly booted system:

[core@cosa-devsh ~]$ ls -alh /etc/.updated 
-rw-r--r--. 1 root root 190 Jan  1  1970 /etc/.updated

Not sure how related: https://www.freedesktop.org/software/systemd/man/systemd-update-done.service.html

The comment in the ostree code does not match the behavior described in the man page.

[travier]

So, it looks like this has already been fixed with #1631 so there is apparently nothing to do here.

The logic is that systemd looks for the /etc/.updated & /var/.updated files and if they exists, compares their timestamps to the one of /usr. https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#ConditionNeedsUpdate=

In #1631, we always remove those fies from new deployments so the condition always triggers: https://github.com/systemd/systemd/blob/3a9e659a0e2f30a94137fca6b6886037be72b120/src/shared/condition.c#L781

Units with ConditionNeedsUpdate should be ordered before systemd-update-done.service, which will always re-create those .updated files with the 0 timestamp of /usr which comes as part of the ostree commit.

To verify that, we should write a test that create that changes the configuration updated by those units (for example a sysusers config file), then create a new deploy (for example by installing a package) and then reboot and verify that the new configuration has been applied (the new user has been created).

[travier]

I've just verified manually that this is working:

# Boot a fresh Fedora CoreOS
$ getent passwd foo
<empty>
# Write a new sysusers config file
$ cat /etc/sysusers.d/30-foo.conf
u      foo  -   "foo"
$ reboot

Next boot:

$ getent passwd foo
<empty> # Still empty as expected as /usr did not change
$ rpm-ostree update # rpm-ostree install foo to create a new deployment
$ reboot

Next boot:

$ getent passwd foo
foo:x:978:978:foo:/:/usr/sbin/nologin
$ sudo journalctl -u systemd-update-done.service 
Nov 07 09:06:00 localhost systemd[1]: Starting systemd-update-done.service - Update is Completed...
Nov 07 09:06:00 localhost systemd[1]: Finished systemd-update-done.service - Update is Completed.
Nov 07 09:37:46 cosa-devsh systemd[1]: systemd-update-done.service: Deactivated successfully.
Nov 07 09:37:46 cosa-devsh systemd[1]: Stopped systemd-update-done.service - Update is Completed.
-- Boot e6b4be4b2834479bb9f4fd24402b33c9 --
Nov 07 09:37:54 localhost systemd[1]: systemd-update-done.service - Update is Completed was skipped because no trigger condition checks were met.
-- Boot fc7f3a387a774ab7b1d5de4c6dc2d472 --
Nov 07 09:38:55 localhost systemd[1]: Starting systemd-update-done.service - Update is Completed...
Nov 07 09:38:55 localhost systemd[1]: Finished systemd-update-done.service - Update is Completed.

[travier]

This test verifies that this is working as expected today: coreos/fedora-coreos-config#2725