Unable to disable zincati.service using Ignition #392

dustymabe · 2020-02-16T03:34:22Z

As reported by a user in a discussion forum thread for some reason disabling zincati.service via Ignition doesn't seem to be working right. There is probably a corner case in here that isn't being accounted for somehow.

Here is my fcc:

variant: fcos
version: 1.0.0
systemd:
  units:
    - name: serial-getty@ttyS0.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override Execstart in main unit
          ExecStart=
          # Add new Execstart with `-` prefix to ignore failure
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
          TTYVTDisallocate=no
    - name: zincati.service
      enabled: false

Here is what I see on the booted system:

[core@localhost ~]$ systemctl is-enabled zincati
enabled
[core@localhost ~]$ systemctl is-active zincati
active
[core@localhost ~]$ 
[core@localhost ~]$ grep zincati /etc/systemd/system-preset/20-ignition.preset
disable zincati.service
[core@localhost ~]$ grep zincati /usr/lib/systemd/system-preset/40-coreos.preset
enable zincati.service

So the disable preset did get put in /etc/ appropriately but for some reason it wasn't applied properly. If I ask systemd to re-evaluate it seems to do the right thing:

[core@localhost ~]$ sudo systemctl preset zincati.service
Removed /etc/systemd/system/multi-user.target.wants/zincati.service.
[core@localhost ~]$ systemctl is-enabled zincati
disabled

Version Info:

[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 31.20200210.2.0 (2020-02-10T21:52:48Z)
                    Commit: 3e4a6a48ed8d6817081c902bc2aa2bbe9df3302e659d4f42d933f9abb22914e8
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
[core@localhost ~]$ rpm -q ignition systemd
ignition-2.1.1-5.git40c0b57.fc31.x86_64
systemd-243.6-1.fc31.x86_64

The text was updated successfully, but these errors were encountered:

dustymabe · 2020-03-06T16:01:49Z

Looks like this and #229 may be the same problem

dustymabe · 2020-03-20T20:20:45Z

ok here may be a clue:

[   15.943369] systemd[1]: Running with unpopulated /etc.

Welcome to Fedora CoreOS 31.20200319.dev.3!

[   15.949061] systemd[1]: Initializing machine ID from KVM UUID.
[   16.244489] systemd[1]: Populated /etc with preset unit settings.

so when systemd first starts /etc/ is empty so it can't see what units are or are not enabled

dustymabe · 2020-03-20T20:23:01Z

well I guess it depends on what is really meant by Running with unpopulated /etc

jlebon · 2020-03-20T20:25:38Z

Re. presets, a lot of discussions here: coreos/fedora-coreos-config#77. That said, the enabled: false from Ignition should result in a later disable zincati preset entry which wins over the default. (Which yeah, this approach runs against coreos/ignition#588). Hmm, so something funny is going on here.

dustymabe · 2020-03-20T20:29:26Z

if I use rd.break=pre-pivot on the kernel command line I see the presets file:

pre-pivot:/# cat /sysroot/etc/systemd/system-preset/20-ignition.preset 
disable zincati.service

and running preset-all against the real root does the right thing:

pre-pivot:/# systemctl --root=/sysroot/ preset-all
Removed /sysroot/etc/systemd/system/multi-user.target.wants/zincati.service.

jlebon · 2020-03-23T17:00:06Z

systemd/systemd#15205

dustymabe · 2020-03-25T14:23:56Z

wow, thanks @jlebon - that was certainly not what I expected to be the problem.

cgwalters · 2020-03-31T14:42:18Z

A workaround until the systemd PR lands is probably something like adding a fragment with ConditionPathExists=/enoent or whatever instead of directly disabling.

Edit: Yep, I can confirm that works.

A lot of backstory on this, but essentially right now, we always bake a run of `systemctl preset-all` into the OSTree because upgrading hosts rely on these links for service enablement. In hindsight, we should've just stuck with pure systemd preset only as canonicaly from the get go, though it's a bit difficult now to transition from one to the other without breaking things. (Though I'll note not impossible, since we do have update barriers which could allow us to e.g. run a script to restore lost symlinks). For now though, let's at least fix the ability to disable services, which is a pretty big gap in our Ignition configuration story right now. Related: systemd/systemd#15205 Related: coreos#77 Closes: coreos/fedora-coreos-tracker#392

sgohl · 2021-05-26T21:18:29Z

A workaround until the systemd PR lands is probably something like adding a fragment with ConditionPathExists=/enoent or whatever instead of directly disabling.

how would a butane config unit look like with this approach?
My main problem is that my provisioning script collides with zincati:

May 26 21:06:06 rdbla229 zincati[744]: [INFO ] target release '34.20210427.3.0' selected, proceeding to stage it
May 26 21:06:07 rdbla229 zincati[744]: [ERROR] failed to stage deployment: rpm-ostree deploy failed:
May 26 21:06:07 rdbla229 zincati[744]:     error: Transaction in progress: --idempotent install open-vm-tools htop cronie

and it will not recover for a long time, showing:

May 26 21:14:26 rdbla229 zincati[732]: [ERROR] failed to register as driver: rpm-ostree deploy --register-driver failed:
May 26 21:14:26 rdbla229 zincati[732]:     error: Transaction in progress: --idempotent install open-vm-tools htop cronie
May 26 21:14:26 rdbla229 zincati[732]:      You can cancel the current transaction with `rpm-ostree cancel`
May 26 21:14:26 rdbla229 zincati[732]:
May 26 21:14:27 rdbla229 zincati[732]: [INFO ] initialization complete, auto-updates logic enabled
May 26 21:14:27 rdbla229 zincati[732]: [INFO ] update strategy: immediate
May 26 21:14:27 rdbla229 systemd[1]: Started Zincati Update Agent.
May 26 21:14:27 rdbla229 zincati[732]: [INFO ] current release detected as not a dead-end

But I want to have an updated system quite after provisioning. I think your approach would fit my need to enable zincati only after my provision script is done

bgilbert · 2021-05-27T06:13:52Z

@sgohl Something like this:

variant: fcos
version: 1.0.0
systemd:
  units:
    - name: zincati.service
      dropins:
        - name: disable-zincati.conf
          contents: |
            [Unit]
            ConditionPathExists=/enoent

jlebon · 2021-05-27T16:12:31Z

But I want to have an updated system quite after provisioning. I think your approach would fit my need to enable zincati only after my provision script is done

Assuming your provisioning script reboots, another approach is to run it with Before=zincati.service. That'll save you from having to then re-enable Zincati on reboot. You'll also want After=systemd-machine-id-commit.service.

A compile time option is added to select behaviour: by default UNIT_FILE_PRESET_ENABLE_ONLY is still used, but the intent is to change to UNIT_FILE_PRESET_FULL at some point in the future. Distros that want to opt-in can use the config option to change the behaviour. (The option is just a boolean: it would be possible to make it multi-valued, and allow full, enable-only, disable-only, none. But so far nobody has asked for this, and it's better not to complicate things needlessly.) With the configuration option flipped, instead of only doing enablements, perform a full preset on first boot. The reason is that although `/etc/machine-id` might be missing, there may be other files provisioned in `/etc` (in fact, this use case is mentioned in `log_execution_mode`). Some of those possible files include enablement symlinks even if presets dictate it should be disabled. Such a seemingly contradictory situation occurs in {RHEL,Fedora} CoreOS, where we ship `/etc` as if `preset-all` were called. However, we want to allow users to disable default-enabled services via Ignition, which does this by creating preset dropins before switchroot. (For why we do `preset-all` at compose time, see: coreos/fedora-coreos-config#77). For example, the composed FCOS image has a `enable zincati.service` preset and an enablement for that in `/etc`, while at boot time when we switch root, there may be a `disable zincati.service` preset with higher precedence. In that case, we want systemd to disable the service. This is essentially a revert of 304b307. It seems like systemd *used* to do this, but it was changed to try to make the container workflow a bit faster. Resolves: coreos/fedora-coreos-tracker#392 Co-authored-by: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>

We need to delete any enablement symlinks for a unit before sending it to a preset directive. This will help to disable that unit completely. This is a short-term solution until the upstream systemd PR (systemd/systemd#15205) gets accepted. Fixes coreos/fedora-coreos-tracker#392

FCOS ships some pre-created enablement symlinks, and the systemd does nothing even after recording in the preset that they should be removed. In order to fix this behavior, we're not only deleting enablement symlinks for a given unit but also recording in the preset directive to disable that unit completely. This is a short-term solution until the upstream systemd PR (systemd/systemd#15205) gets accepted. Fixes coreos/fedora-coreos-tracker#392

For services where FCOS ships a symlink in /etc, if the user tries to disable the service via Ignition, systemd ignores the disablement directive in the preset. This behavior can be fixed by deleting enablement symlinks for a given unit before recording in the preset directive. This is a short-term solution until the upstream systemd PR (systemd/systemd#15205) gets accepted. Fixes coreos/fedora-coreos-tracker#392

For services where FCOS ships a symlink in /etc, if the user tries to disable the service via Ignition, systemd ignores the disablement directive in the preset. Avoid this behavior by deleting the enablement symlinks when disabling a unit, but continue to record the disablement in the preset file. This is a short-term solution until the upstream systemd PR (systemd/systemd#15205) is merged and widely deployed. Fixes coreos/fedora-coreos-tracker#392

dustymabe · 2022-05-06T16:50:56Z

Might be good to have a f-c-c external kola test for this too.

Test for: coreos/fedora-coreos-tracker#392

dustymabe · 2022-05-18T19:18:53Z

This landed in ignition v2.14.0.

dustymabe · 2022-05-25T20:49:31Z

The fix for this went into testing stream release 36.20220522.2.1. Please try out the new release and report issues.

dustymabe · 2022-06-07T19:54:52Z

The fix for this went into stable stream release 36.20220522.3.0.

A compile time option is added to select behaviour: by default UNIT_FILE_PRESET_ENABLE_ONLY is still used, but the intent is to change to UNIT_FILE_PRESET_FULL at some point in the future. Distros that want to opt-in can use the config option to change the behaviour. (The option is just a boolean: it would be possible to make it multi-valued, and allow full, enable-only, disable-only, none. But so far nobody has asked for this, and it's better not to complicate things needlessly.) With the configuration option flipped, instead of only doing enablements, perform a full preset on first boot. The reason is that although `/etc/machine-id` might be missing, there may be other files provisioned in `/etc` (in fact, this use case is mentioned in `log_execution_mode`). Some of those possible files include enablement symlinks even if presets dictate it should be disabled. Such a seemingly contradictory situation occurs in {RHEL,Fedora} CoreOS, where we ship `/etc` as if `preset-all` were called. However, we want to allow users to disable default-enabled services via Ignition, which does this by creating preset dropins before switchroot. (For why we do `preset-all` at compose time, see: coreos/fedora-coreos-config#77). For example, the composed FCOS image has a `enable zincati.service` preset and an enablement for that in `/etc`, while at boot time when we switch root, there may be a `disable zincati.service` preset with higher precedence. In that case, we want systemd to disable the service. This is essentially a revert of 304b307. It seems like systemd *used* to do this, but it was changed to try to make the container workflow a bit faster. Resolves: coreos/fedora-coreos-tracker#392 Co-authored-by: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>

A compile time option is added to select behaviour: by default UNIT_FILE_PRESET_ENABLE_ONLY is still used, but the intent is to change to UNIT_FILE_PRESET_FULL at some point in the future. Distros that want to opt-in can use the config option to change the behaviour. (The option is just a boolean: it would be possible to make it multi-valued, and allow full, enable-only, disable-only, none. But so far nobody has asked for this, and it's better not to complicate things needlessly.) With the configuration option flipped, instead of only doing enablements, perform a full preset on first boot. The reason is that although `/etc/machine-id` might be missing, there may be other files provisioned in `/etc` (in fact, this use case is mentioned in `log_execution_mode`). Some of those possible files include enablement symlinks even if presets dictate it should be disabled. Such a seemingly contradictory situation occurs in {RHEL,Fedora} CoreOS, where we ship `/etc` as if `preset-all` were called. However, we want to allow users to disable default-enabled services via Ignition, which does this by creating preset dropins before switchroot. (For why we do `preset-all` at compose time, see: coreos/fedora-coreos-config#77). For example, the composed FCOS image has a `enable zincati.service` preset and an enablement for that in `/etc`, while at boot time when we switch root, there may be a `disable zincati.service` preset with higher precedence. In that case, we want systemd to disable the service. This is essentially a revert of 304b307. It seems like systemd *used* to do this, but it was changed to try to make the container workflow a bit faster. Resolves: coreos/fedora-coreos-tracker#392 Co-authored-by: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl> (cherry picked from commit 9365158)

A compile time option is added to select behaviour: by default UNIT_FILE_PRESET_ENABLE_ONLY is still used, but the intent is to change to UNIT_FILE_PRESET_FULL at some point in the future. Distros that want to opt-in can use the config option to change the behaviour. (The option is just a boolean: it would be possible to make it multi-valued, and allow full, enable-only, disable-only, none. But so far nobody has asked for this, and it's better not to complicate things needlessly.) With the configuration option flipped, instead of only doing enablements, perform a full preset on first boot. The reason is that although `/etc/machine-id` might be missing, there may be other files provisioned in `/etc` (in fact, this use case is mentioned in `log_execution_mode`). Some of those possible files include enablement symlinks even if presets dictate it should be disabled. Such a seemingly contradictory situation occurs in {RHEL,Fedora} CoreOS, where we ship `/etc` as if `preset-all` were called. However, we want to allow users to disable default-enabled services via Ignition, which does this by creating preset dropins before switchroot. (For why we do `preset-all` at compose time, see: coreos/fedora-coreos-config#77). For example, the composed FCOS image has a `enable zincati.service` preset and an enablement for that in `/etc`, while at boot time when we switch root, there may be a `disable zincati.service` preset with higher precedence. In that case, we want systemd to disable the service. This is essentially a revert of 304b307. It seems like systemd *used* to do this, but it was changed to try to make the container workflow a bit faster. Resolves: coreos/fedora-coreos-tracker#392 Co-authored-by: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>

Test for: coreos/fedora-coreos-tracker#392

A compile time option is added to select behaviour: by default UNIT_FILE_PRESET_ENABLE_ONLY is still used, but the intent is to change to UNIT_FILE_PRESET_FULL at some point in the future. Distros that want to opt-in can use the config option to change the behaviour. (The option is just a boolean: it would be possible to make it multi-valued, and allow full, enable-only, disable-only, none. But so far nobody has asked for this, and it's better not to complicate things needlessly.) With the configuration option flipped, instead of only doing enablements, perform a full preset on first boot. The reason is that although `/etc/machine-id` might be missing, there may be other files provisioned in `/etc` (in fact, this use case is mentioned in `log_execution_mode`). Some of those possible files include enablement symlinks even if presets dictate it should be disabled. Such a seemingly contradictory situation occurs in {RHEL,Fedora} CoreOS, where we ship `/etc` as if `preset-all` were called. However, we want to allow users to disable default-enabled services via Ignition, which does this by creating preset dropins before switchroot. (For why we do `preset-all` at compose time, see: coreos/fedora-coreos-config#77). For example, the composed FCOS image has a `enable zincati.service` preset and an enablement for that in `/etc`, while at boot time when we switch root, there may be a `disable zincati.service` preset with higher precedence. In that case, we want systemd to disable the service. This is essentially a revert of 304b307. It seems like systemd *used* to do this, but it was changed to try to make the container workflow a bit faster. Resolves: coreos/fedora-coreos-tracker#392 Co-authored-by: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>

lucab added component/ignition component/systemd kind/bug needs/investigation labels Feb 18, 2020

dustymabe mentioned this issue Mar 6, 2020

Can't disable docker.socket with Ignition #229

Closed

jlebon mentioned this issue Mar 23, 2020

manager: optionally, do a full preset on first boot systemd/systemd#15205

Merged

LorbusChris mentioned this issue Mar 29, 2020

Uploading to cloud platforms: GCP #147

Closed

dustymabe mentioned this issue Mar 31, 2020

systemd: Activative via zincati.timer, not by default coreos/zincati#251

Closed

This comment has been minimized.

Sign in to view

jlebon mentioned this issue Apr 21, 2020

40ignition-ostree: add service for deleting /etc enablements coreos/fedora-coreos-config#363

Closed

cgwalters mentioned this issue May 8, 2020

kola/tests/ignition: add test for checking journald logs coreos/coreos-assembler#1329

Merged

lucab removed the needs/investigation label May 28, 2020

lucab mentioned this issue May 28, 2020

How to disable Auto-Updates #502

Closed

vrutkovs mentioned this issue Jun 23, 2020

Zincati generates a lot of errors on FCOS okd-project/okd#219

Closed

LorbusChris mentioned this issue Jun 27, 2020

OKD4.4-Installation fails on the latest version(32.20200601.3.0) of FCOS okd-project/okd#229

Closed

cgwalters mentioned this issue Jul 1, 2020

[fcos] disable zincati on masters/workers openshift/machine-config-operator#1890

Merged

LorbusChris mentioned this issue Jul 5, 2020

[fcos] templates/common/_base/units: disable docker openshift/machine-config-operator#1898

Merged

LorbusChris mentioned this issue Jul 14, 2020

templates/common/_base/units: disable Zincati service openshift/machine-config-operator#1919

Merged

sohankunkerkar closed this as completed May 6, 2022

dustymabe added the status/pending-upstream-release Fixed upstream. Waiting on an upstream component source code release. label May 6, 2022

travier added a commit to travier/fedora-coreos-config that referenced this issue May 12, 2022

test/kola: Test disabling units via Ignition

7d8258f

Test for: coreos/fedora-coreos-tracker#392

travier mentioned this issue May 12, 2022

test/kola: Test disabling units via Ignition coreos/fedora-coreos-config#1736

Merged

travier added a commit to travier/fedora-coreos-config that referenced this issue May 18, 2022

test/kola: Test disabling units via Ignition

4ea7915

Test for: coreos/fedora-coreos-tracker#392

dustymabe added status/pending-testing-release Fixed upstream. Waiting on a testing release. and removed status/pending-upstream-release Fixed upstream. Waiting on an upstream component source code release. labels May 18, 2022

travier added a commit to coreos/fedora-coreos-config that referenced this issue May 18, 2022

test/kola: Test disabling units via Ignition

b862d0f

Test for: coreos/fedora-coreos-tracker#392

dustymabe mentioned this issue May 18, 2022

tracker: This Month in Fedora CoreOS May 2022 #1188

Closed

dustymabe added status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. and removed status/pending-testing-release Fixed upstream. Waiting on a testing release. labels May 25, 2022

dustymabe removed the status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. label Jun 7, 2022

HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023

test/kola: Test disabling units via Ignition

3ad4d93

Test for: coreos/fedora-coreos-tracker#392

HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023

test/kola: Test disabling units via Ignition

c9e2687

Test for: coreos/fedora-coreos-tracker#392

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to disable zincati.service using Ignition #392

Unable to disable zincati.service using Ignition #392

dustymabe commented Feb 16, 2020

dustymabe commented Mar 6, 2020

dustymabe commented Mar 20, 2020

dustymabe commented Mar 20, 2020

jlebon commented Mar 20, 2020

dustymabe commented Mar 20, 2020 •

edited

Loading

jlebon commented Mar 23, 2020

dustymabe commented Mar 25, 2020

cgwalters commented Mar 31, 2020 •

edited

Loading

This comment has been minimized.

sgohl commented May 26, 2021

bgilbert commented May 27, 2021

jlebon commented May 27, 2021

dustymabe commented May 6, 2022

dustymabe commented May 18, 2022

dustymabe commented May 25, 2022

dustymabe commented Jun 7, 2022

Unable to disable zincati.service using Ignition #392

Unable to disable zincati.service using Ignition #392

Comments

dustymabe commented Feb 16, 2020

dustymabe commented Mar 6, 2020

dustymabe commented Mar 20, 2020

dustymabe commented Mar 20, 2020

jlebon commented Mar 20, 2020

dustymabe commented Mar 20, 2020 • edited Loading

jlebon commented Mar 23, 2020

dustymabe commented Mar 25, 2020

cgwalters commented Mar 31, 2020 • edited Loading

This comment has been minimized.

sgohl commented May 26, 2021

bgilbert commented May 27, 2021

jlebon commented May 27, 2021

dustymabe commented May 6, 2022

dustymabe commented May 18, 2022

dustymabe commented May 25, 2022

dustymabe commented Jun 7, 2022

dustymabe commented Mar 20, 2020 •

edited

Loading

cgwalters commented Mar 31, 2020 •

edited

Loading