Zincati Stalling Applying Upgrade "Staged Deployment Already Unlocked"

[fifofonix]

Describe the bug

On a recently Fedora40 next provisioned node, having rpm-ostree installed a package with --apply-live, the transient layer, seems somehow to interfere with subsequent OS upgrades with zincati erroring.

Reproduction steps

Full disclosure here, I haven't sought to reproduce this yet, but these are the steps in this nodes history:

1. Provision a Fedora40 node.
2. Layer a package with --apply-live (see systemd unit doing this later).
3. Don't reboot. Transient layer active.
4. Await new OS update and observe zincati error

Expected behavior

Node should apply new OS update.

Actual behavior

Zincati errors with messages like:

zincati[1848]: [INFO  zincati::update_agent::actor] staged deployment '40.20240416.1.1' available, proceeding to finalize it
zincati[1848]: [ERROR zincati::update_agent::actor] failed to finalize deployment: rpm-ostree finalize-deployment failed:
zincati[1848]:     error: Staged deployment already unlocked

System details

• Next 40.20240416.1.0 running on vSphere

Butane or Ignition config

Unit applying rpm-ostree install with its zincati dependency.


[Unit]
Description=Layer open-vm-tools with rpm-ostree
After=network-online.target
# We run before `zincati.service` to avoid conflicting rpm-ostree
# transactions.
Before=zincati.service
ConditionPathExists=!/var/lib/%N.stamp

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/rpm-ostree install --idempotent --apply-live --allow-inactive open-vm-tools
ExecStart=/bin/touch /var/lib/%N.stamp
ExecStart=/usr/bin/systemctl start vmtoolsd

[Install]
WantedBy=multi-user.target

### Additional information

core@d-some-host: ~ $ sudo rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
DriverState: active; update staged: 40.20240416.1.1
Deployments:
fedora:fedora/x86_64/coreos/next
Version: 40.20240416.1.1 (2024-04-19T04:22:07Z)
BaseCommit: 0006e2ccac33f9b6e944c34ab5bfc1cecacd505fb514488fcc5cb35c151af90b
Commit: b620b0ca1029858bfa2f8fdce5af0a4948a6452023968d5b659426cbe6cfd9e0
GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
Diff: 6 upgraded, 10 added
LayeredPackages: open-vm-tools

● fedora:fedora/x86_64/coreos/next
Version: 40.20240416.1.0 (2024-04-16T14:38:34Z)
BootedCommit: 658ac0d06cab242eb022c5dcb5d1a85c2122cd012c26b4bdc24666b8cbe3866d
LiveCommit: 6e4c1608a239d457566360ecbbf7cf290fcb8bc72f366de99a68076a5ecbbba0
LiveDiff: 10 added
GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
Unlocked: transient
core@d-some-host: ~ $ sudo systemctl status zincati
● zincati.service - Zincati Update Agent
Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
/etc/systemd/system/zincati.service.d
└─99-http-proxy.conf
Active: active (running) since Thu 2024-04-18 16:37:33 UTC; 1 week 0 days ago
Docs: https://github.com/coreos/zincati
Main PID: 1848 (zincati)
Status: "update staged: 40.20240416.1.1"
Tasks: 8 (limit: 4580)
Memory: 3.9M (peak: 8.3M)
CPU: 1min 27.403s
CGroup: /system.slice/zincati.service
└─1848 /usr/libexec/zincati agent -v

Apr 26 11:53:52 d-some-host zincati[1848]: [ERROR zincati::update_agent::actor] failed to finalize deployment: rpm-ostree finalize-deployment >
Apr 26 11:53:52 d-some-host zincati[1848]: error: Staged deployment already unlocked
Apr 26 11:53:52 d-some-host zincati[1848]:
Apr 26 11:58:58 d-some-host systemd[1]: zincati.service: Got notification message from PID 687635, but reception only permitted for main PID 1>
Apr 26 11:58:58 d-some-host zincati[1848]: [ERROR zincati::update_agent] failed to check for interactive sessions: failed to deserialize outpu>
Apr 26 11:58:58 d-some-host zincati[1848]: [WARN zincati::update_agent] assuming no active sessions and proceeding anyway
Apr 26 11:58:58 d-some-host zincati[1848]: [INFO zincati::update_agent::actor] staged deployment '40.20240416.1.1' available, proceeding to f>
Apr 26 11:58:59 d-some-host zincati[1848]: [ERROR zincati::update_agent::actor] failed to finalize deployment: rpm-ostree finalize-deployment >
Apr 26 11:58:59 d-some-host zincati[1848]: error: Staged deployment already unlocked
Apr 26 11:58:59 d-some-host zincati[1848]:

[jlebon]

This is a dupe of #1691. Will fix the docs shortly to at least have a workaround.