zincati.service is missing zincati user.

[gtherond]

Hi team, I think I've might have catch up a bug with the way FCOS handle zincati or something unclear/missing within the FCOS documentation.

Zincati service unit specify that the agent need to be runned as the zincati user/group, and therefore that the zincati configuration files should be either owned by such user or created with a relaxed 644 permission for the configuration to be written by root while doing the ignition provisioning but still allowing zincati agent to access/read them.

When using an ignition configuration file specifying user:zincati and group:zincati as user/group directive for the file section, without previously create the appropriate zincati user, it end up with a failed and hanging boot because the zincati user doesn't exist initially within the FCOS image.

So either FCOS need to have this user created by default, or the FCOS documentation need to be updated to reflect that situation.

Here is a capture of the boot error:

TBN: Debugging this issue was a pain point as the boot endup freezing with a emergency.service failed message that do not bring any rescue console. I may have to open another issue related to ignition or FCOS itself as not showing any debug/rescue command when ignition fail hard is a little bit cumbersome.

PS: And I can't get rpm-ostree information due to the upper note.

[dustymabe]

Hi team, I think I've might have catch up a bug with the way FCOS handle zincati or something unclear/missing within the FCOS documentation.

Zincati service unit specify that the agent need to be runned as the zincati user/group, and therefore that the zincati configuration files should be either owned by such user or created with a relaxed 644 permission for the configuration to be written by root while doing the ignition provisioning but still allowing zincati agent to access/read them.

When using an ignition configuration file specifying user:zincati and group:zincati as user/group directive for the file section, without previously create the appropriate zincati user, it end up with a failed and hanging boot because the zincati user doesn't exist initially within the FCOS image.

So either FCOS need to have this user created by default, or the FCOS documentation need to be updated to reflect that situation.

Yes it does seem there might be a bug or a misconfiguration in the OS here. For now you can workaround by doing either

• using the default owner/group (root/root) with 644 permissions
• specify the uid/gid (980/980) instead of the name

I think either of those should work.

Here is a capture of the boot error:

TBN: Debugging this issue was a pain point as the boot endup freezing with a emergency.service failed message that do not bring any rescue console. I may have to open another issue related to ignition or FCOS itself as not showing any debug/rescue command when ignition fail hard is a little bit cumbersome.

I get an emergency shell just fine when reproducing this:

}CRITICAL : Ignition failed: failed to create files: failed to create files: error creating /sysroot/etc/zincati/config.d/51-rollout-wariness.toml: error setting file permissions for /sysroot/etc/zincati/config.d/51-rollout-wariness.toml: failed to determine correct uid and gid for /sysroot/etc/zincati/config.d/51-rollout-wariness.toml: No such user "zincati": user: unknown user user "zincati" not found
INFO     : Ignition 2.1.1
INFO     : Stage: umount
INFO     : reading system config file "/usr/lib/ignition/base.ign"
DEBUG    : parsing config with SHA512: ff6a5153be363997e4d5d3ea8cc4048373a457c48c4a5b134a08a30aacd167c1e0f099f0bdf1e24c99ad180628cd02b767b863b5fe3a8fce3fe1886847eb8e2e
INFO     : umount: umount passed
INFO     : Ignition finished successfully
Press Enter for emergency shell or wait 3 minutes 15 seconds for reboot.      

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


:/#

Can you either try accessing the serial console or removing the console=ttyS0 entry from the kernel command line?

PS: And I can't get rpm-ostree information due to the upper note.

If you just add a link to the artifact you're trying to boot that should be enough information.

[gtherond]

Hi @dustymabe thank for this quick answer, you'll find the used file in here: https://github.com/gtherond/ignition/blob/2c9277fe90ce7148c8bad2954e628696f42ff7b0/src/bootstrap.yml

I didn't used the group directive with this configuration as the group is correctly created and named/detected.

I'll test with the UID later on today or this weekend.

All in all, that issue may be related with /etc/passwd file being locked and so unavailable at ignition boot time as I found that additional message later while debugging.
It arise when trying to merge or replace this config with a remote one that do not re-declare core and zincati users making the whole boot process fail.

[lucab]

@gtherond thanks for the report.

Both the zincati and core user + group are part of the OS, so you don't have to create them.

As Dusty said, if those configuration files do not contain secrets then the default mode 0644 will work fine for Zincati to read them.

Does this work if you drop the zincati user creation from your config?

[dustymabe]

Does this work if you drop the zincati user creation from your config?

I can answer this one. It fails if I specify the user/group ownership of the file via name (zincati).

[cgwalters]

As Dusty said, if those configuration files do not contain secrets then the default mode 0644 will work fine for Zincati to read them.

Yes, this is the correct thing to do. There's no reason to make configuration files in /etc owned by the service user if they're not secret. In fact I'd say doing so is an anti-pattern because it allows a compromised service to potentially change its configuration persistently.

In other words just make it owned by root:root.

[dustymabe]

In other words just make it owned by root:root.

Agree. Though I think there is still a bug here. During ignition we should be able to write files out and make them owned by the zincati user without having to know what the uid of zincati is.

[cgwalters]

During ignition we should be able to write files out and make them owned by the zincati user without having to know what the uid of zincati is.

I think we should be using DynamicUser=yes for users like this anyways; and doing that directly conflicts with writing files owned by those users in Ignition. Basically, don't do it.

[cgwalters]

xref coreos/fedora-coreos-pinger#29

[lucab]

(I'd rather not start hijacking tickets, it's very hard to triage stuff this way)

The zincati system user is used for a few things, among which polkit rules. I don't think it can be dropped in favor of DynamicUser=yes, but my knowledge here is shallow and I may be completely wrong. If so, anybody feel free to bring proposals to a Zincati ticket.

It fails if I specify the user/group ownership of the file via name (zincati).

So, while not a blocker for @gtherond (who can just use default mode/ownership as shown in the docs), this is still a bug worth investigating.

I can clearly see the entries in /etc/passwd and /etc/group on a live system, so this may be some funkiness in the way Ignition consume them. Or something wrong in how we generate/integrate them.

[gtherond]

So, all in all, you're right, having root owning the configuration file and making it available to an isolated service using the 0644 modal access right is better I'll fix my ownership and rights configuration accordingly.

However, the root cause of this issue is the zincati user creation being problematic as @dustymabe explained it way better than me ^^

I'll continue to test and fix my configuration, feel free to consider that issue as close if you think that this user discussion require more than just an issue, however, I would really push for the documentation page to explain that and have a higher documentation page explaining how users are set/configured within FCOS as its kinda frustrating to "discover" it at runtime, thankfully you guys are prompt to answer and follow up either on github or IRC, but I really think that a better documentation could benefit new comers to FCOS.

PS: I'll be more than happy to wrote that documentation if you want.

[cgwalters]

I can clearly see the entries in /etc/passwd and /etc/group on a live system, so this may be some funkiness in the way Ignition consume them. Or something wrong in how we generate/integrate them

I think it's pretty simple, ignition-files.service runs in the initramfs, systemd-sysusers.service runs in the real root.

[lucab]

Ah, that looks indeed painfully right.
Then I guess packaging cannot just rely on the sysusers fragment but needs to have a specfile snippet like https://fedoraproject.org/wiki/Packaging:UsersAndGroups#Dynamic_allocation.

https://fedoraproject.org/wiki/Changes/Adopting_sysusers.d_format seems to be still ongoing.

[lucab]

@gtherond agreed on the docs side comment. I think you are basically looking for coreos/fedora-coreos-docs#23 to be fixed. Feel free to followup there with anything else you see that should be covered.

[gtherond]

@lucab yep, I'll subscribe to that issue and look at what I can fix within it this weekend.

Can I close the issue so ? Or do you guys want it to be opened for records ?

[lucab]

@gtherond let's keep this open, we need to fix the package anyway. @rfairley had some further doubts on the specfile/sysusers split, we'll close this when we land the fix.

[jlebon]

We discussed this in IRC: we need to run systemd-sysusers from the initramfs before the Ignition files stage. Then, Ignition configs will be able to reference users/groups that were created via sysusers.d config files.

This is also discussed in #155.