selinux: relabeling after Ignition breaks on absolute symlinks

[abitzer]

$ journalctl -u ignition-files.service
ignition-files.service: Failed with result 'exit-code'.
Jun 02 14:21:37 localhost ignition[642]: **CRITICAL** : files: op(1b): [**failed**]   relabeling 31 patterns: exit status 255: Cmd: "setfiles" "-vFi0" "-r" "/sysroot" "/sysroot/etc/selinux/targeted/contexts/files/file_contexts" "-f" "-"

on:

rpm-ostree status
State: idle
Deployments:

• ostree://fedora:fedora/x86_64/coreos/next
  Version: 32.20200517.1.0 (2020-05-19T09:23:58Z)
  Commit: 7c23c4735fb3c541586f0a4d3ca956ef93ef7d76f00a19bccf51460bafa7ee97
  GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

But workings on:

rpm-ostree status
State: idle
Deployments:

• ostree://fedora:fedora/x86_64/coreos/testing
  Version: 31.20200517.2.0 (2020-05-19T10:24:32Z)
  Commit: 5c3f8198e72a05adab4eb7d087ef3a625008dcc73bd0416a444cdce084278587
  GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

[lucab]

@abitzer thanks for the report. Which platform/install-method is this on? If not too big, would you be able to attach the Ignition config as well (redacting any secret first)?

[jlebon]

The fact that the boot continued even though Ignition failed should be fixed by coreos/ignition-dracut#188, which will be in the next next and testing releases.

The underlying issue is that setfiles failed. Likely there is a file path there that it's choking on for some reason. (Though that'd be odd too since the only places really that one can drop files is in /var and /etc, and both of those have catch-all patterns).

[abitzer]

I did some more testing and it seems to boil down to:

  links:
    - path: /etc/localtime
      overwrite: true
      user:
        name: test
      group:
        name: test
      target: /usr/share/zoneinfo/Europe/Zurich
      hard: false

The adapted ignition file can be found here.

ignition.ign.txt

I use Qemu on Windows.

[dustymabe]

I'm able to reproduce this with your config.

------                                                                                                                                                                                                                                       
Ignition has failed. Please ensure your config is valid. Note that only                                                                                                                                                                      
Ignition spec v3.0.0+ configs are accepted.                                                                           
                                                                                                                      
A CLI validation tool to check this called ignition-validate can be                                                                                                                                                                          
downloaded from GitHub:                                                                                                                                                                                                                      
    https://github.com/coreos/ignition/releases                                                                                                                                                                                              
------                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                             
Displaying logs from failed units: ignition-files.service                                                             
-- Logs begin at Wed 2020-06-03 01:21:00 UTC, end at Wed 2020-06-03 01:21:06 UTC. --                                  
Jun 03 01:21:05 ignition[676]:         "target": "/usr/share/zoneinfo/Europe/Zurich"                                  
Jun 03 01:21:05 ignition[676]:       }                                                                                                                                                                                                       
Jun 03 01:21:05 ignition[676]:     ]                                                                                                                                                                                                         
Jun 03 01:21:05 ignition[676]:   },                                                                                                                                                                                                          
Jun 03 01:21:05 ignition[676]:   "systemd": {}                                                                                                                                                                                               
Jun 03 01:21:05 ignition[676]: }CRITICAL : Ignition failed: failed to handle relabeling: exit status 255: Cmd: "setfiles" "-vFi0" "-r" "/sysroot" "/sysroot/etc/selinux/targeted/contexts/files/file_contexts" "-f" "-" Stdout: "Relabeled /s
ysroot/etc/passwd from unlabeled to system_u:object_r:passwd_file_t:s0\nRelabeled /sysroot/etc/passwd- from unlabeled to system_u:object_r:passwd_file_t:s0\nRelabeled /sysroot/etc/group from unlabeled to system_u:object_r:passwd_file_t:s
0\nRelabeled /sysroot/etc/group- from unlabeled to system_u:object_r:passwd_file_t:s0\nRelabeled /sysroot/etc/shadow from unlabeled to system_u:object_r:shadow_t:s0\nRelabeled /sysroot/etc/shadow- from unlabeled to system_u:object_r:shad
ow_t:s0\nRelabeled /sysroot/etc/gshadow from unlabeled to system_u:object_r:shadow_t:s0\nRelabeled /sysroot/etc/gshadow- from unlabeled to system_u:object_r:shadow_t:s0\nRelabeled /sysroot/etc/subuid from unlabeled to system_u:object_r:e
tc_t:s0\nRelabeled /sysroot/etc/subuid- from unlabeled to system_u:object_r:etc_t:s0\nRelabeled /sysroot/etc/subgid from unlabeled to system_u:object_r:etc_t:s0\nRelabeled /sysroot/etc/subgid- from unlabeled to system_u:object_r:etc_t:s0
\nRelabeled /sysroot/etc/.pwd.lock from unlabeled to system_u:object_r:passwd_file_t:s0\nRelabeled /sysroot/var/home from unlabeled to system_u:object_r:home_root_t:s0\nRelabeled /sysroot/var/home/core from unlabeled to unconfined_u:obje
ct_r:user_home_dir_t:s0\nRelabeled /sysroot/var/home/core/.bash_logout from unlabeled to unconfined_u:object_r:user_home_t:s0\nRelabeled /sysroot/var/home/core/.bash_profile from unlabeled to unconfined_u:object_r:user_home_t:s0\nRelabel
ed /sysroot/var/home/core/.bashrc from unlabeled to unconfined_u:object_r:user_home_t:s0\nRelabeled /sysroot/var/home/core/.ssh from unlabeled to unconfined_u:object_r:ssh_home_t:s0\nRelabeled /sysroot/var/home/core/.ssh/authorized_keys.
d from unlabeled to unconfined_u:object_r:ssh_home_t:s0\nRelabeled /sysroot/var/home/core/.ssh/authorized_keys.d/ignition from unlabeled to unconfined_u:object_r:ssh_home_t:s0\nRelabeled /sysroot/var/home/test from unlabeled to unconfine
d_u:object_r:user_home_dir_t:s0\nRelabeled /sysroot/var/home/test/.bash_logout from unlabeled to unconfined_u:object_r:user_home_t:s0\nRelabeled /sysroot/var/home/test/.bash_profile from unlabeled to unconfined_u:object_r:user_home_t:s0\
nRelabeled /sysroot/var/home/test/.bashrc from unlabeled to unconfined_u:object_r:user_home_t:s0\nRelabeled /sysroot/var/home/test/.ssh from unlabeled to unconfined_u:object_r:ssh_home_t:s0\nRelabeled /sysroot/var/home/test/.ssh/authoriz
ed_keys.d from unlabeled to unconfined_u:object_r:ssh_home_t:s0\nRelabeled /sysroot/var/home/test/.ssh/authorized_keys.d/ignition from unlabeled to unconfined_u:object_r:ssh_home_t:s0\nRelabeled /sysroot/var/roothome from unlabeled to sy
stem_u:object_r:admin_home_t:s0\nRelabeled /sysroot/var/roothome/.bash_logout from unlabeled to system_u:object_r:admin_home_t:s0\nRelabeled /sysroot/var/roothome/.bash_profile from unlabeled to system_u:object_r:admin_home_t:s0\nRelabel
ed /sysroot/var/roothome/.bashrc from unlabeled to system_u:object_r:admin_home_t:s0\n" Stderr: "setfiles: statfs(/sysroot/etc/localtime) failed: No such file or directory\n"                                                               
Jun 03 01:21:05 systemd[1]: ignition-files.service: Main process exited, code=exited, status=1/FAILURE                                                                                                                                       
Jun 03 01:21:05 systemd[1]: ignition-files.service: Failed with result 'exit-code'.                                                                                                                                                          
Jun 03 01:21:05 systemd[1]: Failed to start Ignition (files).                                                                                                                                                                                
Jun 03 01:21:05 systemd[1]: ignition-files.service: Triggering OnFailure= dependencies.                               
Press Enter for emergency shell or wait 5 minutes for reboot.

The relevant bit in there is:

Stderr: "setfiles: statfs(/sysroot/etc/localtime) failed: No such file or directory

but earlier in the Ignition logs on the scrollback of my console I do see:

files: createFilesystemsFiles: createFiles: op(5): [started]  writing link "/sysroot/etc/localtime" -> "/usr/share/zoneinfo/Europe/Zurich"
files: createFilesystemsFiles: createFiles: op(5): [finished] writing link "/sysroot/etc/localtime" -> "/usr/share/zoneinfo/Europe/Zurich"

It seems that setfiles is blowing up because the target of the symlink doesn't exist (because we're not in the real root yet). From the emergency shell that I got:

:/root# stat /sysroot/etc/localtime >/dev/null
:/root# stat -L /sysroot/etc/localtime >/dev/null
stat: cannot stat '/sysroot/etc/localtime': No such file or directory

So it looks like it doesn't like the symlink being an absolute path. I'd think this wouldn't be a problem because we pass -r /sysroot to setfiles, but it is. It also looks like it is not a problem in our current stable stream so might be something to do with the switch to f32.

You can workaround this issue by using a relative path (../usr/share/zoneinfo/Europe/Zurich) for your symlink.

[lucab]

This is reproducible even outside of Ignition/initramfs context. Additionally, it fails even if we are already passing the -i flag (ignore non-existing paths). Minimum reproducer from an F32:

# mkdir /root/my-chroot && touch /root/my-chroot/link-target && ln -s /link-target /root/my-chroot/symlink
# echo "/root/my-chroot/symlink" | setfiles -vFi -r /root/my-chroot -f - /etc/selinux/targeted/contexts/files/file_contexts
setfiles: statfs(/root/my-chroot/symlink) failed: No such file or directory

Reported upstream at SELinuxProject/selinux#248.

[abitzer]

Thanks for the great support.
Personally I‘m more worried about the fact that ignition did not fail despite the error. I hope this can be fixed soon.

[dustymabe]

Thanks for the great support.
Personally I‘m more worried about the fact that ignition did not fail despite the error. I hope this can be fixed soon.

Correct. @jlebon mentioned it would be fixed in the next set of releases (should be out today or tomorrow). The probably is a change in how systemd upstream worked and we just now discovered it.

[dustymabe]

Also, I would like to thank you @abitzer for running next and helping us find issues there.

[jlebon]

Upstream patch: https://lore.kernel.org/selinux/20200604200831.28866-1-stephen.smalley.work@gmail.com/T/#u. I've verified that it fixes the bug. I've also submitted a patch to clarify documentation around symlink handling: https://lore.kernel.org/selinux/20200604191240.263819-1-jlebon@redhat.com/T/#u.

Finally, discussing this made me realize that we shouldn't be using setfiles -i as good practice, which lead me to: coreos/ignition#996.

[jlebon]

https://src.fedoraproject.org/rpms/libselinux/pull-request/14

[dustymabe]

https://bodhi.fedoraproject.org/updates/FEDORA-2020-b60c1a0a0a

[dustymabe]

The fix for this went into testing stream release 32.20200809.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 32.20200809.3.0.