ostree-finalize-staged.service fails with: Failed to resolve typeattributeset statement at ... (was: selinux issue)

[ahjohannessen]

Describe the bug

Hi, ostree-finalize-staged is failing due to SELinux issues when upgrading Fedora CoreOS 40.20241019.3.0 to Fedora CoreOS 41.20241027.3.0.

In the meantime I turned off zincati. But I would like to know what is wrong and how I can move forward.

Reproduction steps

Not sure how to reproduce

Expected behavior

Fedora CoreOS 40.20241019.3.0 -> Fedora CoreOS 41.20241027.3.0 should work.

Actual behavior

Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Nov 20 10:13:10 ilb01 ostree[3621]: Finalizing staged deployment
Nov 20 10:13:11 ilb01 ostree[3621]: Copying /etc changes: 41 modified, 0 removed, 87 added
Nov 20 10:13:11 ilb01 ostree[3621]: Copying /etc changes: 41 modified, 0 removed, 87 added
Nov 20 10:13:11 ilb01 ostree[3621]: Refreshing SELinux policy
Nov 20 10:13:15 ilb01 ostree[3631]: Failed to resolve typeattributeset statement at /etc/selinux/targeted/tmp/modules/100/apm/cil:107
Nov 20 10:13:15 ilb01 ostree[3631]: Failed to resolve AST
Nov 20 10:13:15 ilb01 ostree[3631]: semodule:  Failed!
Nov 20 10:13:15 ilb01 ostree[3621]: Refreshed SELinux policy in 4084 ms

System details

• platform: VMWare esxi

• rpm-ostree status -b:

● fedora:fedora/x86_64/coreos/stable
                  Version: 40.20241019.3.0 (2024-10-26T12:34:27Z)
               BaseCommit: 6df70065620571076f242857b9080d747891e2279dff3ed1756270f6889731ce
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
          LayeredPackages: dmidecode haproxy keepalived python unzip

Butane or Ignition config

variant: fcos
version: 1.4.0

passwd:

  users:

    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEArKvwajIJWpB4UPpQLqOCn/Y/g+xsmQa5Z8UFX/QZUpI/xeHgn64Zv+z3UJGKBpMXY33cwIi+1R/Vv1+CrEdZitqWlaBIlDfw+fClBgzqe/0QxaEEwQqX3bfxHZnNryy3YR7aOKiRNv1BnmuYiSyb87GHq+5x5mqw0pvyTqVANjEGJpbivHxTjiE7vJKG83IyzjE+c5ja8wE7HdfwiZ1YzLlD/t1uKz96IV3x1wC+Ku/xijkRHMmI8lCjFXgxCXdsm9hueFBxeH3hFaaFjU3shYzcBcZFRYVOCqbFRsYYtThoENg46j5pziYgPzALE0SSUOdK+nG8dTHk+qirkXgEyw== ahjohannessen@gmail.com

systemd:

  units:

    - name: rpm-ostree-install-utils.service
      enabled: true
      contents: |
        [Unit]
        Description=Layer utils with rpm-ostree
        After=systemd-machine-id-commit.service
        After=network-online.target
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/rpm-ostree install --allow-inactive python unzip dmidecode keepalived haproxy
        ExecStart=/bin/touch /var/lib/%N.stamp
        ExecStart=/bin/systemctl --no-block reboot

        [Install]
        WantedBy=multi-user.target

storage:

  disks:

  filesystems:

  directories:

    - path: /home/core/.bashrc.d
      mode: 0755
      user:
        id: 1000
      group:
        id: 1000

  files:

    - path: /etc/hostname
      contents:
        inline: ilb01

    # Tell systemd to not use a pager when printing information
    - path: /etc/profile.d/systemd-pager.sh
      mode: 0644
      contents:
        inline: |
          export SYSTEMD_PAGER=cat

    # Raise console message logging level from DEBUG (7) to WARNING (4)
    # to hide audit messages from the interactive console.
    - path: /etc/sysctl.d/20-silence-audit.conf
      mode: 0644
      contents:
        inline: |
          kernel.printk=4

    - path: /etc/zincati/config.d/55-updates-strategy.toml
      contents:
        inline: |
          [updates]
          strategy = "periodic"
          [[updates.periodic.window]]
          days = ["Wed"]
          start_time = "09:00"
          length_minutes = 90

    - path: /etc/NetworkManager/system-connections/ens33.nmconnection
      mode: 0600
      contents:
        inline: |
          [connection]
          id=ens33
          type=ethernet
          interface-name=ens33
          [ipv4]
          address1=192.168.81.110/24,192.168.81.2
          dhcp-hostname=ilb01
          dns=8.8.8.8;8.8.4.4;
          dns-search=
          may-fail=false
          method=manual

[dustymabe]

do you have any custom SELinux policies applied to this machine?

[ahjohannessen]

Not sure this counts, but I did this via ansible:

setsebool -P haproxy_connect_any 1 for haproxy and net.ipv4.ip_nonlocal_bind=1 in /etc/sysctl.d/00_ip_nonlocal_bind.conf for keepalived.

I tried to do rpm-ostree reset on two testing machines and then upgrade and after reinstall python unzip dmidecode keepalived haproxy - that worked, but doing the same on production machines did not work. So a bit lost. The production machines differ by using:

variant: fcos
version: 1.5.0

Not sure if that is relevant.

[dustymabe]

I'm guessing it's the setsebool -P haproxy_connect_any 1 causing some weird interaction and not the net.ipv4.ip_nonlocal_bind=1.

[travier]

You might have to reset the SELinux policy to the default (https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_selinux_problems) and then re-apply your boolean changes.

See also:

• https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#_why_do_i_get_selinux_denials_after_updates_if_i_have_local_policy_modifications
• RFE: safe SELinux policy updates #701

[ahjohannessen]

Thanks for the help guys :)

I did this:

core@ilb01:~$ sudo systemctl stop keepalived.service
core@ilb01:~$ sudo ostree admin config-diff | grep policy
M    selinux/targeted/active/policy.kern
M    selinux/targeted/active/policy.linked
M    selinux/targeted/policy/policy.33
core@ilb01:~$ sudo rm -rf /etc/selinux
core@ilb01:~$ sudo cp -aT /usr/etc/selinux /etc/selinux
core@ilb01:~$ sudo rpm-ostree update
...
core@ilb01:~$ sudo systemctl reboot
...
[systemd]
Failed Units: 1
  haproxy.service
core@ilb01:~$ sudo setsebool -P haproxy_connect_any 1
core@ilb01:~$ sudo systemctl restart haproxy.service

That solved it. I hope I do not have to do that for the next update.

Doing sudo ostree admin config-diff | grep policy now outputs:

M    selinux/targeted/policy/policy.33
M    selinux/targeted/active/policy.kern
M    selinux/targeted/active/policy.linked
M    selinux/targeted/active/modules/100/policykit
M    selinux/targeted/active/modules/100/policykit/lang_ext
M    selinux/targeted/active/modules/100/policykit/cil
M    selinux/targeted/active/modules/100/policykit/hll

Not sure what that policykit is from.