low memory systems with large disks fail to provision

[dustymabe]

Currently if you provision a system with low memory along with a large disk the system fails to provision:

:/root# systemctl status ignition-ostree-transposefs-autosave-xfs.service  | cat
× ignition-ostree-transposefs-autosave-xfs.service - Ignition OSTree: Autosave XFS Rootfs Partition
     Loaded: loaded (/usr/lib/systemd/system/ignition-ostree-transposefs-autosave-xfs.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-08-02 21:31:34 UTC; 6min ago
    Process: 898 ExecStart=/usr/libexec/ignition-ostree-transposefs autosave-xfs (code=exited, status=1/FAILURE)
   Main PID: 898 (code=exited, status=1/FAILURE)
        CPU: 23ms

Aug 02 21:31:34 ip-172-31-1-113 systemd[1]: Starting ignition-ostree-transposefs-autosave-xfs.service - Ignition OSTree: Autosave XFS Rootfs Partition...
Aug 02 21:31:34 ip-172-31-1-113 ignition-ostree-transposefs[898]: Root reprovisioning requires at least 3G of RAM
Aug 02 21:31:34 ip-172-31-1-113 systemd[1]: ignition-ostree-transposefs-autosave-xfs.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 21:31:34 ip-172-31-1-113 systemd[1]: ignition-ostree-transposefs-autosave-xfs.service: Failed with result 'exit-code'.
Aug 02 21:31:34 ip-172-31-1-113 systemd[1]: Failed to start ignition-ostree-transposefs-autosave-xfs.service - Ignition OSTree: Autosave XFS Rootfs Partition.
Aug 02 21:31:34 ip-172-31-1-113 systemd[1]: ignition-ostree-transposefs-autosave-xfs.service: Triggering OnFailure= dependencies.

I was trying this on a t3.micro instance (1GB/2CPUs) in AWS with a 100G (or maybe 200G) disk. This was just a test I was doing to verify some terraform I had written was working right, but it wasn't something I expected.

I realize that 1G is pretty low (2G would be more reasonable), but this is just an example.

The autosave-xfs work in coreos/fedora-coreos-config#2320 now makes it so that your root filesystem gets reprovisioned if you have a large disk (to fix the suboptimal behavior described in #1183). The reprovisioning code requires 3G of memory so we end up with the Root reprovisioning requires at least 3G of RAM error.

I'm not sure if there is any "fix" here for this but there are probably options for making the user experience better.

• 1 decide that on low memory systems we just won't try to reprovision the root filesystem to fix the agcount XFS issue
• 2 - give the user a better error message in this case. They actually didn't request the rootfs be reprovisioned so this error message is confusing. Maybe we could recommend they deploy a separate /var filesystem.

If nothing else at least we can point to this issue to document the problem when people report it in the future.

[dustymabe]

if we went with 1. we could drop down a CLHM helper that would tell the user about the suboptimal XFS agcount.

[dustymabe]

We discussed this topic in the community meeting today.

13:06:48* dustymabe | #agreed We will attempt to make the autosave-xfs code not request a
                    | reprovision if the memory in the system is too low. We will also
                    | develop a CLHM helper script that checks agcount in the root
                    | filesystem (if it is XFS) and give the user a proper message if
                    | it's too low.