Upgrade fails due to minimum-free-space-percent exceeding 3%

[anuraagrijal3138]

Describe the bug
While upgrading FCOS I get the following error.

Feb 03 21:05:56 kubevirt-worker-3 zincati[1096]: [INFO ] target release '33.20201201.3.0' selected, proceeding to stage it
Feb 03 21:06:06 kubevirt-worker-3 zincati[1096]: [ERROR] failed to stage deployment: rpm-ostree deploy failed:
Feb 03 21:06:06 kubevirt-worker-3 zincati[1096]:     error: While pulling cad80088392fe43bd3cadf0481c3267f199afa7d9f83bc03937ffdbf5ebbc6da: Writing content object: min-free-space-percent '3%' would be exceeded, at least 16.4 kB requested

Reproduction steps
Steps to reproduce the behavior:

1. Upgrade FCOS with current disk status.(Does not apply to everyone)

Expected behavior
The upgrade is successful.

Actual behavior
I tried two ways to upgrade

1. starting zincati.service and editing file "/etc/zincati/config.d/90-disable-auto-updates.toml" to include and rebooting.

[updates]
enabled=true
strategy="immediate"

sudo rpm-ostree cleanup -m
sudo rpm-ostree refresh-md
sudo rpm-ostree upgrade

In both case, I get the same error.

error: While pulling cad80088392fe43bd3cadf0481c3267f199afa7d9f83bc03937ffdbf5ebbc6da: Writing content object: min-free-space-percent '3%' would be exceeded, at least 16.4 kB requested

System details

• Bare Metal
• Fedora CoreOS 32.20200726.3.1 --> Fedora CoreOS 33.20201201.3.0

Additional information
I had installed FCOS in physical servers 5-6 months ago. Auto-update was turned off.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G     0   32G   0% /dev/shm
tmpfs            32G   12M   32G   1% /run
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/sdc4       2.3G  1.7G  592M  75% /sysroot
tmpfs            32G     0   32G   0% /tmp
/dev/sdc5       9.8G  1.8G  7.6G  19% /var
/dev/sdc9       2.0G  6.6M  1.8G   1% /var/home
/dev/sdc8       2.0G  6.1M  1.8G   1% /var/tmp
/dev/sdc6        35G  7.9G   25G  25% /var/lib/docker
/dev/sdc7       3.9G  385M  3.3G  11% /var/log
/dev/sdc1       364M   85M  256M  25% /boot
/dev/sdc2       127M  8.5M  119M   7% /boot/efi
tmpfs           6.3G     0  6.3G   0% /run/user/1001

In which directory is rpm-ostree trying to write the new FCOS image to? Can I download it in another directory and upgrade it?

Thank you!

[jlebon]

What you're hitting is related to #586 (see also the first NOTE about start_mib in https://docs.fedoraproject.org/en-US/fedora-coreos/storage/).

We recommend reserving at least 5G of space to the root partition (that's /dev/sdc4) to allow space for future updates. This isn't strictly enforced however which makes it an easy trap to fall into.

There's no easy way to get out of this unfortunately, and it's possible you may have to reprovision. If you have an extra disk around, you could use that to keep your Docker storage and copy it back in after reprovisioning. (Of course, you could do that for all the other extra partitions too if e.g. you want to keep the logs.)

Another approach if you don't mind shrinking /var (since e.g. you already have /var/home and /var/lib/docker on separate containers) is to reprovision but feeding Ignition the same starting offsets they're currently at, except for /var, which would start later so that the sysroot has more space. The net effect would be that everything in the other partitions would be saved, apart from /var itself (which should be fine).

sudo rpm-ostree upgrade

Note that upgrading directly via rpm-ostree upgrade is not recommended on FCOS. There are barrier releases which we expect nodes to go through which fix bugs or migrate nodes to use new features, etc... Doing rpm-ostree upgrade skips over all that. (We need to do more work there to make that more obvious.)

[jlebon]

You might be able to delay dealing with this by lowering the threshold in /ostree/repo/config and doing rpm-ostree cleanup -p first.

[bgilbert]

We discussed this during the community meeting today. Notes here.

[bgilbert]

We now warn in the MOTD if the root partition is too small. We're using #586 to track behavior improvements in this area, so I'll close this out.