Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RFE] Flatcar boot partition size shrink effort #1381

Open
ader1990 opened this issue Feb 28, 2024 · 16 comments · Fixed by flatcar/scripts#1734
Open

[RFE] Flatcar boot partition size shrink effort #1381

ader1990 opened this issue Feb 28, 2024 · 16 comments · Fixed by flatcar/scripts#1734
Labels
kind/feature A feature request

Comments

@ader1990
Copy link

Current situation

Currently, the Flatcar boot partition is set to 128 MB. There was a slow increase over the years of the Linux kernel, dracut and systemd binaries or other required packages in the initrd. The current usage in the latest Alpha is of around 62MB, which is critically close to the half of the size.

From my limited understanding, if we pass the 64MB mark, there will be issues, as the kernel should fit twice for the A/B update to work.

Impact

Doubling the boot partition will limit the old clients to update. Maybe there are other implications, feel free to update the list.

Ideal future situation

Without increasing the boot partition, I see no way for the medium to long term to implement in an upstream clean way things like systemd upgrades, linux kernel upgrades or security features like LUKS encryption/decryption, HSM integration.

Additional comments

For example, the current upgrade to systemd 255 is blocked by this issue and the only way forward is to axe the kbd upstream package contents with the minimal necessary files. The problem is that if we implement such a hack, there are going to be unforseen issues in the future if a user wants to use the upstream kbd package features that we are removing. This solution will bring technical debt in the future, as the initrd required packages need to go through the same process.
flatcar/scripts#1679

The PR to add TPM/TANG support might be blocked too for the same reason: flatcar/scripts#1560.

If in the future, if we want to add LUKS2 support with HSM, there might be again, the same blocker.
@ader1990 ader1990 added the kind/feature A feature request label Feb 28, 2024
@jepio
Copy link
Member

jepio commented Feb 28, 2024

Doubling the boot partition will limit the old clients to update.

We 100% need to keep supporting updates for old-clients.

@pothos
Copy link
Member

pothos commented Feb 28, 2024 •
edited
Loading

From my observation it's often possible to cut things down a bit but yes, at some point we hit a limit. Without a limit one can easily bloat everything and even 512 MB is already too small for some distro's boot partition.

I mentioned this in the chat: A risky plan would be to repartition from the initrd: Shrink part7 to 1 MB or so and move 2-7, to grow part1 by ~64 MB.
However, we first should explore options in flatcar/scripts#1679 (comment) - I have some more hints in #1074 and #688

@ader1990
Copy link
Author

It seems that the shim PR introduced a 0.5M increase in the initrd and the current state of the HEAD flatcar /boot is 64130 KB used vs 64909 KB free.
https://github.com/flatcar/scripts/pull/1589/files#diff-2b0f856f6376d8b6db23b7dbff406f70eb267961cb693d2cf83bfed3c0761de0R213

Which means that the current runway is ~400KB. I managed to sort out the systemd i18n module size increase problem, but the current 400KB runway is impossible for the systemd upgrade, be it 253 or 254 or 255.

Unless we find a solution to decrease the initrd size now, we cannot upgrade systemd. Even if I can squeeze the systemd upgrade to less than the 400KB current runway, the next initrd update would not be possible.

@gabriel-samfira
Copy link
Member

This seems like a pretty difficult limitation to overcome.

What do you folks think about creating a different "channel" that would allow for larger defaults, without the need to support switching from one channel to another? I know this would add additional burden on support and maintenance, but it seems like we need a plan to transition to something that would allow Flatcar to evolve over time. The current disk space constraints seem to become a blocker for future updates.

@ader1990
Copy link
Author

ader1990 commented Mar 4, 2024

The next release window is coming next week, what would be the decision to be taken on this matter?
I see here a few options short/medium term:

  • use various ways of compression to decrease the /boot partition size. Even if we compress with the best and slowest compression algos, as everything is already compressed, that would save a few 1 or 2 MB at most.
  • cleanup the kernel config by removing some of the enabled modules. This is highly risky as people might be using those features / kernel modules and it will break backwards compatibility. Even is we remove some of the more exotic network/storage modules, this change won't matter on the bigger picture.
  • reuse /usr in the initrd as @pothos suggested. Doing this might work, but at what cost? From what I see, the cost would be to diverge more and more from a systemd / gentoo upstream approach. Doing this in initrd means adding another level of complexity that has to be maintained.

All of this options can take weeks if not months to implement, but the current runway is very close to ending (~400KB of left space until 50% is reached on AMD64).

@ader1990 ader1990 mentioned this issue Mar 6, 2024
Draft
ader1990 added a commit to flatcar/scripts that referenced this issue Mar 6, 2024
@ader1990
initrd_size_decrease: remove mlxsw_spectrum/mlxsw_core kernel modules
0ece7b1 
This commit is part of the effort to decrease the initrd size:
Partially-Fixes: flatcar/Flatcar#1381

Signed-off-by: Adrian Vladu <avladu@cloudbasesolutions.com>
This was referenced Mar 6, 2024
Merged
Merged
ader1990 added a commit to flatcar/scripts that referenced this issue Mar 12, 2024
@ader1990
initrd_size_decrease: remove mlxsw_spectrum/mlxsw_core kernel modules
381bac3 
This commit is part of the effort to decrease the initrd size:
Partially-Fixes: flatcar/Flatcar#1381

Signed-off-by: Adrian Vladu <avladu@cloudbasesolutions.com>
@github-project-automation github-project-automation bot moved this from 📝 Needs Triage to Implemented in Flatcar tactical, release planning, and roadmap Mar 12, 2024
@ader1990 ader1990 reopened this Mar 14, 2024
@github-project-automation github-project-automation bot moved this from Implemented to ⚒️ In Progress in Flatcar tactical, release planning, and roadmap Mar 14, 2024
@ader1990 ader1990 mentioned this issue Mar 14, 2024
Merged
2 tasks
@ader1990
Copy link
Author

We will need to revisit this issue soon enough, as the initial effort PR resulted in ~5MB of runway, but the recent systemd and tpm+tang PRs ate ~3MB, so we are left with around 2MB of runway.

@jepio
Copy link
Member

jepio commented Mar 15, 2024

Can you collect data on how systemd + tang ate 3MB?

@ader1990
Copy link
Author

Can you collect data on how systemd + tang ate 3MB?

sure, I ll update flatcar/scripts#1731.

@pothos
Copy link
Member

pothos commented Mar 15, 2024 •
edited
Loading

Here is the report against the previous nightly: image-diff.txt
And against the previous Alpha: image-diff-alpha.txt

@pothos
Copy link
Member

pothos commented Mar 15, 2024 •
edited
Loading

Here is the report against the previous nightly: image-diff.txt And against the previous Alpha: image-diff-alpha.txt

Maybe tpm2 in the initrd could be loaded from /sysusr? Something also started to pull in libcurl.

@dongsupark dongsupark moved this from 📝 Needs Triage to ⚒️ In Progress in Flatcar tactical, release planning, and roadmap Mar 18, 2024
@github-project-automation github-project-automation bot moved this from ⚒️ In Progress to Implemented in Flatcar tactical, release planning, and roadmap Mar 18, 2024
@jepio jepio reopened this Mar 18, 2024
@github-project-automation github-project-automation bot moved this from Implemented to ⚒️ In Progress in Flatcar tactical, release planning, and roadmap Mar 18, 2024
@github-actions github-actions bot mentioned this issue Mar 22, 2024
Closed
@ader1990 ader1990 changed the title [RFE] Flatcar boot partition size increase [RFE] Flatcar boot partition size shrink effort Mar 25, 2024
@ader1990
Copy link
Author

Linking #1397, which is part of the larger effort.

@ader1990
Copy link
Author

FYI, the current AMD64 image is very close to reaching 50% /boot partition usage. Hopefully, this PR will give us some breathing room: flatcar/scripts#1955.

@chewi
Copy link
Contributor

chewi commented Jun 20, 2024

We're still delaying the inevitable though. I had a look at this for the first time, but I can't see any obvious gains, short of stripping bits out of the kernel, which we presumably don't want to do. I take it the OEM-CONFIG parition after the /boot partition is still used in some cases. Does it need all 64MB? That may be safer than trying to resize the ROOT partition at the end, and if we did want to take the space from ROOT, we'd need to move OEM-CONFIG anyway.

@ader1990
Copy link
Author

Hello, while checking out the nmbl presentation https://pretalx.com/devconf-cz-2024/talk/W3AVCT/, also found out about this idea that might look promising in Flatcar's case too -> https://github.com/containers/initoverlayfs?tab=readme-ov-file#what-is-initoverlayfs. Sharing this here, maybe it is worth some investigation at a later time.

@chewi
Copy link
Contributor

chewi commented Jul 11, 2024

I mentioned on Matrix but forgot to say here that GRUB does support ext4, xfs, and btrfs these days, which probably means we could just move the whole kernel image into the usr partition, which should make managing them easier too. I was going to try this out.

@ader1990
Copy link
Author

ader1990 commented Oct 9, 2024

The gentoo profiles update PR brought a ~5% decrease flatcar/scripts#2287

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature A feature request
Projects
Flatcar tactical, release planning, and roadmap
Status: ⚒️ In Progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

initrd_size_decrease: remove mlxsw_spectrum/mlxsw_core kernel modules flatcar/scripts
5 participants
@chewi @gabriel-samfira @pothos @ader1990 @jepio