Flatcar SDK: add experimental prefix builds #1158

t-lo · 2023-09-20T08:50:36Z

This change adds experimental prefix builds to the Flatcar SDK.

Prefix builds use a custom sys prefix path and emerge all binaries and runtime dependencies into that prefix.

This path can then e.g. be shipped as a portable sysext since it includes all dependencies, and has libraries at a custom path so these do not conflict with libraries on target systems.

Prefix uses a staging environment (path) featuring a full-blown development environment, and a "final" environment for installing. Staging and final need to be created using setup_prefix first, which will also create an emerge wrapper to emerge ebuilds into staging and subsequently final. The root fs in final may then e.g. be used to create a distro independent, portable sysext.

See the readme included with this PR, at https://github.com/flatcar/scripts/blob/t-lo/experimental-prefix-support/PREFIX.md, for more information.

This PR is a vetted, rebased, and cleaned up version of #1151 . In the previous PR @pothos had helpful and interesting feedback regarding library linker paths - happy to continue this discussion here!

That PR in turn was a rebased version of the original Flatcar prefix work by @chewi , @jepio , and @HappyTobi here: https://github.com/chewi/flatcar-distro/tree/hackathon-prefix-builds .

How to use

Prefix requires cross-boss to be available locally.
Tested with the nightly SDK version 3740.0.0+nightly-20230927-2100 (the nightly after that failed so there's no newer SDK available).
The latest development is in the t-lo/experimental-prefix-support branch.

git clone https://github.com/chewi/flatcar-distro.git
cd flatcar-distro
git checkout t-lo/experimental-prefix-support

git clone https://github.com/chewi/cross-boss

./run_sdk_container -t -V 3740.0.0+nightly-20230927-2100

In the SDK container, initialise a prefix, then build (for example) python into it:

# Create prefix staging and final; bootstrap staging env
# Syntax is "setup_prefix <name> <prefix-path>
./setup_prefix prefixtest /usr/local/
emerge-prefix-prefixtest-amd64-usr python

Prefixes are architecture aware; the same prefix name can be used for multiple archs

./setup_prefix --board=arm64-usr prefixtest /usr/local/
emerge-prefix-prefixtest-arm64-usr python

"Final" environments will be in __prefix__/<board>/prefixtest after the builds conclude. __prefix__/<board>/prefixtest/root contain the root FS and can be used to create sysexts.

Testing done

Built the above (see "How to use") for amd64 and arm64, created a python sysext, tested on Flatcar x86_64 and arm64 QEmu instances.

Also, added upstream Gentoo as a 4th overlay to a prefix and was able to seamlessly emerge Gentoo packages. Tested a "midnight commander" sysext on both amd64 and arm64 qemu instances.

Advanced usage

Developers can actually add Gentoo as a custom overlay to their prefix once it was set up.

Assuming a prefix my-testprefix with path /usr/local/pfx, simply:
- create /build/prefix-amd64-usr/my-testprefix/root/usr/local/pfx/etc/portage/repos.conf/gentoo.conf and /home/sdk/trunk/src/scripts/__prefix__/amd64-usr/my-testprefix/root/usr/local/pfx/etc/portage/repos.conf/gentoo.conf
  with
```
[gentoo]
location = /mnt/host/source/src/third_party/gentoo
priority = -1000
```
- clone Gentoo
```
git clone --depth=1 https://github.com/gentoo/gentoo.git sdk_container/src/third_party/gentoo
```
Now you can use emerge-prefix-my-testprefix-amd64-usr app-misc/mc to emerge midnight commander from Gentoo.

Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
n/a Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

t-lo · 2023-09-20T12:58:05Z

Cgroupsv1 and Cilium tests failed: https://github.com/flatcar/scripts/actions/runs/6246234944?pr=1158 (and the test results markdown is too large to be added as a comment here, lol) - but this seems unrelated to this PR?

All other tests pass.

github-actions · 2023-09-20T12:58:17Z

Build action triggered: https://github.com/flatcar/scripts/actions/runs/6362248084

dongsupark · 2023-09-20T13:04:51Z

That is the same issue as the recent CI failures of cgroup v1 tests with Kernel 6.1.54.
@jepio reported upstream.

t-lo · 2023-09-21T08:32:46Z

Cgroupsv1 and Cilium tests failed: https://github.com/flatcar/scripts/actions/runs/6246234944?pr=1158 (and the test results markdown is too large to be added as a comment here, lol) - but this seems unrelated to this PR?

All other tests pass.

As per Dongsu's comment the cgroupsv1 issue is unrelated, and all other tests pass. The PR should be ready for review.

t-lo · 2023-09-25T12:52:32Z

Rebased on latest main to resolve a conflict with glibc ebuilds.

t-lo · 2023-09-26T10:04:55Z

Rebased on the latest main.
This allows folks to build prefixes using the nightly SDK again - wasn't possible yesterday because the weekend nightly SDK build failed.

t-lo · 2023-09-27T08:06:08Z

Rebased on latest main to ingest the kernel update that fixes the cgroups v1 issue.
Kicking off an new CI build.

krnowak

I'm not entirely sure if another overlay is really needed. I think that ideally everything we need is in coreos-overlay for our stuff and in portage-stable for stuff straight from Gentoo.

Not a fan of having duplicated packages in prefix overlay. I assume this is temporary as they have some fixes for prefix builds that will eventually find their way to either portage-stable or coreos-overlay (I'm looking at app-misc/ca-certificates and sys-apps/baselayout here).

Also, is this really Flatcar specific? I mean - it uses Flatcar tooling and build scripts, but looks like a thing you could pull off using Gentoo repo. I suppose it could be made generic to support building stuff either from Gentoo as primary + extra overlays or from portage-stable as primary + coreos-overlay.

PREFIX.md

build_library/prefix_util.sh

krnowak · 2023-09-29T07:48:02Z

sdk_container/src/third_party/prefix-overlay/eclass/python-utils-r1.eclass

Why is this eclass being added here instead of using the eclass from portage-stable?

This is an import from upstream Gentoo to fix a prefix build issue.
Integrating it with portage-stable is part of the prefix stabilisation process; for now prefix is experimental.

(Also, we carry prefix specific patches on top of upstream)

sdk_container/src/third_party/prefix-overlay/prefix/prefix-final/prefix-final-0.0.1.ebuild

krnowak · 2023-09-29T07:50:29Z

sdk_container/src/third_party/prefix-overlay/skel/etc/portage/package.accept_keywords/portage

@@ -0,0 +1 @@
+sys-apps/portage ~amd64


Not sure I like the idea of pulling in unstable portage.

This shouldn't be necessary now that 3.0.51 is stable. At least 3.0.50 was needed for the transition to Meson, which was primarily motivated by this work.

Unfortunately this is necessary for cross-boss' cb-bootstrap to work. It's only used for bootstrapping the "staging" area and explicitly disabled via PREFIX_KEYWORDS when emerging into the prefix.

gcc-13 is unstable too, right? But if we switch to gcc-13 then we can remove the above.

Stable on amd64, unstable on arm64. We have some accept keywords in coreos-overlay, I suppose that prefix-overlay will need its own.

krnowak · 2023-09-29T08:57:14Z

sdk_container/src/third_party/prefix-overlay/skel/etc/portage/package.mask/coreos

+app-misc/ca-certificates::coreos # coreos ca-certificates doesn't support prefix
+sys-apps/baselayout::coreos # coreos baselayout depends on systemd


Ah, so this explains why those packages are added to the prefix-overlay. Do you plan to work on adding the missing features to coreos-overlay counterparts?

The packages are fundamentally different to their upstream counterparts, which is a shame. I feel like they don't have to differ quite so much. The coreos baselayout could possibly even be an additional package instead.

It depends on the feasibility - but it's sure worth researching. Baselayout in coreos-overlay in particular is very Flatcar specific; if we can manage to use stock Gentoo baselayout then this can be removed here.

Using Gentoo baselayout instead of our own stuff would be nice. Maybe it's even possible, who knows. Just no one had a look at it yet. We have a bunch of our own stuff in our baselayout fork, though.

But until then, I suppose we would need to do some work on our fork to support eprefix in order to make the prefix overlay come out of experimental status.

krnowak · 2023-09-29T08:57:55Z

sdk_container/src/third_party/prefix-overlay/skel/etc/portage/package.mask/cross

@@ -0,0 +1 @@
+>=sys-devel/gcc-13 # FIXME: Cannot cross-compile gcc 13+ using older version


We are going to have gcc 13 soon in SDK, so this could be dropped, I think.

Hopefully! For now it's required to make bootstrapping work. BTW if we switch to gcc-13 then I believe we can also remove the unstable un-mask from bootstrapping.

sdk_container/src/third_party/prefix-overlay/dev-libs/libgpg-error/libgpg-error-1.47.ebuild

t-lo · 2023-09-29T13:07:27Z

I'm not entirely sure if another overlay is really needed. I think that ideally everything we need is in coreos-overlay for our stuff and in portage-stable for stuff straight from Gentoo.

Absolutely agree - for the first (experimental) prefix support in the SDK these are a (temporaty) necessity for 2 reasons:

we need to pull in a few updates from upstream for prefix bootstrapping to work
we need to carry a few patches on top of upstream to fix prefix builds.

We aim for prefix-overlay to eventually be merged into our regular overlays. Keeping it separate for now allows us to ingest the prefix feature and ship it with the SDK, which will enable more devs to work on this feature.

Not a fan of having duplicated packages in prefix overlay. I assume this is temporary as they have some fixes for prefix builds that will eventually find their way to either portage-stable or coreos-overlay (I'm looking at app-misc/ca-certificates and sys-apps/baselayout here).

All packages in prefix-overlay need to be justified - if it's just duplicates then we should not carry them.

Also, is this really Flatcar specific? I mean - it uses Flatcar tooling and build scripts, but looks like a thing you could pull off using Gentoo repo. I suppose it could be made generic to support building stuff either from Gentoo as primary + extra overlays or from portage-stable as primary + coreos-overlay.

This feature is meant to extend the scope of the SDK to also build distro independent packages. It's not Flatcar specific but it uses the portability of our SDK container and the well-defined environment we provide in our SDK releases to make prefix builds approachable and easy for a larger developer audience.

This change adds experimental prefix builds to the Flatcar SDK. Prefix builds use a custom sys prefix path and emerge all binaries and runtime dependencies into that prefix. This path can then e.g. be shipped as a portable sysext since it includes all dependencies, and has libraries at a custom path so these do not conflict with libraries on target systems. Prefix uses a staging environment (path) featuring a full-blown development environment, and a "final" environment for installing. Staging and final need to be created using setup_prefix first, which will also create an emerge wrapper to emerge ebuilds into staging and subsequently final. The root fs in final may then e.g. be used to create a distro independent, portable sysext. Co-authored-by: James Le Cuirot <chewi@gentoo.org> Co-authored-by: Jeremi Piotrowski <jpiotrowski@microsoft.com> Co-authored-by: Thilo Fromm <thilofromm@microsoft.com>

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>

@krnowak

Thank you @krnowak! Great review and awesome feedback, as always. Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>

- remove explicit "-multilib" from prefix keywordsas it is set in profile - split heredoc for generating emerge wrapper so we don't need to escape - add sys-apps/bubblewrap and virtual/tmpfiles to package update automation list - use prefix build fix for libgpg-error from upstream Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>

t-lo · 2023-09-29T14:41:06Z

Cleaned up (removed python from prefix; wasn't necessary after all), rebased on latest main, and rebuilt locally.

I've added Gentoo as a custom overlay to my test prefix repos.conf and was able to successfully emerge midnight commander. Tested on qemu flatcar (both amd64 and arm64):

t-lo · 2023-09-29T14:55:53Z

NOTE that for testing the PR you'll need to use the nightly SDK from 2 nights ago - last night's build failed and there is no SDK. To do this, start the container like this:

./run_sdk_container -t -V 3740.0.0+nightly-20230927-2100

krnowak

I'd say let's get this in. Ideally there would be some automatic thing going on (like a github action), so it would catch breakage quickly before it bitrots too much.

t-lo · 2023-09-30T10:38:15Z

I'd say let's get this in. Ideally there would be some automatic thing going on (like a github action), so it would catch breakage quickly before it bitrots too much.

Agreed @krnowak . I've created flatcar/Flatcar#1195 for tracking and added it to the relevant boards.

t-lo temporarily deployed to development September 20, 2023 08:50 — with GitHub Actions Inactive

t-lo requested review from a team and pothos September 20, 2023 08:50

t-lo had a problem deploying to development September 20, 2023 13:20 — with GitHub Actions Error

dongsupark added the main label Sep 21, 2023

t-lo force-pushed the t-lo/experimental-prefix-support branch from 81fb322 to beb1bc3 Compare September 25, 2023 12:51

t-lo had a problem deploying to development September 25, 2023 12:52 — with GitHub Actions Error

t-lo force-pushed the t-lo/experimental-prefix-support branch from beb1bc3 to e8d7d81 Compare September 26, 2023 10:03

t-lo had a problem deploying to development September 26, 2023 10:03 — with GitHub Actions Error

t-lo requested a review from jepio September 26, 2023 10:05

t-lo force-pushed the t-lo/experimental-prefix-support branch from e8d7d81 to 4bc898d Compare September 27, 2023 08:01

t-lo temporarily deployed to development September 27, 2023 08:01 — with GitHub Actions Inactive

krnowak reviewed Sep 29, 2023

View reviewed changes

chewi reviewed Sep 29, 2023

View reviewed changes

sdk_container/src/third_party/prefix-overlay/dev-libs/libgpg-error/libgpg-error-1.47.ebuild Outdated Show resolved Hide resolved

t-lo had a problem deploying to development September 29, 2023 12:54 — with GitHub Actions Error

t-lo had a problem deploying to development September 29, 2023 13:02 — with GitHub Actions Error

t-lo requested review from chewi and krnowak September 29, 2023 13:07

t-lo and others added 4 commits September 29, 2023 15:22

Prefix: add stabilisation TODOs, changelog entry

6f38b9a

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>

Apply suggestions from code review

ba4342d

Thank you @krnowak! Great review and awesome feedback, as always. Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>

SDK prefix: removed spurious dev-libs/python from prefix overlay

f846b03

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>

t-lo force-pushed the t-lo/experimental-prefix-support branch from fe39f9a to f846b03 Compare September 29, 2023 14:37

t-lo had a problem deploying to development September 29, 2023 14:37 — with GitHub Actions Failure

krnowak approved these changes Sep 29, 2023

View reviewed changes

t-lo merged commit 455645c into main Sep 30, 2023
1 check failed

t-lo deleted the t-lo/experimental-prefix-support branch September 30, 2023 10:38

t-lo mentioned this pull request Oct 2, 2023

prefix: remove gcc-13 mask as the SDK now uses gcc-13 #1205

Merged

github-actions bot mentioned this pull request Nov 9, 2023

Monthly contributions report 2023-09-22 - 2023-10-21 flatcar/Flatcar#1244

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flatcar SDK: add experimental prefix builds #1158

Flatcar SDK: add experimental prefix builds #1158

t-lo commented Sep 20, 2023 •

edited

Loading

t-lo commented Sep 20, 2023

github-actions bot commented Sep 20, 2023 •

edited

Loading

dongsupark commented Sep 20, 2023

t-lo commented Sep 21, 2023

t-lo commented Sep 25, 2023

t-lo commented Sep 26, 2023

t-lo commented Sep 27, 2023

krnowak left a comment

krnowak Sep 29, 2023

t-lo Sep 29, 2023

t-lo Sep 29, 2023

krnowak Sep 29, 2023

chewi Sep 29, 2023

t-lo Sep 29, 2023

t-lo Sep 29, 2023

krnowak Sep 29, 2023

krnowak Sep 29, 2023

chewi Sep 29, 2023

t-lo Sep 29, 2023

krnowak Sep 29, 2023

krnowak Sep 29, 2023

krnowak Sep 29, 2023

t-lo Sep 29, 2023

t-lo commented Sep 29, 2023

t-lo commented Sep 29, 2023 •

edited

Loading

t-lo commented Sep 29, 2023

krnowak left a comment

t-lo commented Sep 30, 2023

		app-misc/ca-certificates::coreos # coreos ca-certificates doesn't support prefix
		sys-apps/baselayout::coreos # coreos baselayout depends on systemd

		@@ -0,0 +1 @@
		>=sys-devel/gcc-13 # FIXME: Cannot cross-compile gcc 13+ using older version

Flatcar SDK: add experimental prefix builds #1158

Flatcar SDK: add experimental prefix builds #1158

Conversation

t-lo commented Sep 20, 2023 • edited Loading

How to use

Testing done

Advanced usage

t-lo commented Sep 20, 2023

github-actions bot commented Sep 20, 2023 • edited Loading

dongsupark commented Sep 20, 2023

t-lo commented Sep 21, 2023

t-lo commented Sep 25, 2023

t-lo commented Sep 26, 2023

t-lo commented Sep 27, 2023

krnowak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t-lo commented Sep 29, 2023

t-lo commented Sep 29, 2023 • edited Loading

t-lo commented Sep 29, 2023

krnowak left a comment

Choose a reason for hiding this comment

t-lo commented Sep 30, 2023

t-lo commented Sep 20, 2023 •

edited

Loading

github-actions bot commented Sep 20, 2023 •

edited

Loading

t-lo commented Sep 29, 2023 •

edited

Loading