-
Notifications
You must be signed in to change notification settings - Fork 155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add overrides for multi-arch #374
Conversation
@jcajka, can you test this? |
(And yeah, we need to pick up discussions again about making it easier to add overrides so they're not mostly duplicated across 4 files.) |
Testing now. Looks good so far on ppc64le. |
s390x kind of too(I have too start issue about initscripts there, but that is other issue ;)) and aarch64 is ok too. Thanks for quick fix :). |
Actually, that is new failure on ppc64le build(fetch is ok).
|
Same for build on aarch64. I assume that might be related to the new packages, rpm-ostree? |
Hmm, that's odd. What version of As a sanity-check, do you get a successful compose if you only have the |
Checking it now.
With manifest lock it fails as noted:
|
Keeping just the crypto-policies in the lock file leads to the fallowing failure during build, fetch is ok:
Time to move to f32 base image? ;) |
OK, so the issue here is that this is pulling in a version of container-selinux that is much older than what's in the updates repo, and it doesn't have
What happens if you add: container-selinux:
evra: 2:2.124.0-3.fc31.noarch ? That should force libsolv to print out the conflict preventing it from picking the latest version.
Yup, we're planning to move there in two releases (three releases for stable). :) |
I can confirm that this is the fix here. On topic of f32, I can say that at least for the cosa images it works just fine on multiarch(not used here). |
9e76772
to
390751d
Compare
OK, I added a second commit to this PR:
|
(To clarify, that latter issue has nothing to do with multi-arch. It just happened to come up at the same time. :) ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
With coreos#355, the pool is now used for fetching lockfiles only. This however broke multi-arch builds because although there are no lockfiles, it's currently not possible to get a depsolve without at least locking crypto-policies: coreos#335 (comment) While we can't yet maintain base lockfiles for other arches, we can at least start maintaining overrides. This will also in the process fix builds there.
Latest version has a hard dep on selinux-policy-minimal, which wants python: https://src.fedoraproject.org/rpms/container-selinux/pull-request/3
390751d
to
c9d69bb
Compare
Seems sane to me though we clearly have two problems:
|
I think that also boils down if and how much COS wants to integrate in to the current release processes for packages.
IMO locks/pin/overrides should be at NVR level not NVRA(as currently) for most cases and packages as I don't believe there will be arch dependent overrides that will be that common(as they shouldn't be, to discourage any short term hacky workarounds) and ideally managed via some automated tool(-ing)(IMHO, as packager, bodhi/koji could be used/repurposed for this, but really rel-eng, infra will be able to answer that). One of my current main motivations is that I really do not have cycles to manage the multi-arch lock files on top of dealing with breakages and general fixes to the forming multi-arch. |
Right. And actually, one thing @dustymabe and I were discussing recently was that we should try to get FCOS CI directly into Bodhi so that packages don't get pushed to the updates repo if CI fails.
As mentioned in #355 (comment), base lockfiles should not be manually maintained. Only overrides. See e.g. coreos/fedora-coreos-pipeline#229, which we just merged to (re-)automate this again. We could make this blindly use NEVR instead of NEVRA, but the whole point of lockfiles is to get reproducible builds which passed validation. Using NEVR to lock multi-arch builds without having actually sanity-checked it is way less meaningful. So I think we just need to sort out coreos/fedora-coreos-tracker#262 and get proper multi-arch builds orchestration. |
Not sure I follow; !x86_64 isn't locked today; in the interim it'd help them test the exact state that matches the main build. |
It's not the exact state though. It's an almost completely different set of RPMs (and doesn't capture all the non-x86_64 arch-specific packages like If it helps, we can work towards this. But I think we should instead spend our energy fleshing out and working on coreos/fedora-coreos-tracker#262 instead. |
I agree for overrides though we probably want an NEVR lockfile to make it easier to maintain. Something like |
How so? Aren't we using koji to get the same source built across all the arches?
Sure, I doubt those are problematic very often though (and when they are it's something the arch team is in a position to debug and fix).
Yes we definitely need that. Though as you know I am increasingly convinced the lockfiles should manage repos: coreos/rpm-ostree#1858 (review) |
Yeah, but binary files themselves are different per architecture. So we're locking inputs we haven't technically tested, let alone composed.
The design though is strongly geared towards having the full OS locked. Not locking those means it wouldn't compose with I know these points might seem like nitpicking, and I don't disagree there's still some value to doing it anyway, but the original idea of lockfiles was locking the OS to a known green state and only allowing validated bumps at all steps. I'm wary of doing lots of work here (both for us and the arch team) that doesn't end up aligning with where we want to go for coreos/fedora-coreos-tracker#262. |
Let's start maintaining overrides for all arches in prep for promoting testing-devel to f32 (which already has multi-arc overrides -- see coreos#374). Stay tuned for a more generic `manifest-lock.overrides.yaml` file to help dedupe this stuff.
Let's start maintaining overrides for all arches in prep for promoting testing-devel to f32 (which already has multi-arc overrides -- see coreos#374). Stay tuned for a more generic `manifest-lock.overrides.yaml` file to help dedupe this stuff.
Let's start maintaining overrides for all arches in prep for promoting testing-devel to f32 (which already has multi-arc overrides -- see coreos#374). Stay tuned for a more generic `manifest-lock.overrides.yaml` file to help dedupe this stuff.
Let's start maintaining overrides for all arches in prep for promoting testing-devel to f32 (which already has multi-arc overrides -- see #374). Stay tuned for a more generic `manifest-lock.overrides.yaml` file to help dedupe this stuff.
With #355, the pool
is now used for fetching lockfiles only. This however broke multi-arch
builds because although there are no lockfiles, it's currently not
possible to get a depsolve without at least locking
crypto-policies
:#335 (comment)
While we can't yet maintain base lockfiles for other arches, we can at
least start maintaining overrides. This will also in the process fix
builds there.