-
Notifications
You must be signed in to change notification settings - Fork 165
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[4.11] Backports for RHCOS CI (--transient option & test entrypoint) #2963
Commits on Jul 6, 2022
-
init: Add
--transient
flag for ephemeral buildsWe go to a lot of effort to create various caches of builds, and this is very useful for local incremental development to avoid re-downloading RPMs and rewriting data. It is useless for CI builds that discard everything when they're done. Add a `cosa init --transient` flag that for now just disables `fsync()` on `tmp/repo`. There's more we can do here in the future, for example we need to propagate this too into `cache/repo-build` and `cache/pkgcache-repo`, and actually in general we don't need the pkgcache repo at all here.
Configuration menu - View commit details
-
Copy full SHA for 736d1f3 - Browse repository at this point
Copy the full SHA 736d1f3View commit details -
To make it clearer that things worked.
Configuration menu - View commit details
-
Copy full SHA for 6d9956c - Browse repository at this point
Copy the full SHA 6d9956cView commit details -
build: Use
cache=unsafe
forcache.qcow2
and transient buildsFollowup to the addition of `--transient` for CI flows. If we had coreos/rpm-ostree#3719 this would mostly be unnecessary, but today in CI and prod builds, ostree is invoking `fsync()` inside the supermin VM, and then qemu in the container is going to `fsync()` all changes down to the host system (overlay)fs. For transient flows, switch to `cache=unsafe` so we stop doing that which should greatly help speed.
Configuration menu - View commit details
-
Copy full SHA for d960e9d - Browse repository at this point
Copy the full SHA d960e9dView commit details -
ci/prow-rhcos.sh: use
--transient
So we test it, and so we also gain the speed benefits.
Configuration menu - View commit details
-
Copy full SHA for 8b8f99c - Browse repository at this point
Copy the full SHA 8b8f99cView commit details -
ci/prow-rhcos: Adjust for recent changes in openshift/os
Hopefully will fix the `ci/prow/rhcos` context here.
Configuration menu - View commit details
-
Copy full SHA for 3c54104 - Browse repository at this point
Copy the full SHA 3c54104View commit details -
ci/prow-rhcos: Use COSA specific test entry point
Use a COSA specifc test entry point to focus on tests relevant for COSA.
Configuration menu - View commit details
-
Copy full SHA for 37b5fc0 - Browse repository at this point
Copy the full SHA 37b5fc0View commit details
Commits on Jul 7, 2022
-
Insert generated image.json into the ostree commit [retry]
This is part of coreos/fedora-coreos-tracker#1151 Our generated disk images are largely just a "shell" around the egg of an ostree commit. There is almost nothing that lives in the disk image that isn't in the commit. (This is especially true now that a preparatory commit previous to this moved the *content* of our static `grub.cfg` into `image.json`) In the original coreos-assembler design I'd tried to cleanly separate builds of the ostree from disk image builds, but also support linking them together (with matching version numbers, etc.) The separate `image.yaml` was part of this. This...mostly worked. This change furthers that separation by having image builds input from *just the ostree commit*. Crucially we would no longer need the config git repository to perform an image build. And this in turn unlocks truly better separating ostree builds from disk image builds in the pipeline *and* supporting downstream tooling generating disk images from custom containers. One neat thing here is we will finally fix a longstanding issue where coreos-assembler fails when just the `image.yaml` changes: Closes: coreos#972
Configuration menu - View commit details
-
Copy full SHA for 3422e6d - Browse repository at this point
Copy the full SHA 3422e6dView commit details -
build: Handle case of previous build with no image.json in ostree
For now, we need to handle this. But I think we can switch back to hard requiring it later, which will happen naturally when we rework the build system to more cleanly separate ostree container builds and disk image builds. Closes: coreos#2885
Configuration menu - View commit details
-
Copy full SHA for 5e570db - Browse repository at this point
Copy the full SHA 5e570dbView commit details -
cmd-fetch: pass --force-nocache
If the cache was nuked, we should re-download everything even if nothing changed since the last build. Otherwise, it's impossible to do a `cosa build --force` afterwards.
Configuration menu - View commit details
-
Copy full SHA for e4e014d - Browse repository at this point
Copy the full SHA e4e014dView commit details -
cmdlib: factor out runnning supermin with cache
Prep for future patch.
Configuration menu - View commit details
-
Copy full SHA for 9399aaf - Browse repository at this point
Copy the full SHA 9399aafView commit details -
cmdlib: drop
impl_rpmostree_compose
This inlines back `impl_rpmostree_compose` in the two places where it's called. Prep for future patch.
Configuration menu - View commit details
-
Copy full SHA for 4283768 - Browse repository at this point
Copy the full SHA 4283768View commit details -
cmd-build: globalize path to
compose.json
Prep for future patch.
Configuration menu - View commit details
-
Copy full SHA for 83e9722 - Browse repository at this point
Copy the full SHA 83e9722View commit details -
cmd-build: move
--write-composejson-to
switch to cmdlibNow that the variable is declared publicly, we can just add it to the list of rpm-ostree args as part of the compose helper. This has no functional change in the `cosa build` path. In the `cosa fetch` path, this will add the switch but rpm-ostree will ignore it since we're not actually writing a commit.
Configuration menu - View commit details
-
Copy full SHA for db06b47 - Browse repository at this point
Copy the full SHA db06b47View commit details -
cmdlib: rename
cosa-image-json
ostree overlayAll the other overlays have the prefix `overlay/`. Do that here too for consistency.
Configuration menu - View commit details
-
Copy full SHA for 71a294c - Browse repository at this point
Copy the full SHA 71a294cView commit details -
Don't
pull-local
over 9p during rpm-ostree composeInstead of having rpm-ostree effectively doing a `pull-local` under the hood over 9p, change things so we compose in a local repo, export the commit to an archive and only copy *that* over 9p. This should greatly help with pipelines hitting ENOMEM due to transferring many small files over 9p: openshift/os#594 An initial approach exported to OCI archive instead, but encapsulating and unencapsulating are more expensive operations. Unfortunately, we still need to unpack into `tmp/repo` given that many follow-up commands expect the commit to be available in `tmp/repo`. If we can drop that assumption (and get rid of `tmp/repo` entirely), we could refactor things further so that `compose.sh` creates the final chunked OCI artifact upfront. In local testing, this adds about 30s to `cosa build`. We still compress just once, and we get hardlinks pulling out of the tarball.
Configuration menu - View commit details
-
Copy full SHA for c36f1ac - Browse repository at this point
Copy the full SHA c36f1acView commit details