Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[4.11] Backports for RHCOS CI (--transient option & test entrypoint) #2963

Merged
merged 15 commits into from
Jul 7, 2022

Commits on Jul 6, 2022

  1. init: Add --transient flag for ephemeral builds

    We go to a lot of effort to create various caches of builds, and this
    is very useful for local incremental development to avoid re-downloading
    RPMs and rewriting data.
    
    It is useless for CI builds that discard everything when they're done.
    Add a `cosa init --transient` flag that for now just disables `fsync()`
    on `tmp/repo`.
    
    There's more we can do here in the future, for example we need to
    propagate this too into `cache/repo-build` and `cache/pkgcache-repo`,
    and actually in general we don't need the pkgcache repo at all here.
    cgwalters authored and travier committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    736d1f3 View commit details
    Browse the repository at this point in the history
  2. init: Add a success message

    To make it clearer that things worked.
    cgwalters authored and travier committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    6d9956c View commit details
    Browse the repository at this point in the history
  3. build: Use cache=unsafe for cache.qcow2 and transient builds

    Followup to the addition of `--transient` for CI flows.  If
    we had coreos/rpm-ostree#3719
    this would mostly be unnecessary, but today in CI and prod builds,
    ostree is invoking `fsync()` inside the supermin VM, and then
    qemu in the container is going to `fsync()` all changes down to
    the host system (overlay)fs.
    
    For transient flows, switch to `cache=unsafe` so we stop doing
    that which should greatly help speed.
    cgwalters authored and travier committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    d960e9d View commit details
    Browse the repository at this point in the history
  4. ci/prow-rhcos.sh: use --transient

    So we test it, and so we also gain the speed benefits.
    cgwalters authored and travier committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    8b8f99c View commit details
    Browse the repository at this point in the history
  5. ci/prow-rhcos: Adjust for recent changes in openshift/os

    Hopefully will fix the `ci/prow/rhcos` context here.
    cgwalters authored and travier committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    3c54104 View commit details
    Browse the repository at this point in the history
  6. ci/prow-rhcos: Use COSA specific test entry point

    Use a COSA specifc test entry point to focus on tests relevant for COSA.
    travier committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    37b5fc0 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2022

  1. Insert generated image.json into the ostree commit [retry]

    This is part of coreos/fedora-coreos-tracker#1151
    
    Our generated disk images are largely just a "shell" around the egg
    of an ostree commit.  There is almost nothing that lives
    in the disk image that isn't in the commit.
    
    (This is especially true now that a preparatory commit previous to
     this moved the *content* of our static `grub.cfg` into `image.json`)
    
    In the original coreos-assembler design I'd tried to cleanly
    separate builds of the ostree from disk image builds, but also
    support linking them together (with matching version numbers, etc.)
    The separate `image.yaml` was part of this.  This...mostly worked.
    
    This change furthers that separation by having image builds input from
    *just the ostree commit*.  Crucially we would no longer need the config
    git repository to perform an image build.
    
    And this in turn unlocks truly better separating ostree builds from
    disk image builds in the pipeline *and* supporting
    downstream tooling generating disk images from custom containers.
    
    One neat thing here is we will finally fix a longstanding issue
    where coreos-assembler fails when just the `image.yaml` changes:
    
    Closes: coreos#972
    cgwalters authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    3422e6d View commit details
    Browse the repository at this point in the history
  2. build: Handle case of previous build with no image.json in ostree

    For now, we need to handle this.  But I think we can switch
    back to hard requiring it later, which will happen naturally
    when we rework the build system to more cleanly separate
    ostree container builds and disk image builds.
    
    Closes: coreos#2885
    cgwalters authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    5e570db View commit details
    Browse the repository at this point in the history
  3. cmd-fetch: pass --force-nocache

    If the cache was nuked, we should re-download everything even if
    nothing changed since the last build. Otherwise, it's impossible to do a
    `cosa build --force` afterwards.
    jlebon authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    e4e014d View commit details
    Browse the repository at this point in the history
  4. cmdlib: factor out runnning supermin with cache

    Prep for future patch.
    jlebon authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    9399aaf View commit details
    Browse the repository at this point in the history
  5. cmdlib: drop impl_rpmostree_compose

    This inlines back `impl_rpmostree_compose` in the two places where it's
    called.
    
    Prep for future patch.
    jlebon authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    4283768 View commit details
    Browse the repository at this point in the history
  6. cmd-build: globalize path to compose.json

    Prep for future patch.
    jlebon authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    83e9722 View commit details
    Browse the repository at this point in the history
  7. cmd-build: move --write-composejson-to switch to cmdlib

    Now that the variable is declared publicly, we can just add it to the
    list of rpm-ostree args as part of the compose helper.
    
    This has no functional change in the `cosa build` path. In the
    `cosa fetch` path, this will add the switch but rpm-ostree will ignore
    it since we're not actually writing a commit.
    jlebon authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    db06b47 View commit details
    Browse the repository at this point in the history
  8. cmdlib: rename cosa-image-json ostree overlay

    All the other overlays have the prefix `overlay/`. Do that here too for
    consistency.
    jlebon authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    71a294c View commit details
    Browse the repository at this point in the history
  9. Don't pull-local over 9p during rpm-ostree compose

    Instead of having rpm-ostree effectively doing a `pull-local` under the
    hood over 9p, change things so we compose in a local repo, export the
    commit to an archive and only copy *that* over 9p.
    
    This should greatly help with pipelines hitting ENOMEM due to
    transferring many small files over 9p:
    
    openshift/os#594
    
    An initial approach exported to OCI archive instead, but encapsulating
    and unencapsulating are more expensive operations. Unfortunately, we
    still need to unpack into `tmp/repo` given that many follow-up commands
    expect the commit to be available in `tmp/repo`. If we can drop that
    assumption (and get rid of `tmp/repo` entirely), we could refactor
    things further so that `compose.sh` creates the final chunked OCI
    artifact upfront.
    
    In local testing, this adds about 30s to `cosa build`. We still compress
    just once, and we get hardlinks pulling out of the tarball.
    jlebon authored and travier committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    c36f1ac View commit details
    Browse the repository at this point in the history