Add new multi-arch build layout mode #580
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jun 28, 2019
jlebon
changed the title
|
OK, rebased! This is the final piece of the puzzle. |
ajeddeloh
approved these changes
Jul 10, 2019
| @@ -48,7 +48,12 @@ def cmd_upload_s3(args): | |||
| if args.build == 'latest': | |||
| args.build = builds.get_latest() | |||
| print(f"Targeting build: {args.build}") | |||
| s3_upload_build(args, builds.get_build_dir(args.build), args.build) | |||
| if builds.is_legacy(): | |||
There was a problem hiding this comment.
how long are we going to carry this?
There was a problem hiding this comment.
At least until the RHCOS bucket migrates to the new layout as well.
|
Thanks for the review! Will sanity test things one last time before merging in a bit. |
|
Tiny, tiny nit, and I feel bad for bringing it up so late: the plural of |
Add a new mode which allows cosa to manipulate multi-arch build layouts: ``` $ find builds builds builds/builds.json builds/30.1 builds/30.1/x86_64 builds/30.1/x86_64/coreos-assembler-config.tar.gz builds/30.1/x86_64/coreos-assembler-config-git.json builds/30.1/x86_64/fedora-coreos-30.1-qemu.qcow2 ... ``` A pipeline could e.g. dispatch builds for each architecture on different nodes, then group them back into a single workdir and have it manipulated by e.g. `buildupload` seamlessly. This new layout also matches the bucket layout for FCOS (see coreos/fedora-coreos-tracker#189). The basic idea is to add a `schema-version` to `builds.json` and denote the legacy behaviour as "pre-1.0.0", while `1.0.0` contains a different schema: each element in the `builds` array is now an object, which has an `id`, and a list of `archs` for which that build has been built: ``` $ cat builds/builds.json { "schema-version": "1.0.0", "builds": [ { "id": "30.1", "archs": [ "x86_64" ] } ], "timestamp": "2019-06-28T20:50:54Z" } ``` We retain backwards-compatibility by simply checking the schema version. Right now, only new workdirs will have this layout. Pipelines which use `buildprep` will fetch `builds.json` as is and key off of its contents to determine the bucket layout as well. We can write new code in the future to convert previously single-arch buckets into the new layout to then enable multi-arch.
Thanks, updated! Final sanity checks performed; merging. 🚀 |
arithx
added a commit
to arithx/fedora-coreos-releng-automation
that referenced
this pull request
Jul 11, 2019
In coreos/coreos-assembler#580 the `archs` key of builds.json was updated to be `arches`.
jlebon
added a commit
to coreos/fedora-coreos-browser
that referenced
this pull request
Jul 12, 2019
As described in coreos/coreos-assembler#580. The word "support" is used loosely here. Properly supporting and presenting multi-arch info would require reworking the UI with e.g. a drop-down or selectors for choosing which arch's artifacts to download. For now, the layout doesn't change: in the non-legacy case, we simply show the artifacts related to the first arch (which right now is the only arch: x86_64).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a new mode which allows cosa to manipulate multi-arch build layouts:
A pipeline could e.g. dispatch builds for each architecture on different
nodes, then group them back into a single workdir and have it
manipulated by e.g.
builduploadseamlessly.This new layout also matches the bucket layout for FCOS (see
coreos/fedora-coreos-tracker#189).
The basic idea is to add a
schema-versiontobuilds.jsonand denotethe legacy behaviour as "pre-1.0.0", while
1.0.0contains a differentschema: each element in the
buildsarray is now an object, which hasan
id, and a list ofarchsfor which that build has been built:We retain backwards-compatibility by simply checking the schema version.
Right now, only new workdirs will have this layout. Pipelines which use
buildprepwill fetchbuilds.jsonas is and key off of its contentsto determine the bucket layout as well. We can write new code in the
future to convert previously single-arch buckets into the new layout to
then enable multi-arch.