next: new release on 2020-06-02 (32.20200601.1.1) #109

lucab · 2020-05-19T20:12:51Z

First, verify that you meet all the prerequisites

Name this issue next: new release on YYYY-MM-DD with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.2.0).

Pre-release

Promote next-devel changes to next

From the checkout for fedora-coreos-config (replace upstream below with
whichever remote name tracks coreos/):

git fetch upstream
git checkout next
git reset --hard upstream/next
/path/to/fedora-coreos-releng-automation/scripts/promote-config.sh next-devel
Sanity check promotion with git show
Open PR against the next branch on https://github.com/coreos/fedora-coreos-config
Post a link to the PR as a comment to this issue
Ideally have at least one other person check it and approve
Once CI has passed, merge it

Build

Start a pipeline build (select next, leave all other defaults)
Post a link to the job as a comment to this issue
Wait for the job to finish

Sanity-check the build

Using the the build browser for the next stream:

Verify that the parent commit and version match the previous next release (in the future, we'll want to integrate this check in the release job)
Check kola AWS run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

Run the release job, filling in for parameters next and the new version ID
Post a link to the job as a comment to this issue
Wait for job to finish
Verify that the OSTree commit and its signature are present and valid by booting a VM at the previous release (e.g. cosa run -d /path/to/previous.qcow2) and verifying that rpm-ostree upgrade works and rpm-ostree status shows a valid signature.

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/next/releases.json  -output-file=streams/next.json -pretty-print

Update the updates metadata, editing updates/next.json:
- Find the last-known-good release (whose rollout has a start_percentage of 1.0) and set its version to the most recent completed rollout
- Delete releases with completed rollouts
- Add a new rollout:
  - Set version field to the new version
  - Set start_epoch field to a future timestamp for the rollout start (e.g. date -d '2019/09/10 14:30UTC' +%s)
  - Set start_percentage field to 0.0
  - Set duration_minutes field to a reasonable rollout window (e.g. 2880 for 48h)
- Update the last-modified field to current time (e.g. date -u +%Y-%m-%dT%H:%M:%SZ)

A reviewer can validate the start_epoch time by running date -u -d @<EPOCH>. An example of encoding and decoding in one step: date -d '2019/09/10 14:30UTC' +%s | xargs -I{} date -u -d @{}.

Commit the changes and open a PR against the repo.
Post a link to the PR as a comment to this issue
Wait for the PR to be approved.
Once approved, merge it and push the content to S3:

aws s3 sync --acl public-read --cache-control 'max-age=60' --exclude '*' --include 'streams/*' --include 'updates/*' . s3://fcos-builds

Verify the new version shows up on the download page
Verify the incoming edges are showing up in the update graph:

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=next&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated and a syncer will push the updated metadata to S3.

Open an issue for the next release

Open an issue in this repo with the approximate date in the title of the next release in this stream.
- Add the jira label to the ticket

The text was updated successfully, but these errors were encountered:

dustymabe · 2020-06-02T18:22:47Z

coreos/fedora-coreos-config#446

Note that we are promoting from testing-devel and not next-devel because next and testing are just shadows of each other for now since we are promoting to f32.

dustymabe · 2020-06-02T19:56:09Z

found an issue (coreos/fedora-coreos-config#447) so opened a new sync PR: coreos/fedora-coreos-config#449

dustymabe · 2020-06-02T20:13:32Z

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/12937/

dustymabe · 2020-06-02T20:31:08Z

PR to stop previous rollouts: #110

dustymabe · 2020-06-03T02:45:14Z

one more issue (coreos/fedora-coreos-config#450). New sync PR coreos/fedora-coreos-config#452

new build: https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/12974/

dustymabe · 2020-06-03T13:56:55Z

The Kola AWS run did fail. This is currently a known issue documented in coreos/fedora-coreos-tracker#507

We will continue on to ship. As decided over in #108

dustymabe · 2020-06-03T14:04:51Z

merged #110 and verified there are no edges in the update graph.

dustymabe · 2020-06-03T14:11:58Z

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/543/

dustymabe · 2020-06-03T14:36:44Z

[core@localhost ~]$ sudo rpm-ostree status 
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/next
                   Version: 32.20200601.1.0 (2020-06-02T20:19:31Z)
                    Commit: 8cffe35be831fa2601d315002cb39fb22509a4e7d3db10e61f880523f69b3bf6
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora:fedora/x86_64/coreos/next
                   Version: 32.20200517.1.0 (2020-05-19T09:23:58Z)
                    Commit: 7c23c4735fb3c541586f0a4d3ca956ef93ef7d76f00a19bccf51460bafa7ee97
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

dustymabe · 2020-06-03T15:17:25Z

#111

dustymabe · 2020-06-03T17:53:43Z

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/543/

I accidentally ran this release job against the 32.20200601.1.0 build.. Will fix soon.

We accidentally ran the release job against the wrong release for next so let's roll this back briefly while we fix things up. coreos#109 (comment)

We accidentally ran the release job against the wrong release for next so let's roll this back briefly while we fix things up. #109 (comment)

dustymabe · 2020-06-03T21:36:41Z

ok I reset the ostree repo to point back to the old commit from the previous release (the ostree importer has some sanity checking code that would raise an error) and i'm running the release pipeline job against the correct version 32.20200601.1.1 now:

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/544/

dustymabe · 2020-06-03T21:55:40Z

[core@localhost ~]$ sudo rpm-ostree status 
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/next
                   Version: 32.20200601.1.1 (2020-06-03T02:26:26Z)
                    Commit: 08040bebbab87a3343a281f94bb68010df618eb6ce6ac3d4230d2595959b5da1
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora:fedora/x86_64/coreos/next
                   Version: 32.20200517.1.0 (2020-05-19T09:23:58Z)
                    Commit: 7c23c4735fb3c541586f0a4d3ca956ef93ef7d76f00a19bccf51460bafa7ee97
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

Now with 100% more correct version!

dustymabe · 2020-06-03T21:56:10Z

#114

lucab · 2020-06-04T09:24:35Z

Graph looks good this time.

dustymabe · 2020-06-04T14:27:02Z

The rollout started 30 minutes ago. It's the same content as the testing stream so I don't anticipate any issues.

dustymabe · 2020-06-04T15:06:49Z

issue for next time: #118

lucab added the kind/release label May 19, 2020

dustymabe changed the title ~~next: new release~~ next: new release on 2020-06-02 May 19, 2020

dustymabe self-assigned this May 19, 2020

dustymabe added the jira label May 19, 2020

dustymabe changed the title ~~next: new release on 2020-06-02~~ next: new release on 2020-06-02 (32.20200601.1.0) Jun 2, 2020

dustymabe changed the title ~~next: new release on 2020-06-02 (32.20200601.1.0)~~ next: new release on 2020-06-02 (32.20200601.1.1) Jun 3, 2020

dustymabe mentioned this issue Jun 3, 2020

updates/next: drop 32.20200601.1.1 until real release #112

Merged

dustymabe mentioned this issue Jun 3, 2020

streams/next: rollback stream metadata to 32.20200517.1.0 briefly #113

Merged

dustymabe added a commit that referenced this issue Jun 3, 2020

streams/next: rollback stream metadata to 32.20200517.1.0 briefly

fc30a4b

We accidentally ran the release job against the wrong release for next so let's roll this back briefly while we fix things up. #109 (comment)

dustymabe closed this as completed Jun 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

next: new release on 2020-06-02 (32.20200601.1.1) #109

next: new release on 2020-06-02 (32.20200601.1.1) #109

lucab commented May 19, 2020 •

edited by dustymabe

Loading

dustymabe commented Jun 2, 2020

dustymabe commented Jun 2, 2020

dustymabe commented Jun 2, 2020

dustymabe commented Jun 2, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

lucab commented Jun 4, 2020

dustymabe commented Jun 4, 2020

dustymabe commented Jun 4, 2020

next: new release on 2020-06-02 (32.20200601.1.1) #109

next: new release on 2020-06-02 (32.20200601.1.1) #109

Comments

lucab commented May 19, 2020 • edited by dustymabe Loading

Pre-release

Promote next-devel changes to next

Build

Sanity-check the build

⚠️ Release ⚠️

Run the release job

Refresh metadata (stream and updates)

Open an issue for the next release

dustymabe commented Jun 2, 2020

dustymabe commented Jun 2, 2020

dustymabe commented Jun 2, 2020

dustymabe commented Jun 2, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

dustymabe commented Jun 3, 2020

lucab commented Jun 4, 2020

dustymabe commented Jun 4, 2020

dustymabe commented Jun 4, 2020

lucab commented May 19, 2020 •

edited by dustymabe

Loading