stable: new release on 2020-06-16 (32.20200601.3.0)

[dustymabe]

First, verify that you meet all the prerequisites

Name this issue stable: new release on YYYY-MM-DD with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.3.0).

Pre-release

Promote testing changes to stable

From the checkout for fedora-coreos-config (replace upstream below with
whichever remote name tracks coreos/):

• git fetch upstream
• git checkout stable
• git reset --hard upstream/stable
• /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh testing
• Sanity check promotion with git show
• Open PR against the stable branch on https://github.com/coreos/fedora-coreos-config
• Post a link to the PR as a comment to this issue
• Ideally have at least one other person check it and approve
• Once CI has passed, merge it

Build

• Start a pipeline build (select stable, leave all other defaults)
• Post a link to the job as a comment to this issue
• Wait for the job to finish

Sanity-check the build

Using the the build browser for the stable stream:

• Verify that the parent commit and version match the previous stable release (in the future, we'll want to integrate this check in the release job)
• Check kola AWS run to make sure it didn't fail
• Check kola GCP run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

• Run the release job, filling in for parameters stable and the new version ID
• Post a link to the job as a comment to this issue
• Wait for job to finish
• Verify that the OSTree commit and its signature are present and valid by booting a VM at the previous release (e.g. cosa run -d /path/to/previous.qcow2) and verifying that rpm-ostree upgrade works and rpm-ostree status shows a valid signature.

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

• Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/stable/releases.json  -output-file=streams/stable.json -pretty-print

• Update the updates metadata, editing updates/stable.json:
  • Find the last-known-good release (whose rollout has a start_percentage of 1.0) and set its version to the most recent completed rollout
  • Delete releases with completed rollouts
  • Add a new rollout:
    • Set version field to the new version
    • Set start_epoch field to a future timestamp for the rollout start (e.g. date -d '2019/09/10 14:30UTC' +%s)
    • Set start_percentage field to 0.0
    • Set duration_minutes field to a reasonable rollout window (e.g. 2880 for 48h)
  • Update the last-modified field to current time (e.g. date -u +%Y-%m-%dT%H:%M:%SZ)

A reviewer can validate the start_epoch time by running date -u -d @<EPOCH>. An example of encoding and decoding in one step: date -d '2019/09/10 14:30UTC' +%s | xargs -I{} date -u -d @{}.

• Commit the changes and open a PR against the repo.
• Post a link to the PR as a comment to this issue
• Wait for the PR to be approved.
• Once approved, merge it and verify that the sync-stream-metadata job syncs the contents to S3
• Verify the new version shows up on the download page
• Verify the incoming edges are showing up in the update graph:

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=stable&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated.

Open an issue for the next release

• Open an issue in this repo with the approximate date in the title of the next release in this stream.
  • Add the jira label to the ticket

[sinnykumari]

config changes PR - coreos/fedora-coreos-config#470

[dustymabe]

before you run the release jobs can you open a PR like #110 to stop all rollouts?

[sinnykumari]

before you run the release jobs can you open a PR like #110 to stop all rollouts?

ack, will create the pr.

[sinnykumari]

pipeline build - https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/13155/

[sinnykumari]

AWS kola test run is still failing with the known issue coreos/fedora-coreos-tracker#507

[sinnykumari]

PR to stop previous rollouts #123

[dustymabe]

AWS kola test run is still failing with the known issue coreos/fedora-coreos-tracker#507

It is interesting to note I don't see the same failure in that test run as is mentioned in coreos/fedora-coreos-tracker#507. I do see

harness.go:823: Found kernel warning (drivers/xen/events/events_base.c:1111 unbind_from_irqhandler+0x4e/0x60) on machine i-0f81075a5867c50a5 console

but not

Found kernel warning (lib/percpu-refcount.c:160 percpu_ref_switch_to_atomic_rcu+0x12f/0x140) on machine i-01e1bae926c44107a console

None of these happen on the m5.large instance type so let's not block the release for now. I'll continue to dig.

[sinnykumari]

Release job - https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/555/

[sinnykumari]

AWS kola test run is still failing with the known issue coreos/fedora-coreos-tracker#507

My bad, I searched in the AWS kola test for "Your kernel does not support Block I/O weight or the cgroup is not mounted." which was second last line from log.

Do we want to add new error message into issue coreos/fedora-coreos-tracker#507 ?

[sinnykumari]

OSTree commit and signature looks good.

$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200601.3.0 (2020-06-16T08:52:21Z)
                    Commit: b51037798e93e5aae5123633fb596c80ddf30302b5110b0581900dbc5b2f0d24
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200517.3.0 (2020-06-01T17:15:29Z)
                    Commit: 967b7b8d624e6d10ff51c2e81ef198fae966c567ac2e9b479771c693d0987949
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

[sinnykumari]

rollout to 32.20200601.3.0 - #125

[sinnykumari]

incoming edges are looking good, all previous release index has upgrade path to latest one.

[sinnykumari]

rollout of 32.20200601.3.0 is scheduled to start at 2020-06-16 16:30:00 UTC for duration of 72 hours

[sinnykumari]

issue for next release - #126

[dustymabe]

rollout to 32.20200601.3.0 has been completed.