testing: new release on 2020-05-14 (31.20200505.2.1)

[dustymabe]

First, verify that you meet all the prerequisites

Name this issue testing: new release on YYYY-MM-DD with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.2.0).

Pre-release

Promote testing-devel changes to testing

From the checkout for fedora-coreos-config (replace upstream below with
whichever remote name tracks coreos/):

• git fetch upstream
• git checkout testing
• git reset --hard upstream/testing
• /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh testing-devel
• Sanity check promotion with git show
• Open PR against the testing branch on https://github.com/coreos/fedora-coreos-config
• Post a link to the PR as a comment to this issue
• Ideally have at least one other person check it and approve
• Once CI has passed, merge it

Build

• Start a pipeline build (select testing, leave all other defaults)
• Post a link to the job as a comment to this issue
• Wait for the job to finish

Sanity-check the build

Using the the build browser for the testing stream:

• Verify that the parent commit and version match the previous testing release (in the future, we'll want to integrate this check in the release job)
• Check kola AWS run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

• Run the release job, filling in for parameters testing and the new version ID
• Post a link to the job as a comment to this issue
• Wait for job to finish
• Verify that the OSTree commit and its signature are present and valid by booting a VM at the previous release (e.g. cosa run -d /path/to/previous.qcow2) and verifying that rpm-ostree upgrade works and rpm-ostree status shows a valid signature.

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

• Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/testing/releases.json  -output-file=streams/testing.json -pretty-print

• Update the updates metadata, editing updates/testing.json:
  • Find the last-known-good release (whose rollout has a start_percentage of 1.0) and set its version to the most recent completed rollout
  • Delete releases with completed rollouts
  • Add a new rollout:
    • Set version field to the new version
    • Set start_epoch field to a future timestamp for the rollout start (e.g. date -d '2019/09/10 14:30UTC' +%s)
    • Set start_percentage field to 0.0
    • Set duration_minutes field to a reasonable rollout window (e.g. 2880 for 48h)
  • Update the last-modified field to current time (e.g. date -u +%Y-%m-%dT%H:%M:%SZ)

A reviewer can validate the start_epoch time by running date -u -d @<EPOCH>. An example of encoding and decoding in one step: date -d '2019/09/10 14:30UTC' +%s | xargs -I{} date -u -d @{}.

• Commit the changes and open a PR against the repo.
• Post a link to the PR as a comment to this issue
• Wait for the PR to be approved.
• Once approved, merge it and push the content to S3:

aws s3 sync --acl public-read --cache-control 'max-age=60' --exclude '*' --include 'streams/*' --include 'updates/*' . s3://fcos-builds

• Verify the new version shows up on the download page
• Verify the incoming edges are showing up in the update graph:

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=testing&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated and a syncer will push the updated metadata to S3.

Open an issue for the next release

• Open an issue in this repo with the approximate date in the title of the next release in this stream.
  • Add the jira label to the ticket

[dustymabe]

this is a quickfix release for coreos/fedora-coreos-tracker#479 so we are skipping some steps

[dustymabe]

coreos/fedora-coreos-config#397

[dustymabe]

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/12548/

[dustymabe]

kola AWS run passed: https://jenkins-fedora-coreos.apps.ci.centos.org/blue/organizations/jenkins/fedora-coreos%2Ffedora-coreos-fedora-coreos-pipeline-kola-aws/detail/fedora-coreos-fedora-coreos-pipeline-kola-aws/423/pipeline/35

[dustymabe]

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/510/

[dustymabe]

I hit an issue with the ostree import where the commit was imported but the ref wasn't updated yet. I can fix this once the existing imports from pungi (fedora releng) finish.

I did verify that an upgrade seems to work successfully and the commit is signed.

[dustymabe]

#102

[dustymabe]

I hit an issue with the ostree import where the commit was imported but the ref wasn't updated yet. I can fix this once the existing imports from pungi (fedora releng) finish.

I updated the ref file manually. The next ostree summary -u that runs (presumably at the end of the releng imports from pungi) should pick up the change.

sh-5.0$ echo 'a047a3c97511d242bab1a9c136debcf1cf47981bea9deb7affa38fb202dae0b3' > refs/heads/fedora/x86_64/coreos/testing
sh-5.0$ ls -l refs/heads/fedora/x86_64/coreos/testing
-rw-r--r--. 1 1000340000 263 65 May 14 03:08 refs/heads/fedora/x86_64/coreos/testing
sh-5.0$ cat refs/heads/fedora/x86_64/coreos/testing
a047a3c97511d242bab1a9c136debcf1cf47981bea9deb7affa38fb202dae0b3

[dustymabe]

The next ostree summary -u that runs (presumably at the end of the releng imports from pungi) should pick up the change.

yep. all looks good now:

[core@localhost ~]$ sudo rpm-ostree upgrade
⠉ Scanning metadata: 2510 
Scanning metadata: 2510... done
⠁  
Staging deployment... done
Upgraded:
  conmon 2:2.0.15-1.fc31 -> 2:2.0.16-2.fc31
  containernetworking-plugins 0.8.5-1.fc31 -> 0.8.6-1.fc31
  podman 2:1.9.0-1.fc31 -> 2:1.9.2-1.fc31
  podman-plugins 2:1.9.0-1.fc31 -> 2:1.9.2-1.fc31
Run "systemctl reboot" to start a reboot
[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
  ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 31.20200505.2.1 (2020-05-13T23:33:50Z)
                    Commit: a047a3c97511d242bab1a9c136debcf1cf47981bea9deb7affa38fb202dae0b3
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
                      Diff: 4 upgraded

* ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 31.20200505.2.0 (2020-05-06T16:42:59Z)
                    Commit: a4395eae3e1844d806a79b5cf51d44e60c96c7ab261a715f5d7fd89584c6963b
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

  ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 31.20200407.2.2 (2020-04-08T14:35:24Z)
                    Commit: b82607fd924510e0c828263d7cd38f9622420fcee8a345ed495570a36ff198fa
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

[dustymabe]

ok pushed to s3 - waiting for the download page to update

[dustymabe]

download page is updated and the graph shows the new edge!

[dustymabe]

ok this is rolling out now. The jira ticket for the next testing release already exists so no need to create a new one.