next: new release on 2023-11-06 (39.20231106.1.0)

[c4rt0]

First, verify that you meet all the prerequisites

Edit the issue title to include today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.1.0).

Pre-release

Promote next-devel changes to next

• Add the ok-to-promote label to the issue
• Review the promotion PR against the next branch on https://github.com/coreos/fedora-coreos-config
• Once CI has passed, merge it

Manual alternative

Sometimes you need to run the process manually like if you need to add an extra commit to change something in manifest.yaml. The steps for this are:

• git fetch upstream
• git checkout next
• git reset --hard upstream/next
• /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh next-devel
• Open PR against the next branch on https://github.com/coreos/fedora-coreos-config

Build

• Start a build job (select next, leave all other defaults). This will automatically run multi-arch builds.
• Post links to the jobs as a comment to this issue
  • x86_64
  • aarch64 (multi-arch build job)
  • ppc64le (multi-arch build job)
  • s390x (multi-arch build job)
• Wait for the jobs to finish and succeed
  • x86_64
  • aarch64
  • ppc64le
  • s390x

Sanity-check the build

Using the the build browser for the next stream:

• Verify that the parent commit and version match the previous next release (in the future, we'll want to integrate this check in the release job)
  • x86_64
  • aarch64
  • ppc64le
  • s390x
• Check kola extended upgrade runs to make sure they didn't fail
  • x86_64
  • aarch64
  • ppc64le
  • s390x
• Check kola AWS runs to make sure they didn't fail
  • x86_64
  • aarch64
• Check kola OpenStack runs to make sure they didn't fail
  • x86_64
  • aarch64
• Check kola Azure run to make sure it didn't fail
  • x86_64
• Check kola GCP runs to make sure they didn't fail
  • x86_64
  • aarch64

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

• Run the release job, filling in for parameters next and the new version ID
• Post a link to the job as a comment to this issue
• Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

• Wait for all releases that will be released simultaneously to reach this step in the process
• Go to the rollout workflow, click "Run workflow", and fill out the form

Rollout general guidelines

Risk	Day of the week	Rollout Start Time	Time allocation
risky	Tuesday	2PM UTC	72H
common	Tuesday	2PM UTC	48H
rapid	Tuesday	2PM UTC	24H

When setting a rollout start time ask "when would be the best time to react to
any errors or regressions from updates?". Commonly we select 2PM UTC so that the
rollout's start at 10am EST(±1 for daylight savings), but these can be fluid and
adjust after talking with the fedora-coreos IRC. Note, this is impacted by the
day of the week and holidays.

The later in the week the release gets held up due to unforeseen issues the more
likely the rollout time allocation will need to shrink or the release will need
to be deferred.

Manual alternative

• Make sure your fedora-coreos-stream-generator binary is up-to-date.

From a checkout of this repo:

• Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/next/releases.json  -output-file=streams/next.json -pretty-print

• Add a rollout. For example, for a 48-hour rollout starting at 10 AM ET the same day, run:

./rollout.py add next <version> "10 am ET today" 48

• Commit the changes and open a PR against the repo

• Verify that the expected OS versions appear in the PR on https://github.com/coreos/fedora-coreos-streams
• Post a link to the resulting PR as a comment to this issue
• Review and approve the PR, then wait for someone else to approve it also
• Once approved, merge it and verify that the sync-stream-metadata job syncs the contents to S3
• Verify the new version shows up on the download page
• Verify the incoming edges are showing up in the update graph.
  • x86_64
  • aarch64
  • ppc64le
  • s390x

Update graph manual check

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=next&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=next&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=ppc64le&stream=next&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=s390x&stream=next&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated.

Housekeeping

• If one doesn't already exist, open an issue in this repo for the next release in this stream. Use the approximate date of the release in the title.
• Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the rotation.

[c4rt0]

Hey @marmijo - I think I'm a little bit confused here. As I can see the FCOS release rotation doc changed since morning. If that's ok with you (and since it's my turn to run releases) I will proceed with all of the tasks starting tomorrow morning, considering Dusty's comment related to testing.

[marmijo]

@c4rt0 no worries, we are just going back to our normal schedule now that F39 will be released. The release schedule is now going to shift by a week so I changed the schedule to reflect that we're doing triple releases this week instead of next week. I'll continue for now and you're welcome to take over tomorrow. 😄

[c4rt0]

SGTM, thanks for clarifying.

[marmijo]

Promotion PR: coreos/fedora-coreos-config#2717

[marmijo]

Build jobs:

• x86_64: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build/1979/
• aarch64: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build-arch/2942/
• ppc64le: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build-arch/2943/
• s390x: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build-arch/2944/

[marmijo]

Release job: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/release/980/

[dustymabe]

Setting the AMI launch permissions failed in the release job. I cleaned up a few to get us under quota and then set it for ap-south-2, eu-central-2 and ap-southeast-3 AMIs:

[dustymabe@media Downloads]$ export AWS_DEFAULT_REGION=eu-central-2
[dustymabe@media Downloads]$ images=$(jq -r '.Images[] | select(.Public==false) | select(.Name | match("fedora-coreos-39")) | .ImageId' images.json)
[dustymabe@media Downloads]$ echo $images
ami-07443ed4c1205dd94 ami-00cee532c19d3c92a
[dustymabe@media Downloads]$ for image in $images; do
    set -x
    aws ec2 modify-image-attribute --launch-permission "Add=[{Group=all}]" --image-id $image
    set +x      
done
+ aws ec2 modify-image-attribute --launch-permission 'Add=[{Group=all}]' --image-id ami-07443ed4c1205dd94
+ set +x
+ aws ec2 modify-image-attribute --launch-permission 'Add=[{Group=all}]' --image-id ami-00cee532c19d3c92a
+ set +x

[dustymabe@media Downloads]$ export AWS_DEFAULT_REGION=ap-south-2
[dustymabe@media Downloads]$ aws ec2 describe-images --owners 125523088429 --output=json > images.json
[dustymabe@media Downloads]$ images=$(jq -r '.Images[] | select(.Public==false) | select(.Name | match("fedora-coreos-39")) | .ImageId' images.json)
[dustymabe@media Downloads]$ echo $images
ami-0adb6d6a6b5bf1fa1 ami-0c1267602f50eb0f7
[dustymabe@media Downloads]$ for image in $images; do
    set -x
    aws ec2 modify-image-attribute --launch-permission "Add=[{Group=all}]" --image-id $image
    set +x      
done
+ aws ec2 modify-image-attribute --launch-permission 'Add=[{Group=all}]' --image-id ami-0adb6d6a6b5bf1fa1
+ set +x
+ aws ec2 modify-image-attribute --launch-permission 'Add=[{Group=all}]' --image-id ami-0c1267602f50eb0f7
+ set +x


[dustymabe@media Downloads]$ export AWS_DEFAULT_REGION=ap-southeast-3
[dustymabe@media Downloads]$ aws ec2 describe-images --owners 125523088429 --output=json > images.json
[dustymabe@media Downloads]$ images=$(jq -r '.Images[] | select(.Public==false) | select(.Name | match("fedora-coreos-39")) | .ImageId' images.json)
[dustymabe@media Downloads]$ echo $images
ami-05711c6520da2ce7e ami-0b9172739df9b92da ami-0f989055a653a38f3 ami-05bcde91a073a2c9c
[dustymabe@media Downloads]$ for image in $images; do
    set -x
    aws ec2 modify-image-attribute --launch-permission "Add=[{Group=all}]" --image-id $image
    set +x      
done
+ aws ec2 modify-image-attribute --launch-permission 'Add=[{Group=all}]' --image-id ami-05711c6520da2ce7e
+ set +x
+ aws ec2 modify-image-attribute --launch-permission 'Add=[{Group=all}]' --image-id ami-0b9172739df9b92da
+ set +x
+ aws ec2 modify-image-attribute --launch-permission 'Add=[{Group=all}]' --image-id ami-0f989055a653a38f3
+ set +x
+ aws ec2 modify-image-attribute --launch-permission 'Add=[{Group=all}]' --image-id ami-05bcde91a073a2c9c
+ set +x

[marmijo]

Rollout PR: #807