OCP 4.9 Release Notes Tracker

[jeana-redhat]

Please leave comments here for anything that should be highlighted in the 4.9 release notes. If possible, provide a link to the Jira or BZ related to your item. Thank you!

Update 21 September

A note about known issues
All Bugzilla items that need to be documented as known issues in the release notes should have Doc Type = Known Issue and the Doc Text field populated accordingly. Before adding a known issue to this tracker:

1. Ensure the Doc Type and Doc Text fields are filled in correctly.
2. See if your issue shows up in the query we will use to populate the release notes.
3. If the fields are set correctly, and the issue is not in the query, add it here.

Resources

• OpenShift Container Platform 4.9 Release Notes Jira epic with timeline
• Feature GA Status Tracker

Key to reactions

👀 ACK
👍 Done
👎 Not going in 4.9 rel notes

[lbarbeevargas]

The Metering Operator was deprecated in 4.6. If it is still scheduled to be removed in 4.9:

• Add a release note in the "Removed features" section that the Metering Operator is removed.
• Update the 4.9 Deprecated and removed features tracker table for the Metering Operator.

OSDOCS-2249 tracks this release note.

[damemi]

Descheduler Operator v1beta1 API has been removed for v1 openshift/cluster-kube-descheduler-operator#199

[dulek]

OpenShift on OpenStack: In order to support LoadBalancer Services using OpenStack Octavia with OVN provider, the security group rules allowing NodePort traffic to master and worker nodes are now changed to open 0.0.0.0/0 and not just the cluster CIDR. This is because OVN loadbalancers are preserving the original source IP of the traffic, so for LoadBalancer services it can be anything. This wasn't required to support Amphora Octavia provider as Amphora loadbalancers change source IP to the IP of the LB itself which is guaranteed to be in the cluster CIDR.

Maybe a more detailed explanation: openshift/installer#5052 (comment)

[mandre]

OpenShift on OpenStack: The openstack cloud provider LoadBalancer configuration now defaults to 'use-octavia=True', unless deploying with Kuryr, in which case 'use-octavia' is set to false.

Context: openshift/installer#5047

[bgilbert]

Nodes installed with coreos-installer previously retained the installation Ignition config in /boot/ignition/config.ign. Starting with the OpenShift 4.9 install image, that file is removed when the node is provisioned. This change currently does not affect clusters that were originally installed on previous OpenShift versions, and are thus using an older bootimage.

[mikemckiernan]

MetalLB and the MetalLB Operator for a platform-native load balancer implementation on bare metal: #35705

[SNiemann15]

PR for IBM Z and IBM Power Systems input #35828

[tmalove]

PR to support etcd to the list of control plane components.
#35923

[bparees]

we need loud and clear discussion of all the k8s beta apis that are being removed in this release. Just putting them in the table probably isn't even enough (but i don't see them listed as removed in the current tables, either)

cc @mfojtik @deads2k @sttts

[bparees]

We are also going to require explicit manual admin acks regarding the removed apis before a 4.8 cluster can be upgraded to 4.9, so that should be called out as well.

@wallylewis you're driving the product docs on that, maybe you can help with the wording here as well.

[jeana-redhat]

@bparees thanks - that's super critical content. Just to clarify, are you referring to the table in the draft Rel Notes I just sent out, or to the one linked above? Not everything has gotten into the draft Rel Notes version of the table yet, but we will be working from the Google Sheet as a source for updates, so getting it correct in there would be a great (and hopefully easy) first step 🙂

[damemi]

Descheduler Operator: Users should update to the latest 4.8 operator before updating to the 4.9 operator to ensure proper conversion of existing Descheduler CRDs (openshift/cluster-kube-descheduler-operator#215)

[bparees]

@bparees thanks - that's super critical content. Just to clarify, are you referring to the table in the draft Rel Notes I just sent out, or to the one linked above?

I was looking at https://docs.openshift.com/container-platform/4.9/release_notes/ocp-4-9-release-notes.html

which has, for example, CRDs:

so at a minimum that 4.9 column will need to say "removed", but again i think we need more explicit wording than just hoping someone sees it in the table.

[tmalove]

Support the automatic rotation of etcd certificates.
https://issues.redhat.com/browse/OSDOCS-2348

[jeana-redhat]

@bparees ok, I think we are on the same page here. Typically, we also include some content below the table about things that have changed, so that would be a normal part of the Rel Notes process. We will definitely want to make sure that this is super obvious to folks.

[romfreiman]

@jeana-redhat we should add documentation about Single Node Openshift

[jiajliu]

About the upgrade to v4.9, afaik we have several new updates here. we need announce them in the release note for users.

1. we provide a new way performing a canary rollout update , related prs are GH34445: Doc on using machine config pool during update 2  #35420, Doc on using machine config pool during update #34445

2. About the v4.8 to v4.9 upgrade, we have added several gate/check to block the upgrade. One is about https://bugzilla.redhat.com/show_bug.cgi?id=1978376, which admin need ack the admin-gates to unblock the upgrade. Another is about the etcdbackup is needed to be done before upgrading to v4.9, more info refers to https://issues.redhat.com/browse/OTA-464.

cc @jianlinliu @shellyyang1989 if any missing part to be added for v4.9 release notes.

[xiaojiey]

About the Rhel8 scaleup for GCP platform, there is a known issue that when fips mode enabled, scaleup Rhel8 will fail due to fail to install packages from the default RHUI. Seen from https://bugzilla.redhat.com/show_bug.cgi?id=1997516.
cc: @gpei @pdhamdhe

[xiaojiey]

About the Rhel8 scaleup for GCP platform, there is a known issue that when fips mode enabled, scaleup Rhel8 will fail due to fail to install packages from the default RHUI. Seen from https://bugzilla.redhat.com/show_bug.cgi?id=1997516.
cc: @gpei @pdhamdhe

Create a tracker bug https://bugzilla.redhat.com/show_bug.cgi?id=2001464 for OCP to track the status.

[lihongan]

Routing new features/enhancements are missing, below should be covered in release notes:

• Make HAProxy timeout Variables Configurable (https://issues.redhat.com/browse/NE-412)
• Global Options to Enforce HTTP Strict Transport Security (HSTS) (https://issues.redhat.com/browse/NE-310)
• Allow Setting mTLS Through the Ingress Operator (https://issues.redhat.com/browse/NE-323)
• Support TLS 1.3 for OpenShift 4.x Ingress (https://issues.redhat.com/browse/NE-472)
• Enable http-ignore-probes and dontlognull as Defaults (https://issues.redhat.com/browse/NE-308)
• Ability to Customize HAProxy 2.x Error Page (https://issues.redhat.com/browse/NE-379)

cc @quarterpin

[jianlinliu]

2. About the v4.8 to v4.9 upgrade, we have added several `gate/check` to block the upgrade. One is about https://bugzilla.redhat.com/show_bug.cgi?id=1978376, which admin need ack the admin-gates to unblock the upgrade. 

Thanks for @jiajliu raising this. Some more info about the user acknowledge before v4.8 to v4.9 upgrade, it is being tracking in https://bugzilla.redhat.com/show_bug.cgi?id=1999092, which need to be merged in some 4.8.z version, once it is merged, 4.8.z to 4.9 upgrade will be blocked until user manually acknowledge it. Maybe we need to mention that in 4.9 release note somewhere though the 4.8.z bug is not merged yet.

[simonpasquier]

Monitoring:

• #1312 Support label to exclude namespaces from user-workload monitoring. Service/pod monitors and Prometheus rules living in a namespace labeled with openshift.io/user-monitoring: "false" will not be picked up by the user-workload monitoring stack.
• #1308 Add config option to specify remote write configuration for both platform and user-workload monitoring Prometheus.
• #1241 Add config option to disable Grafana deployment.
• #1278 Add config option to set the EnforcedTargetLimit parameter for the user-workload monitoring Prometheus.
• #1291 Drop high cardinality of cAdvisor metrics via kube-prometheus #1250.
• #1270 Show a message in the Degraded condition when the platform monitoring Prometheus runs without persistent storage.
• #1241 Add config option to configure additional Alertmanagers for platform and user-workload monitoring stacks.
• #1293 Add config option to disable the local Alertmanager.
• #1310 Adjust severity of alerts shipped by the cluster monitoring operator, fewer critical alerts with more accurate triggering condition.
• #1356 Add links to runbooks for all critical alerts.
• #1242 Add HighlyAvailableWorkloadIncorrectlySpread alert to detect when two instances of a highly available monitoring component are running on the same node and have persistent volumes attached.

[adambkaplan]

Builds:

• Build Volumes using Secrets and ConfigMaps: https://issues.redhat.com/browse/BUILD-257
• BuildConfigs with ImageChange triggers can be used in GitOps: https://issues.redhat.com/browse/BUILD-190

[gpei]

Installer:

• Support RHEL 8 Server worker/infra nodes - https://issues.redhat.com/browse/CORS-1650
• AWS: Support for China Regions - https://issues.redhat.com/browse/CORS-1591
• UPI install workflow for Azure Stack Hub support - https://issues.redhat.com/browse/CORS-1433

[jeana-redhat]

Adding from @cuppett via email: Look into linking to CRI-O 1.22 release notes

[yunjiang29]

@jeana-redhat known issue for installer, should be added in release note:
Bug 1997059 - Failed to create cluster in AWS us-east-1 region due to a local zone is used

cc @staebler @codyhoag

[jianzhangbjz]

1. OLM fails to update the operator immediately when switching the operator channels. Workaround: recreate the corresponding CatalogSource pod. Details: https://bugzilla.redhat.com/show_bug.cgi?id=2002276
2. opm render doesn't create the olm.bundle.object automatically, this will lead to no packagemanifest display on console or backend. The user has to add it manually for now.
   Bug: https://bugzilla.redhat.com/show_bug.cgi?id=2003894#c1
   Story: https://issues.redhat.com/browse/OLM-2331
   PR: render: include olm.bundle.object properties in rendered bundle images operator-framework/operator-registry#807
3. opm render fails to extract the images defined in the deployment to the relatedImages field, details: https://bugzilla.redhat.com/show_bug.cgi?id=2000379

cc: @kevinrizza

[jinyunma]

Known issue when enabling Tang disk encryption on upi-on-vsphere cluster, need to highlight in release notes.
Bug 1975701 - [vsphere][upi] Network is changed to dhcp configuration after second reboot when Tang disk encryption is enabled

Since there is workaround in the bug, it's better to add this workaround in doc Encrypting and mirroring disks during installation

cc @dustymabe if any more comments about this.

[zhaozhanqi]

For users if enabling Tang disk encryption on upi-on-vsphere and OVN kubernetes plugin with version 4.8.z want to upgrade 4.9. there is a known bug as below. Not user if there is user is using this kind of profile until now, I think we need to highlight this issue in case since it will cause worker become not ready when upgrading.
Bug 2006756 - Nodes go to NotReady when a both Tang and OVN enabled vSphere cluster is being upgraded to 4.9 from 4.8.12

cc @jcaamano @anuragthehatter

[xingxingxia]

1991448 for epic AUTH-13 may be needed to be documented under "Known issues" section of the release notes. CC @yaoli-redhat , @adambkaplan @alicerum , @s-urbaniak @slaskawi

[yunjiang29]

@jeana-redhat another known issue for AWS ap-northeast-3 region should be added in release note, please refer to @staebler 's comment , thanks

Bug 1996544 - AWS region ap-northeast-3 is missing in installer prompt

cc @codyhoag

[yapei]

https://docs.openshift.com/container-platform/4.9/release_notes/ocp-4-9-release-notes.html#ocp-4-9-assessing-node-logs-from-the-node-details-page

Node Details page is different from Node Logs page, Node Logs page is where user can view node logs, see attachment

[kasturinarra]

@jeana-redhat add GA in 4.9 column for oc CLI-Plugins ? Similar as what we have done for 4.8 ? Thanks !! CC @zhouying7780

[1] https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.html#ocp-4-8-technology-preview

[zhaozhanqi]

Pod may go into error status due to 'error killing pod' for 'openshift-sdn' plugin. there is a bug
https://bugzilla.redhat.com/show_bug.cgi?id=1997476
Bug 1997476 - pod in Error due to KillPodSandbox
There is workaround is recreate that pod can be fixed.

[zhaozhanqi]

Sometimes pods are stuck in 'ContainerCreating' state with error 'failed to configure pod interface: timed out waiting for OVS port binding' when creating many pods for OVN-kubernetes plugin. There is known issue
Bug 2005598 - [4.9]Failed to configure pod interface: timed out waiting for OVS port binding
cc @anuragthehatter @qiliRedHat

[huiran0826]

Known issue for ovn egressip ,after reboot egress nodes, lr-policy-list and snat rules related to egressip might not be correct.
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1995887. The workaround is restart ovn-kubemaster pods.

[weliang1]

Please help to add below known issue in the networking section for v4.9 release notes.
Multicast is broken when multicast traffic across nodes.
https://bugzilla.redhat.com/show_bug.cgi?id=2010374

The issue is fixed in v4.10 but not backport to v4.9 yet

[asood-rh]

@mikemckiernan Could we release note the issue https://bugzilla.redhat.com/show_bug.cgi?id=1987445? Though functionally there are no side affects but the behavior could lead to issues mentioned in comment #1

[anuragthehatter]

We might need to mention this prometheus perf metrics reporting issue in release notes starting 4.6 to 4.9 https://bugzilla.redhat.com/show_bug.cgi?id=2002868. Host/System OVS was introduced in 4.6. cc @dcbw

[jianzhangbjz]

This bug https://bugzilla.redhat.com/show_bug.cgi?id=2013384 had been merged to 4.9 payload: https://amd64.ocp.releases.ci.openshift.org/releasestream/4-stable/release/4.9.0-rc.8, so we should remove this part: Removing Operands when uninstalling an Operator using the web console

[yapei]

I think we may need to include following two console bugs in Known issues for awareness
Bug 2013132 - [OCP 4.9 + OCS 4.8.3] Overview tab is missing under Storage after successful deployment on UI
Bug 2013088 - The nodes page in the openshift console doesn't work. You just get a blank page

[vikram-redhat]

I am closing this issue now that the window to submit new issues has past. If a new issue needs to be added to the RN for 4.9 GA, please email @sjstout @jeana-redhat or me.