OpenShift 3.7 Release Notes Tracker

[mfojtik]

All notes related to the Origin / OCP 3.7 release

[ahardin-rh]

Notes are already being tracked via #4021

Thanks!

[mfojtik]

@ahardin-rh sorry this was for 3.7 ;-) wrong copy/paste...

[mfojtik]

The 'generatedeploymentconfig' API endpoint will be removed: openshift/origin#15585

[ironcladlou]

We are deprecating two deployment environment variables: OPENSHIFT_MASTER and KUBERNETES_MASTER. The variables were scrutinized in a Bugzilla report.

Although the variables aren't documented and aren't used by OpenShift's supported deployment strategies, it's possible there are custom deployer images or hooks in the wild which may be using the variables.

@smarterclayton, do you know of any user-facing docs around these variables? A search of the docs repo turned up the following potential references:

• https://github.com/openshift/openshift-docs/blob/master/architecture/core_concepts/pods_and_services.adoc
• https://github.com/openshift/openshift-docs/blob/master/using_images/xpaas_images/fuse.adoc

Release note draft

OpenShift deployments using a custom strategy or hooks are provided with a container environment which includes
two variables for API connectivity:

• OPENSHIFT_MASTER: a URL to the OpenShift API
• KUBERNETES_MASTER: a URL to the Kubernetes API exposed by OpenShift

These variables are now deprecated, as they refer to internal endpoints rather than the published OpenShift API service endpoints. To connect to the OpenShift API in these contexts, use service DNS or the automatically exposed KUBERENTES service environment variables.

The OPENSHIFT_MASTER and KUBERNETES_MASTER environment variables will be removed from deployment container environments as of $OPENSHIFT_VERSION.

[adellape]

The openshift_hosted_{logging,metrics}_* Ansible variables used by the installer have been deprecated. Installation documentation has been updated to use the newer variable names. The deprecated variable names are planned for removal in the next minor release of OpenShift Container Platform.

(Per https://trello.com/c/6FsbeKeJ.)

[simo5]

We are deprecating a large number of policy related APIs and commands.
In 3.7 we completely removed the policy objects, and we are using native RBAC instead.
Any command trying to directly manipulate a policy object will fail. Roles and Rolebindings endpoints are still available, and they proxy the operation to create native RBAC objects instead.
The following commands do not work against a 3.7 server:

oadm overwrite-policy
oadm migrate authorization
oc create policybinding

Note: a 3.7 client will display an error message when trying these command against a 3.7 server, but will still work against a previous server version, and old client will just fail hard against a 3.7 server.

[vikram-redhat]

From Ben B: The odds of someone hitting this (https://bugzilla.redhat.com/show_bug.cgi?id=1500961) are so slim there's no point. Especially if we get the ostree update out in a day or two after 7.4.2.

Where we might make a note is in the OCP docs for 3.7, stating that 7.4.2.1 or newer is required for a containerized install.

@adellape @ahardin-rh - might need an update in both the release notes and install docs.

[sdodson]

Starting with 3.7 versions of the installer if you've configured AWS provider credentials you must also ensure that all instances are labeled and then set openshift_clusterid variable to the cluster id. Please see #5732 for more details.

[php-coder]

Some SCCs became stricter and now drop the following capabilities:

• nonroot drops KILL, MKNOD, SETUID, and SETGID
• hostaccess drops KILL, MKNOD, SETUID, and SETGID
• hostmount-anyuid drops MKNOD
  It's possible that the pods that previously were admitted by these SCCs and were using such capabilities will fail after upgrade. In this (rare) cases, cluster administrator should create a custom SCC for such pods.

Related PR: openshift/origin#16436

[adellape]

Updated installer support for CFME 4.6 on OCP 3.7.

[enj]

A 3.7 master will return an unstructured response instead of structured JSON when an action is forbidden. This is a known issue (openshift/origin/issues/16731) and will be fixed in 3.8.

[sdodson]

Known issue: The installer can not deploy system container based installations when the specified registry requires authentication credentials in order to pull the required system container images. The fix for this depends on an update to the atomic command which will be updated after 3.7 GA.

https://bugzilla.redhat.com/show_bug.cgi?id=1505744

[tsmetana]

Known issue: The volume snapshot tech-preview feature may not be available to non-admin users by default due to API RBAC settings. When the volume snapshot controller and provisioner are installed and run, the cluster admin needs to configure the API access to the VolumeSnapshot objects by creating Roles/ClusterRoles and assigning them to the desired users or user groups.

https://bugzilla.redhat.com/show_bug.cgi?id=1502945

[enj]

The node authorizer and admission plugin are used to manage and limit node's permissions in 3.7. Thus nodes should be removed from the group that previously granted them broad permissions across the cluster:

oc adm policy remove-cluster-role-from-group system:node system:nodes

[enj]

@ahardin-rh In regard to the last release note (#4906 (comment)), we plan to perform this step automatically via Ansible as a post upgrade step once the cluster is upgraded to 3.8. Specifically, since this a new feature, we want to give users a release (i.e. 3.7) before forcing it on for them.

[knobunc]

The kube-service-catalog namespace is now also made global (by ansible openshift/openshift-ansible#4756), so if you want multicast to work in vnid 0, you have to set the netnamespace.network.openshift.io/multicast-enabled=true annotation on both namespaces (i.e. default and kube-service-catalog).

[enj]

Migration to Kubernetes Role Based Access Control (RBAC)

Steps Taken During the 3.6 Release

A custom migration controller was created to automatically migrate OpenShift authorization policy resources to the equivalent RBAC resources:

1. If an OpenShift authorization policy resource was created or modified or deleted, the action was automatically mirrored to the corresponding RBAC resource
2. Changes directly applied to RBAC resources were generally automatically rolled back and forced to match the corresponding Openshift authorization policy resource. If no corresponding resource existed, the RBAC resource would be deleted.

In essence, OpenShift authorization policy objects were the source of truth, and the RBAC objects were forced into matching these objects.

---

Release 3.6 Pre-upgrade steps before upgrading to 3.7

There is a small set of configurations that are possible in OpenShift authorization policy resources that are not supported by RBAC. Such configurations require manual migration based on the customer's use case. To guarantee that all Openshift authorization policy objects are in sync with RBAC, the oc adm migrate authorization command has been added. This read-only command emulates the migration controller logic, and reports if any resource is out of sync. It is run as a pre-upgrade step via an ansible playbook and will cause the upgrade to fail if the objects are not in sync.

---

During rolling upgrade from release 3.6 to 3.7

The following scenario describes a rolling upgrade

1. One master is upgraded and starts proxying OpenShift authorization policy resources and authorizing against RBAC objects
2. Old masters are still running the migration controller and one of them holds the controller leader election lock (either because it already had it or because it gained it by the first master being upgraded)
3. The new master cannot modify any RBAC or proxied OpenShift authorization policy objects because the migration controller will undo all changes
4. Old masters can change OpenShift authorization policy resources and the migration controller will sync these to RBAC, making the changes visible to the new master
5. The new master does not have the migration controller
6. Controllers only speak to their local masters in OpenShift installed via Ansible, thus the migration controller is guaranteed to only communicate with the old masters
7. There is a small chance that a 3.7 controller process will become the leader once two masters have been upgraded (meaning no migrations of policy objects will occur after this point)
8. Once all masters have been upgraded from 3.6 to 3.7, OpenShift authorization policy objects will be always proxied to RBAC objects
9. The migration controller will be gone and it will be possible to make changes to RBAC objects directly

Considerations for administrators during rolling upgrade

Avoid actions that require changes to OpenShift authorization policy resources such as the creation of new projects. If a project is created against a new master, the RBAC resources it creates will be deleted by the migration controller since they will be seen as out of sync from the OpenShift authorization policy resources. If a project is created against an old master and the migration controller is no longer present due to a 3.7 controller process being the leader, then its policy objects will not be synced and it will have no RBAC resources. After the 3.7 upgrade is complete, the following read-only script can be used to determine what namespaces lack RBAC role bindings (it is up to the cluster administrator to decide how to remediate these namespaces):

#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

for namespace in $(oc get namespace -o name); do
    ns=$(echo "${namespace}" | cut -d / -f 2)
    rolebindings_count=$(oc get rolebinding.rbac -o name -n "${ns}" | wc -l)
    if [[ "${rolebindings_count}" == "0" ]]; then
        echo "Namespace ${ns} has no role bindings which may require further investigation"
    else
        echo "Namespace ${ns}: ok"
    fi
done

---

RBAC and OpenShift authorization policy in release 3.7

In 3.7, the RBAC objects become the source of truth. The OpenShift authorization policy objects no longer exist as real objects; the APIs are proxied to the RBAC resources. Therefore creating or modifying or deleting OpenShift authorization policy resources seamlessly results in actions against RBAC objects. The API master handles the conversion between these resources and legacy clients will continue to work as if nothing has changed. The RBAC objects also support watches, unlike the OpenShift authorization policy resources.

Policy-based resources have been removed in 3.7. However, RBAC role and binding objects are available and provide equivalent functionality.

[mwithrow]

is the catalog api opened back up in 3.7? I can't find any info on it.

[chengzhang1016]

@mfojtik
We(svc-catalog & asb QE) hit a issue in ocp3.6 - ASB bootstrap failed while apb registry exist images of 3.7
https://bugzilla.redhat.com/show_bug.cgi?id=1515211

That means if apb registry only have 3.6 apb images, asb work well. But, will have problem while we upgrade to ocp3.7.

@jwmatthews @pmorie Please help to confirm this issue and decide if need to add release note. Thanks.

[liggitt]

@openshift/team-documentation can this be closed now?