Change of OAuth URL not working as expected

[devops-42]

Describe the bug
We use custom certificates for our OKD clusters in multiple environments. Therefore we tried to adapt the oauth-url from something like oauth-openshift.apps.<our.base.domain.tld> to oauth-openshift.<our.base.domain.tld> according to the documentation https://docs.okd.io/latest/authentication/configuring-internal-oauth.html#customizing-the-oauth-server-url_configuring-internal-oauth.
Side note: We have a custom console URL configured too (using our certificate).

Steps we've done see below in "How reproducable"

The outcome was, that:

• Our custom console route lost it's certificates (reverted back to the default ingress)
• The authentication pods stuck in ContainerCreating because of a missing secret:

MountVolume.SetUp failed for volume "v4-0-config-system-router-certs" : secret "v4-0-config-system-router-certs" not found

The OAuth and the console operators got degraded.

Question: Is there a possibility to get the domain changed (i.e. to remove the apps sub-domain)?

Thanks in advance for your help and comments!

Version
4.11.0-0.okd-2022-08-20-022919

How reproducible
Follow the documentation (linked above):

1. Create a TLS secret in namespace openshift-config
2. Edit the Ingress object cluster by using oc edit ingress.config.openshift.io cluster

apiVersion: config.openshift.io/v1
kind: Ingress
metadata:
  name: cluster
spec:
  componentRoutes:
    - name: oauth-openshift
      namespace: openshift-authentication
      hostname: oauth-openshift.<our.base.domain.tld>
      servingCertKeyPairSecret:
        name: <secret_name> 

We tried to change add the domain key and set it to <our.base.domain.tld> but without success.

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[devops-42]

Hi,

must-gather fails with:

error running backup collection: errors ocurred while gathering data:
    [skipping gathering clusterroles.rbac.authorization.k8s.io/system:registry due to error: clusterroles.rbac.authorization.k8s.io "system:registry" not found, skipping gathering clusterrolebindings.rbac.authorization.k8s.io/registry-registry-role due to error: clusterrolebindings.rbac.authorization.k8s.io "registry-registry-role" not found, skipping gathering secrets/support due to error: secrets "support" not found, skipping gathering CloudPrivateIPConfig.cloud.network.openshift.io due to error: the server doesn't have a resource type "CloudPrivateIPConfig", skipping gathering endpoints/host-etcd-2 due to error: endpoints "host-etcd-2" not found, skipping gathering sharedconfigmaps.sharedresource.openshift.io due to error: the server doesn't have a resource type "sharedconfigmaps", skipping gathering sharedsecrets.sharedresource.openshift.io due to error: the server doesn't have a resource type "sharedsecrets"]error: unable to download output from pod must-gather-qp6w8: error extracting tar at destination directory: unexpected EOF

According to the remaining output the cluster is working well:

When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: d9ea8769-b51a-468e-b708-d2f284e7b846
ClusterVersion: Stable at "4.11.0-0.okd-2022-08-20-022919"
ClusterOperators:
	All healthy and stable

[vrutkovs]

You can connect directly to the master when ssh'ing to one of the masters:

export KUBECONFIG=/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig
oc get nodes # To verify connection is working
oc adm must-gather

[devops-42]

Hi,

that's exactly what I did :)

To be sure I've repeated must-gather and got the same error messages, but I've seen, that the log bundle has been created anyway.
I took a look into the generated files: They contain a plethora of IP addresses, domain names and deployment details of our production system which we aren't allowed to disclose. Are there any possibilities to make the log information anonymous? Is it possible to limit the log information to the relevant pods/operators?

Thanks a lot for your patience and help!

[LinzerToertchen]

Hi!

We are facing the same issue, so maybe the must-gather from my cluster can help. must-gather

[llomgui]

Hello,

I am running a 4.11.0-0.okd-2022-12-02-145640 cluster.

I simply edit using oc edit ingress.config.openshift.io cluster with the following:

spec:
  componentRoutes:
  - hostname: console.<our.base.domain.tld>
    name: console
    namespace: openshift-console
    servingCertKeyPairSecret:
      name: console-serving-cert
  - hostname: oauth-openshift.<our.base.domain.tld>
    name: oauth-openshift
    namespace: openshift-authentication
    servingCertKeyPairSecret:
      name: console-serving-cert
  domain: apps.<our.base.domain.tld>

Once kube-api servers are at the latest revision, clean up the old console-operator spec.

It should be working as expected.