[release-1.26] Failing secret webhook causes unexpected behavior

[brandond]

• Failing secret webhook causes unexpected behavior #7654 backport

[est-suse]

Validated using RC v1.26.6-rc1+k3s1

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04

Cluster Configuration:
Two node, One server, One agent

Steps to reproduce as shared in the issue:
Install k3s
Create a bad webhook configuration (this one just calls out to httpbin.org for a 502 error response)

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: rancher.cattle.io
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    url: https://httpbin.org/status/502
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: rancher.cattle.io.secrets
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - secrets
    scope: Namespaced
  sideEffects: NoneOnDryRun
  timeoutSeconds: 5

Attempt to join a new agent

Results from reproducing the issue

NAME             STATUS   ROLES                       AGE   VERSION
ip-172-1-1-6   Ready    control-plane,etcd,master   20m   v1.26.5+k3s1

Agent is unable to join the cluster. Hangs during the joining process.

Error message from server

time="2023-06-16T14:41:41Z" level=error msg="Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"https://httpbin.org/status/502?timeout=5s\": context deadline exceeded"

Results from validating the issue:

Agent has joined the cluster.

NAME               STATUS   ROLES                       AGE     VERSION
ip-172-1-1-1   Ready    control-plane,etcd,master   4m5s    v1.26.6-rc1+k3s1
ip-172-1-1-1  Ready    <none>                      2m10s   v1.26.6-rc1+k3s1

NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-59b4f5bbd5-mb8zm                  1/1     Running     0          3m42s
kube-system   helm-install-traefik-496j4                0/1     Completed   1          3m43s
kube-system   helm-install-traefik-crd-6mrcs            0/1     Completed   0          3m43s
kube-system   local-path-provisioner-76d776f6f9-l9xbr   1/1     Running     0          3m42s
kube-system   metrics-server-68cf49699b-cr9kb           1/1     Running     0          3m42s
kube-system   svclb-traefik-d301a0af-6r4qd              2/2     Running     0          2m1s
kube-system   svclb-traefik-d301a0af-tx92j              2/2     Running     0          3m30s
kube-system   traefik-57c84cf78d-8h8pv                  1/1     Running     0          3m30s