Unable to authenticate the request err="verifying certificate failed: x509: certificate signed by unknown authority""

[hariapollo]

As suggested here, we have tried to upgrade stack driver to newer version though we got the certificate authenticate issue.
GKE Version: 1.26.11-gke.1055000
Custom stack driver version: v0.14.2-gke.0

After upgrading we got the below error.

E0321 13:07:02.418167    3420 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
I0321 07:24:00.553580       1 adapter.go:200] serverOptions: {true true true true false   false false}
I0321 07:24:00.553651       1 adapter.go:210] ListFullCustomMetrics is disabled, which would only list 1 metric resource to reduce memory usage. Add --list-full-custom-metrics to list full metric resources for debugging.
I0321 07:24:01.754403       1 request.go:665] Waited for 1.00918148s due to client-side throttling, not priority and fairness, request: GET:https://10.124.0.1:443/apis/kafka.strimzi.io/v1beta1?timeout=32s
I0321 07:24:03.347423       1 serving.go:341] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0321 07:24:07.552005       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0321 07:24:07.552034       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0321 07:24:07.552284       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0321 07:24:07.552087       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0321 07:24:07.552656       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0321 07:24:07.552236       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0321 07:24:07.552617       1 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0321 07:24:07.553506       1 secure_serving.go:256] Serving securely on [::]:443
I0321 07:24:07.553568       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0321 07:24:07.739320       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E0321 07:24:07.740818       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.741585       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.741994       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.742335       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.742633       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.742915       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.743207       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.743494       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.743779       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.744091       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.744368       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.744583       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.744790       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.744965       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.745090       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
I0321 07:24:07.839380       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
E0321 07:24:07.840068       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840227       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840440       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840605       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840695       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840720       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840843       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840865       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840941       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840977       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.841015       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840069       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.841054       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
I0321 07:24:07.840090       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E0321 07:24:07.840157       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
E0321 07:24:07.840790       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=23850158042506833490065535935480109479, SKID=, AKID=53:D7:E6:89:D7:7D:31:34:C7:C2:64:E4:50:89:76:08:7E:53:17:A9 failed: x509: certificate signed by unknown authority"
I0321 07:24:47.623921       1 trace.go:205] Trace[145270202]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|sidekiq_latency,user-agent:vpa-recommender/v0.0.0 (linux/amd64) kubernetes/$Format/metrics-horizontal-pod-autoscaler,audit-id:565dde9d-e469-431f-a9dd-72eb798df7e2,client:10.126.0.4,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (21-Mar-2024 07:24:46.075) (total time: 1548ms):
Trace[145270202]: ---"Listing from storage done" 1548ms (07:24:47.623)
Trace[145270202]: [1.548559548s] [1.548559548s] END
I0321 07:28:58.587083       1 trace.go:205] Trace[1540409068]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|sidekiq_latency,user-agent:vpa-recommender/v0.0.0 (linux/amd64) kubernetes/$Format/metrics-horizontal-pod-autoscaler,audit-id:62aceb74-c5e7-4f22-8f76-c37ef2db794d,client:10.126.0.4,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (21-Mar-2024 07:28:57.871) (total time: 715ms):
Trace[1540409068]: ---"Listing from storage done" 715ms (07:28:58.586)
Trace[1540409068]: [715.65654ms] [715.65654ms] END
I0321 07:34:06.973729       1 trace.go:205] Trace[993740446]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|sidekiq_latency,user-agent:vpa-recommender/v0.0.0 (linux/amd64) kubernetes/$Format/metrics-horizontal-pod-autoscaler,audit-id:320d954c-e85d-45de-9511-398536215ea7,client:10.126.0.4,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (21-Mar-2024 07:34:05.496) (total time: 1477ms):
Trace[993740446]: ---"Listing from storage done" 1477ms (07:34:06.973)
Trace[993740446]: [1.477411244s] [1.477411244s] END
I0321 07:36:31.582131       1 trace.go:205] Trace[71168755]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|sidekiq_latency,user-agent:vpa-recommender/v0.0.0 (linux/amd64) kubernetes/$Format/metrics-horizontal-pod-autoscaler,audit-id:f707bdfd-1d77-4d58-8b00-caa6a6fcc2df,client:10.126.0.4,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (21-Mar-2024 07:36:30.991) (total time: 590ms):
Trace[71168755]: ---"Listing from storage done" 590ms (07:36:31.581)
Trace[71168755]: [590.539263ms] [590.539263ms] END

[CatherineF-dev]

qq: is your HPA still working?

[CatherineF-dev]

This error seems transient E0321 07:24:07.740818

[KDanisme]

I'm getting the same issue, in version v0.15.0-gke.0, cluster version: 1.29.1-gke.1589017.
The metric isn't registering in the metric-server. so hpa is not working.

[CatherineF-dev]

@KDanisme could you give some detailed steps on reproducing this issue?