metrics-server request canceled while waiting for connection #141

f4ct0r · 2018-09-20T08:53:35Z

I have deployed a metrics-server in kubernetes following https://github.com/kubernetes-incubator/metrics-server/tree/v0.2.1/deploy, and my metrics-server image is from gcr.io/google_containers/metrics-server-amd64:v0.2.1.
Metric server is running fine and but I am not able to get metrics from it. And I'm using Kubernetes version is 1.11.0.
The problem I met as followings:

[root@k8s-master metrics-server]# kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: 2018-09-12T09:29:13Z
  name: v1beta1.metrics.k8s.io
  resourceVersion: "828496"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: 4fc0035b-b66e-11e8-aa6e-000c29985c64
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: 2018-09-12T09:29:13Z
    message: 'no response from https://169.169.50.19:443: Get https://169.169.50.19:443:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

And I use curl https://169.169.50.19:443, but nothing return.

metircs-server logs:

I0920 06:40:56.625243       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --requestheader-client-ca-file=/etc/kubernetes/ssl/kubelet_client.crt
I0920 06:40:56.625285       1 heapster.go:72] Metrics Server version v0.2.1
I0920 06:40:56.625439       1 configs.go:61] Using Kubernetes client with master "https://169.169.0.1:443" and version
I0920 06:40:56.625447       1 configs.go:62] Using kubelet port 10250
I0920 06:40:56.626126       1 heapster.go:128] Starting with Metric Sink
I0920 06:40:56.988875       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0920 06:40:57.446216       1 heapster.go:101] Starting Heapster API server...
[restful] 2018/09/20 06:40:57 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/09/20 06:40:57 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0920 06:40:57.448570       1 serve.go:85] Serving securely on 0.0.0.0:443

Kubernetes master:
apiserver:

ETCD_SERVER="--etcd-servers=http://127.0.0.1:2379"
INSECURE_BIND_ADDRESS="--insecure-bind-address=0.0.0.0"
INSECURE_PORT="--insecure-port=8080"
SECURE_PORT="--secure-port=6443"
SERVICE_CLUSTER_IP_RANGE="--service-cluster-ip-range=169.169.0.0/16"
SERVICE_NODE_PORT_RANGE="--service-node-port-range=1-65535"
ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=4"
CLIENT_CA_FILE="--client-ca-file=/etc/kubernetes/ssl/ca.crt"
TLS_PRIVATE_KEY="--tls-private-key-file=/etc/kubernetes/ssl/server.key"
TLS_CERT_FILE="--tls-cert-file=/etc/kubernetes/ssl/server.crt"
SERVICE_ACCOUNT_KEY_FILE="--service-account-key-file=/etc/kubernetes/ssl/server.key"
REQUESTHEADER_CLIENT_CA_FILE="--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.crt"
REQUESTHEADER_ALLOWED_NAMES="--requestheader-allowed-names=aggregator"
REQUESTHEADER_EXTRA_HEADERS_PREFIX="--requestheader-extra-headers-prefix=X-Remote-Extra-"
REQUESTHEADER_GROUP_HEADERS="--requestheader-group-headers=X-Remote-Group"
REQUESTHEADER_USERNAME_HEADERS="--requestheader-username-headers=X-Remote-User"
KUBE_API_ARGS="--enable-swagger-ui=true"

controller-manager:

API_MASTER="--master=https://k8s-master:6443"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=4"
SERVICE_ACCOUNT_KEY_FILE="--service-account-private-key-file=/etc/kubernetes/ssl/server.key"
CLUSTER_SIGNING_CERT_FILE=""
CLUSTER_SIGNING_KEY_FILE=""
ROOT_CA_FILE="--root-ca-file=/etc/kubernetes/ssl/ca.crt"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kubeconfig"
KUBE_CONTROLLER_MANAGER_ARGS='--cloud-provider=""'

kubeconfig:

apiVersion: v1
kind: Config
users:
- name: controllermanager
  user:
    client-certificate: /etc/kubernetes/ssl/cs_client.crt
    client-key: /etc/kubernetes/ssl/cs_client.key
  clusters:
  - name: local
    cluster:
      certificate-authority: /etc/kubernetes/ssl/ca.crt
contexts:
- context:
    cluster: local
    user: controllermanager
  name: my-context
current-context: my-context

proxy:

API_MASTER="--master=https://k8s-master:6443"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=4"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kubeconfig"
KUBE_PROXY_ARGS=""

Kubernetes node:
kubelet:

ADDRESS="--address=10.0.83.63"
HOSTNAME_OVERRIDE="--hostname-override=10.0.83.63"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=5"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kubeconfig"
PAUSE_IMAGE="--pod_infra_container_image=kubernetes/pause"
KUBELET_ARGS='--cloud-provider=""'

proxy:

API_MASTER="--master=https://k8s-master:6443"
LOGTOSTDERR="--logtostderr=false"
LOGDIR="--log-dir=/var/log/kubernetes"
LOG_LEVEL="--v=4"
KUBECONFIG="--kubeconfig=/etc/kubernetes/kubeconfig"
KUBE_PROXY_ARGS=""

kubeconfig:

apiVersion: v1
users:
- name: kubelet
  user:
    client-certificate: /etc/kubernetes/ssl/kubelet_client.crt
    client-key: /etc/kubernetes/ssl/kubelet_client.key
clusters:
- cluster:
    certificate-authority: ssl/ca.crt
    server: https://k8s-master:6443
  name: local
contexts:
- context:
    cluster: local
    user: kubelet
  name: my-context
current-context: my-context
kind: Config
preferences: {}

And Metrics-server YAML metrics-server-deployment.yaml l I just modified

apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.2.1
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/kubernetes/ssl
          name: ca-ssl
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250
        - --requestheader-client-ca-file=/etc/kubernetes/ssl/kubelet_client.crt
      volumes:
       - name: ca-ssl
         hostPath:
          path: /etc/kubernetes/ssl

So, is there any configuration fault? Anyone has suggestion to solve this problem?

The text was updated successfully, but these errors were encountered:

DirectXMan12 · 2018-09-24T15:26:18Z

Are you running some overlay network on your cluster that prevents the main API server from talking to the the service proxies or pods?

f4ct0r · 2018-09-25T03:55:52Z

@DirectXMan12 Thanks for your reply. overlay network? you mean CNI or other network plugin?

DirectXMan12 · 2018-09-25T15:30:13Z

yeah, some CNI plugins create isolation between different parts of the system. Sometimes this isolation breaks Kubernetes features

f4ct0r · 2018-09-26T07:50:26Z

I have not installed any network plugins before, so master cannot access pods on other nodes?
When I installed CNI plugin sucessfully, then this problem can be solved, right?
Could you please give me an User Guide to install CNI?

DirectXMan12 · 2018-09-27T19:49:03Z

You'll have to check the kubernetes documentation -- there's a guide here: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/

f4ct0r · 2018-10-08T04:01:18Z

I have installed CNI plugin, but metrics-server pod cannot be created, and error return

Name:           metrics-server-68c7fb9fd4-jtv6h
Namespace:      kube-system
Node:           worker-node/10.0.83.63
Start Time:     Sun, 30 Sep 2018 17:10:32 +0800
Labels:         k8s-app=metrics-server
                pod-template-hash=2473965980
Annotations:    <none>
Status:         Pending
IP:
Controlled By:  ReplicaSet/metrics-server-68c7fb9fd4
Containers:
  metrics-server:
    Container ID:
    Image:         docker.tophant.com/google_containers/metrics-server-amd64:v0.2.1
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /metrics-server
      --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250
      --requestheader-client-ca-file=/etc/kubernetes/ssl/kubelet.pem
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/kubernetes/ssl from ca-ssl (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-wlzn9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  ca-ssl:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/ssl
    HostPathType:
  metrics-server-token-wlzn9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-wlzn9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason                  Age                From                  Message
  ----     ------                  ----               ----                  -------
  Warning  FailedCreatePodSandBox  4m (x141 over 1h)  kubelet, worker-node  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d3eee93c3d626d7a37286f7568a0bee6e03ab2684b788ed7ffd2d38c95660a28" network for pod "metrics-server-68c7fb9fd4-jtv6h": NetworkPlugin cni failed to set up pod "metrics-server-68c7fb9fd4-jtv6h_kube-system" network: Get https://[10.1.0.1]:443/api/v1/namespaces/kube-system/pods/metrics-server-68c7fb9fd4-jtv6h: dial tcp 10.1.0.1:443: i/o timeout

It said NetworkPlugin cni failed to set up pod, but I don't know why
And all the pod cannot be create because of this error

mingfang · 2018-10-17T13:45:25Z

This is a major problem in the whole master -> metrics server design. Masters are not supposed to be on the overlay network for security reasons.
With the old InfluxDB setup I was able to run InfluxDB on the host network, same as the master, but this new Metric Server does not support that setup.

As an enhancement I think the Metric Server should be able to run host network.

DirectXMan12 · 2018-11-08T00:30:49Z

you could run metrics-server in hostnetwork mode. It's not a metrics-server issue -- it's in the core design of the Kubernetes aggregation layer. If you have concerns with this model, please discuss in the main kubernetes repo with the SIG APIMachinery team.

f4ct0r · 2018-11-22T02:55:46Z

I run metrics-server successfully, thanks for these advises :)

zonyes · 2019-04-02T09:06:14Z

i want to know you how to fix? metrics-server in hostnetwork mode? how to setting? @f4ct0r

allansun · 2019-04-08T15:23:52Z

@zonyes

I ran into the same problem and have been trying different things to get metrics-server working on EKS with weave.

And all the comments above are valid.

The only things you need to do is ( after following all the standard procedures mentioned above, I used helm chart to install metrics-server without much customization):

kubectl -n metrics-server edit deployment YOUR_METRICS_SERVER_DEPLOYMENT

Add hostNetwork:true under spec:template:spec (the same level with containers)

Save it and then, it all works like a charm....

kuzm1ch · 2019-04-09T14:26:01Z

The previous answer fixes this issue for me. I've encountered the same issue after upgrade cluster to 1.11.x using kops.
You can just change this line in values of chart:
https://github.com/helm/charts/blob/master/stable/metrics-server/values.yaml#L27

RamanPndy · 2019-06-11T11:59:53Z

@zonyes

I ran into the same problem and have been trying different things to get metrics-server working on EKS with weave.

And all the comments above are valid.

The only things you need to do is ( after following all the standard procedures mentioned above, I used helm chart to install metrics-server without much customization):
kubectl -n metrics-server edit deployment YOUR_METRICS_SERVER_DEPLOYMENT
Add hostNetwork:true under spec:template:spec (the same level with containers)

Save it and then, it all works like a charm....

Hi @allansun ,
even after enabling the host network to true. it didn't work. i'm using flannel as overlay network. and i'm facing this issue.
Name: v1beta1.metrics.k8s.io
Namespace:
Labels: app=metrics-server
chart=metrics-server-2.8.2
heritage=Tiller
release=metrics-server
Annotations:
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2019-06-11T11:49:33Z
Resource Version: 1681
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
UID: fad12538-8c3e-11e9-a91a-005056a5efb4
Spec:
Group: metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: metrics-server
Namespace: kube-system
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2019-06-11T11:49:33Z
Message: no response from https://10.233.47.243:443: Get https://10.233.47.243:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Reason: FailedDiscoveryCheck
Status: False
Type: Available
Events:

event i'm not able to curl https://10.233.47.243:443. it's not giving any response.
also kube-proxy and kube-flannel pods are running fine.
Please suggest me any possible solution. I'm struck with this.
Please let me know if you require any other info.

RaymondKYLiu · 2019-11-20T08:08:12Z

I ran into the same problem on AWS EKS with Secondary Cidr extended.

$ kubectl get apiservice v1beta1.metrics.k8s.io -o yaml                                                                                                                                                                            [15:53:39]
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: "2019-11-20T07:34:35Z"
  labels:
    app: metrics-server
    chart: metrics-server-2.8.8
    heritage: Helm
    release: metrics-server
  name: v1beta1.metrics.k8s.io
  resourceVersion: "1947084"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: 33541d8e-0b68-11ea-9755-065377ff5cdc
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2019-11-20T07:34:35Z"
    message: 'no response from https://100.64.54.127:8443: Get https://100.64.54.127:8443:
      Address is not allowed'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

Because I enable Custom Network Config for the VPC CNI.
I follow @allansun to add hostNetwork: true, the issue was resolved.

2018-11-27 · 2019-12-24T08:55:24Z

@f4ct0r I had the same problem, and I struggled for it.

libintomgit · 2021-08-31T13:11:52Z

I run metrics-server successfully, thanks for these advises :)

How ? that would really help.

serathius · 2021-08-31T13:41:14Z

Please don't post on 3 years old issue as during that time MS code was totally rewritten. If you have similar problem it has definitely different cause and using same fix will not work for your case.

Please create a new issue and provide information required to debug it.

f4ct0r closed this as completed Nov 22, 2018

murali-reddy mentioned this issue Jan 29, 2019

Weave with AWS EKS is not working weaveworks/weave#3335

Closed

lajchon mentioned this issue Nov 19, 2019

Rolling upgrade breaks sidecar-injector webhook on EKS (failed calling admission webhook "sidecar-injector.istio.io": context deadline exceeded) istio/istio#13840

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metrics-server request canceled while waiting for connection #141

metrics-server request canceled while waiting for connection #141

f4ct0r commented Sep 20, 2018

DirectXMan12 commented Sep 24, 2018

f4ct0r commented Sep 25, 2018

DirectXMan12 commented Sep 25, 2018

f4ct0r commented Sep 26, 2018

DirectXMan12 commented Sep 27, 2018

f4ct0r commented Oct 8, 2018 •

edited

Loading

mingfang commented Oct 17, 2018

DirectXMan12 commented Nov 8, 2018

f4ct0r commented Nov 22, 2018

zonyes commented Apr 2, 2019

allansun commented Apr 8, 2019

kuzm1ch commented Apr 9, 2019

RamanPndy commented Jun 11, 2019

RaymondKYLiu commented Nov 20, 2019

2018-11-27 commented Dec 24, 2019

libintomgit commented Aug 31, 2021

serathius commented Aug 31, 2021 •

edited

Loading

metrics-server request canceled while waiting for connection #141

metrics-server request canceled while waiting for connection #141

Comments

f4ct0r commented Sep 20, 2018

DirectXMan12 commented Sep 24, 2018

f4ct0r commented Sep 25, 2018

DirectXMan12 commented Sep 25, 2018

f4ct0r commented Sep 26, 2018

DirectXMan12 commented Sep 27, 2018

f4ct0r commented Oct 8, 2018 • edited Loading

mingfang commented Oct 17, 2018

DirectXMan12 commented Nov 8, 2018

f4ct0r commented Nov 22, 2018

zonyes commented Apr 2, 2019

allansun commented Apr 8, 2019

kuzm1ch commented Apr 9, 2019

RamanPndy commented Jun 11, 2019

RaymondKYLiu commented Nov 20, 2019

2018-11-27 commented Dec 24, 2019

libintomgit commented Aug 31, 2021

serathius commented Aug 31, 2021 • edited Loading

f4ct0r commented Oct 8, 2018 •

edited

Loading

serathius commented Aug 31, 2021 •

edited

Loading