Load Balancer #250

yunkon-kim · 2022-03-21T06:02:56Z

yunkon-kim
Mar 21, 2022
Maintainer

I'd like to share an issue and study of load balancer in this discussion.

2 types of clusters will be used.

(On-prem K3s) K3s cluster on a local network (1 master and 2 nodes)
(MCKS) Kubernetes cluster across multi-clouds (1 master and 2 nodes on different CSPs)

yunkon-kim · 2022-03-21T06:24:27Z

yunkon-kim
Mar 21, 2022
Maintainer Author

[Command]

kubectl get services --all-namespaces

[On-prem K3s]

Status: It's working.

NAMESPACE     NAME             TYPE           CLUSTER-IP     EXTERNAL-IP                                    PORT(S)                      AGE
default       kubernetes       ClusterIP      10.43.0.1      <none>                                         443/TCP                      33d
kube-system   kube-dns         ClusterIP      10.43.0.10     <none>                                         53/UDP,53/TCP,9153/TCP       33d
kube-system   metrics-server   ClusterIP      10.43.255.18   <none>                                         443/TCP                      33d
kube-system   traefik          LoadBalancer   10.43.1.115    192.168.10.100,192.168.10.101,192.168.10.102   80:30519/TCP,443:30328/TCP   33d

[MCKS]

Status: It's pending.

Log:

Error from server: Get "https://192.168.2.135:10250/containerLogs/default/traefik-64754ffbc5-f6cq2/traefik": dial tcp 192.168.2.135:10250: i/o timeout

Reason: Traefik dials to IP address of host's default network interface (i.e., eth0, ens4, and ens5) instead of cbnet0

NAMESPACE     NAME         TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE
default       kubernetes   ClusterIP      10.96.0.1     <none>        443/TCP                      36m
default       traefik      LoadBalancer   10.111.19.8   <pending>     80:30112/TCP,443:31684/TCP   30m
kube-system   kube-dns     ClusterIP      10.96.0.10    <none>        53/UDP,53/TCP,9153/TCP       36m

0 replies

yunkon-kim · 2022-03-22T09:24:33Z

yunkon-kim
Mar 22, 2022
Maintainer Author

[MCKS]

Resovled (see the memo below):

Traefik dials to IP address of host's default network interface (i.e., eth0, ens4, and ens5) instead of cbnet0

Issue:

Current stats: Unreachable to `service/kubernetes as follows
Guess: It would be related to Traefik's TLS setting.

kubectl -n traefik logs traefik-64754ffbc5-2v4ht

time="2022-03-22T08:57:15Z" level=info msg="Configuration loaded from flags."
time="2022-03-22T08:57:45Z" level=error msg="Error watching kubernetes events: could not retrieve server version: Get \"https://10.96.0.1:443/version?timeout=32s\": dial tcp 10.96.0.1:443: i/o timeout" providerName=kubernetes
I0322 08:57:45.156349       1 trace.go:205] Trace[623375845]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (22-Mar-2022 08:57:15.154) (total time: 30001ms):
Trace[623375845]: [30.001329406s] [30.001329406s] END
E0322 08:57:45.156403       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.IngressRoute: failed to list *v1alpha1.IngressRoute: Get "https://10.96.0.1:443/apis/traefik.containo.us/v1alpha1/ingressroutes?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0322 08:57:45.156474       1 trace.go:205] Trace[1255077237]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (22-Mar-2022 08:57:15.154) (total time: 30001ms):
Trace[1255077237]: [30.001606219s] [30.001606219s] END
I0322 08:57:45.156500       1 trace.go:205] Trace[2106520400]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (22-Mar-2022 08:57:15.154) (total time: 30001ms):
Trace[2106520400]: [30.001685609s] [30.001685609s] END
E0322 08:57:45.156500       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TraefikService: failed to list *v1alpha1.TraefikService: Get "https://10.96.0.1:443/apis/traefik.containo.us/v1alpha1/traefikservices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
E0322 08:57:45.156513       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.ServersTransport: failed to list *v1alpha1.ServersTransport: Get "https://10.96.0.1:443/apis/traefik.containo.us/v1alpha1/serverstransports?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0322 08:57:45.156475       1 trace.go:205] Trace[827674776]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (22-Mar-2022 08:57:15.155) (total time: 30001ms):
Trace[827674776]: [30.001166571s] [30.001166571s] END
I0322 08:57:45.156623       1 trace.go:205] Trace[98167010]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (22-Mar-2022 08:57:15.156) (total time: 30000ms):
Trace[98167010]: [30.000486895s] [30.000486895s] END

kubectl get all -o wide --all-namespaces

NAMESPACE     NAME                                           READY   STATUS    RESTARTS      AGE    IP            NODE                             NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-64897985d-p6kgl                    0/1     Running   1 (24m ago)   101m   10.77.2.4     ns01-yk3m-c8sn2mag4u9u9eap5jq0   <none>           <none>
kube-system   pod/coredns-64897985d-vwmcg                    0/1     Running   1 (24m ago)   101m   10.77.2.5     ns01-yk3m-c8sn2mag4u9u9eap5jq0   <none>           <none>
kube-system   pod/etcd-ip-192-168-2-136                      1/1     Running   2 (24m ago)   102m   192.168.0.2   ip-192-168-2-136                 <none>           <none>
kube-system   pod/kube-apiserver-ip-192-168-2-136            1/1     Running   2 (23m ago)   102m   192.168.0.2   ip-192-168-2-136                 <none>           <none>
kube-system   pod/kube-controller-manager-ip-192-168-2-136   1/1     Running   1 (24m ago)   102m   192.168.0.2   ip-192-168-2-136                 <none>           <none>
kube-system   pod/kube-flannel-ds-7m9mr                      1/1     Running   1 (24m ago)   71m    192.168.0.2   ip-192-168-2-136                 <none>           <none>
kube-system   pod/kube-flannel-ds-rxm8b                      1/1     Running   1 (24m ago)   71m    192.168.0.6   ns01-yk3m-c8sn2n2g4u9u9eap5jqg   <none>           <none>
kube-system   pod/kube-flannel-ds-tgwwr                      1/1     Running   2 (22m ago)   71m    192.168.0.5   ns01-yk3m-c8sn2mag4u9u9eap5jq0   <none>           <none>
kube-system   pod/kube-proxy-7s25q                           1/1     Running   1 (24m ago)   101m   192.168.0.2   ip-192-168-2-136                 <none>           <none>
kube-system   pod/kube-proxy-99rz9                           1/1     Running   1 (24m ago)   101m   192.168.0.6   ns01-yk3m-c8sn2n2g4u9u9eap5jqg   <none>           <none>
kube-system   pod/kube-proxy-nfnwg                           1/1     Running   1 (24m ago)   101m   192.168.0.5   ns01-yk3m-c8sn2mag4u9u9eap5jq0   <none>           <none>
kube-system   pod/kube-scheduler-ip-192-168-2-136            1/1     Running   1 (24m ago)   102m   192.168.0.2   ip-192-168-2-136                 <none>           <none>
traefik       pod/traefik-64754ffbc5-2v4ht                   0/1     Running   0             8s     10.77.1.5     ns01-yk3m-c8sn2n2g4u9u9eap5jqg   <none>           <none>

NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE    SELECTOR
default       service/kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP                      102m   <none>
kube-system   service/kube-dns     ClusterIP      10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP       102m   k8s-app=kube-dns
traefik       service/traefik      LoadBalancer   10.104.86.117   <pending>     80:31131/TCP,443:32755/TCP   8s     app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik

NAMESPACE     NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE    CONTAINERS     IMAGES                                        SELECTOR
kube-system   daemonset.apps/kube-flannel-ds   3         3         3       3            3           <none>                   71m    kube-flannel   rancher/mirrored-flannelcni-flannel:v0.17.0   app=flannel
kube-system   daemonset.apps/kube-proxy        3         3         3       3            3           kubernetes.io/os=linux   102m   kube-proxy     k8s.gcr.io/kube-proxy:v1.23.5                 k8s-app=kube-proxy

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                              SELECTOR
kube-system   deployment.apps/coredns   0/2     2            0           102m   coredns      k8s.gcr.io/coredns/coredns:v1.8.6   k8s-app=kube-dns
traefik       deployment.apps/traefik   0/1     1            0           8s     traefik      traefik:2.6.1                       app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE    CONTAINERS   IMAGES                              SELECTOR
kube-system   replicaset.apps/coredns-64897985d    2         2         0       101m   coredns      k8s.gcr.io/coredns/coredns:v1.8.6   k8s-app=kube-dns,pod-template-hash=64897985d
traefik       replicaset.apps/traefik-64754ffbc5   1         1         0       8s     traefik      traefik:2.6.1                       app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik,pod-template-hash=64754ffbc5

[Memo]

Change default network interface

Download kube-flannel.yml and add `- --iface=cbnet0`

      containers:
      - name: kube-flannel
       #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply)
        image: rancher/mirrored-flannelcni-flannel:v0.17.0
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=cbnet0

Apply the `kube-flannel.yml`

kubectl apply -f kube-flannel.yml

Change the node IP address in each node

Check node IP address (i.e., INTERNAL-IP)

kubectl get nodes -o wide

NAME                             STATUS   ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
ip-192-168-2-136                 Ready    control-plane,master   41m   v1.23.5   192.168.2.136   <none>        Ubuntu 18.04.3 LTS   4.15.0-1051-aws    docker://20.10.13
ns01-yk3m-c8sn2mag4u9u9eap5jq0   Ready    <none>                 41m   v1.23.5   192.168.3.132   <none>        Ubuntu 18.04.5 LTS   5.4.0-1051-azure   docker://20.10.13
ns01-yk3m-c8sn2n2g4u9u9eap5jqg   Ready    <none>                 41m   v1.23.5   192.168.4.130   <none>        Ubuntu 18.04.3 LTS   5.0.0-1021-gcp     docker://20.10.13

Change (overwrite) node IP

sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

(the last line: --node-ip)

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip 192.168.0.2

sudo systemctl daemon-reload && sudo systemctl restart kubelet

Result (INTERNAL-IPs are changed)

kubectl get node -o wide

NAME                             STATUS   ROLES                  AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
ip-192-168-2-136                 Ready    control-plane,master   103m   v1.23.5   192.168.0.2   <none>        Ubuntu 18.04.3 LTS   4.15.0-1051-aws    docker://20.10.13
ns01-yk3m-c8sn2mag4u9u9eap5jq0   Ready    <none>                 103m   v1.23.5   192.168.0.5   <none>        Ubuntu 18.04.5 LTS   5.4.0-1051-azure   docker://20.10.13
ns01-yk3m-c8sn2n2g4u9u9eap5jqg   Ready    <none>                 103m   v1.23.5   192.168.0.6   <none>        Ubuntu 18.04.3 LTS   5.0.0-1021-gcp     docker://20.10.13

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load Balancer #250

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments

{{title}}

{{title}}

Select a reply

Load Balancer #250

yunkon-kim Mar 21, 2022 Maintainer

Replies: 2 comments

yunkon-kim Mar 21, 2022 Maintainer Author

yunkon-kim Mar 22, 2022 Maintainer Author

Change default network interface

Download kube-flannel.yml and add - --iface=cbnet0

Apply the kube-flannel.yml

Change the node IP address in each node

Check node IP address (i.e., INTERNAL-IP)

Change (overwrite) node IP

Result (INTERNAL-IPs are changed)

yunkon-kim
Mar 21, 2022
Maintainer

yunkon-kim
Mar 21, 2022
Maintainer Author

yunkon-kim
Mar 22, 2022
Maintainer Author

Download kube-flannel.yml and add `- --iface=cbnet0`

Apply the `kube-flannel.yml`