Cannot add master node

[danghung-dev]

I have 2 master nodes, I have run successful make metail

It only show 1 node k get nodes

ssh to 2 nodes then run k3s kubectl get nodes

Here is the log of the error node

k3s kubectl get nodes -v=10
I0214 20:34:41.685229    3600 loader.go:395] Config loaded from file:  /etc/rancher/k3s/k3s.yaml
I0214 20:34:41.685921    3600 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.28.3+k3s2 (linux/amd64) kubernetes/bbafb86" 'https://127.0.0.1:6443/api?timeout=32s'
I0214 20:34:41.686366    3600 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:6443 succeed
I0214 20:34:41.689324    3600 round_trippers.go:553] GET https://127.0.0.1:6443/api?timeout=32s 503 Service Unavailable in 3 milliseconds
I0214 20:34:41.689368    3600 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 2 ms ServerProcessing 0 ms Duration 3 ms
I0214 20:34:41.689378    3600 round_trippers.go:577] Response Headers:
I0214 20:34:41.689387    3600 round_trippers.go:580]     Content-Type: application/json
I0214 20:34:41.689395    3600 round_trippers.go:580]     Content-Length: 124
I0214 20:34:41.689402    3600 round_trippers.go:580]     Date: Thu, 15 Feb 2024 01:34:41 GMT
I0214 20:34:41.689624    3600 request.go:1212] Response Body: {"kind":"Status","metadata":{},"status":"Failure","message":"apiserver not ready","reason":"ServiceUnavailable","code":503}
E0214 20:34:41.689811    3600 memcache.go:265] couldn't get current server API group list: the server is currently unable to handle the request
I0214 20:34:41.689835    3600 cached_discovery.go:120] skipped caching discovery info due to the server is currently unable to handle the request
I0214 20:34:41.689904    3600 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.28.3+k3s2 (linux/amd64) kubernetes/bbafb86" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" 'https://127.0.0.1:6443/api?timeout=32s'
I0214 20:34:41.690395    3600 round_trippers.go:553] GET https://127.0.0.1:6443/api?timeout=32s 503 Service Unavailable in 0 milliseconds
I0214 20:34:41.690412    3600 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0214 20:34:41.690420    3600 round_trippers.go:577] Response Headers:
I0214 20:34:41.690428    3600 round_trippers.go:580]     Content-Type: application/json
I0214 20:34:41.690435    3600 round_trippers.go:580]     Content-Length: 124
I0214 20:34:41.690442    3600 round_trippers.go:580]     Date: Thu, 15 Feb 2024 01:34:41 GMT
I0214 20:34:41.690620    3600 request.go:1212] Response Body: {"kind":"Status","metadata":{},"status":"Failure","message":"apiserver not ready","reason":"ServiceUnavailable","code":503}
E0214 20:34:41.690796    3600 memcache.go:265] couldn't get current server API group list: the server is currently unable to handle the request
I0214 20:34:41.690832    3600 cached_discovery.go:120] skipped caching discovery info due to the server is currently unable to handle the request
I0214 20:34:41.690847    3600 shortcut.go:100] Error loading discovery information: the server is currently unable to handle the request
I0214 20:34:41.690904    3600 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.28.3+k3s2 (linux/amd64) kubernetes/bbafb86" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" 'https://127.0.0.1:6443/api?timeout=32s'
I0214 20:34:41.691369    3600 round_trippers.go:553] GET https://127.0.0.1:6443/api?timeout=32s 503 Service Unavailable in 0 milliseconds
I0214 20:34:41.691404    3600 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0214 20:34:41.691418    3600 round_trippers.go:577] Response Headers:
I0214 20:34:41.691432    3600 round_trippers.go:580]     Content-Type: application/json
I0214 20:34:41.691446    3600 round_trippers.go:580]     Content-Length: 124
I0214 20:34:41.691461    3600 round_trippers.go:580]     Date: Thu, 15 Feb 2024 01:34:41 GMT
I0214 20:34:41.691589    3600 request.go:1212] Response Body: {"kind":"Status","metadata":{},"status":"Failure","message":"apiserver not ready","reason":"ServiceUnavailable","code":503}
E0214 20:34:41.691754    3600 memcache.go:265] couldn't get current server API group list: the server is currently unable to handle the request
I0214 20:34:41.691828    3600 cached_discovery.go:120] skipped caching discovery info due to the server is currently unable to handle the request
I0214 20:34:41.691930    3600 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.28.3+k3s2 (linux/amd64) kubernetes/bbafb86" 'https://127.0.0.1:6443/api?timeout=32s'
I0214 20:34:41.692483    3600 round_trippers.go:553] GET https://127.0.0.1:6443/api?timeout=32s 503 Service Unavailable in 0 milliseconds
I0214 20:34:41.692526    3600 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0214 20:34:41.692542    3600 round_trippers.go:577] Response Headers:
I0214 20:34:41.692558    3600 round_trippers.go:580]     Content-Type: application/json
I0214 20:34:41.692572    3600 round_trippers.go:580]     Content-Length: 124
I0214 20:34:41.692596    3600 round_trippers.go:580]     Date: Thu, 15 Feb 2024 01:34:41 GMT
I0214 20:34:41.692758    3600 request.go:1212] Response Body: {"kind":"Status","metadata":{},"status":"Failure","message":"apiserver not ready","reason":"ServiceUnavailable","code":503}
E0214 20:34:41.692974    3600 memcache.go:265] couldn't get current server API group list: the server is currently unable to handle the request
I0214 20:34:41.692998    3600 cached_discovery.go:120] skipped caching discovery info due to the server is currently unable to handle the request
I0214 20:34:41.693074    3600 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.28.3+k3s2 (linux/amd64) kubernetes/bbafb86" 'https://127.0.0.1:6443/api?timeout=32s'
I0214 20:34:41.693582    3600 round_trippers.go:553] GET https://127.0.0.1:6443/api?timeout=32s 503 Service Unavailable in 0 milliseconds
I0214 20:34:41.693622    3600 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0214 20:34:41.693636    3600 round_trippers.go:577] Response Headers:
I0214 20:34:41.693653    3600 round_trippers.go:580]     Content-Type: application/json
I0214 20:34:41.693661    3600 round_trippers.go:580]     Content-Length: 124
I0214 20:34:41.693668    3600 round_trippers.go:580]     Date: Thu, 15 Feb 2024 01:34:41 GMT
I0214 20:34:41.693860    3600 request.go:1212] Response Body: {"kind":"Status","metadata":{},"status":"Failure","message":"apiserver not ready","reason":"ServiceUnavailable","code":503}
E0214 20:34:41.694072    3600 memcache.go:265] couldn't get current server API group list: the server is currently unable to handle the request
I0214 20:34:41.694094    3600 cached_discovery.go:120] skipped caching discovery info due to the server is currently unable to handle the request
I0214 20:34:41.694292    3600 helpers.go:246] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server is currently unable to handle the request",
  "reason": "ServiceUnavailable",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "unknown"
      }
    ]
  },
  "code": 503
}]
Error from server (ServiceUnavailable): the server is currently unable to handle the request
root@k3s-vm02:~# CURL https://127.0.0.1:6443/api
-bash: CURL: command not found
root@k3s-vm02:~# wget https://127.0.0.1:6443/api
--2024-02-14 20:35:07--  https://127.0.0.1:6443/api
Connecting to 127.0.0.1:6443... connected.
ERROR: The certificate of ‘127.0.0.1’ is not trusted.
ERROR: The certificate of ‘127.0.0.1’ doesn't have a known issuer.
root@k3s-vm02:~# lsof -i :6443
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
k3s-serve 1290 root    7u  IPv4   1883      0t0  TCP k3s-vm02:56420->10.86.101.237:sun-sr-https (ESTABLISHED)
k3s-serve 1290 root    8u  IPv6  11022      0t0  TCP *:sun-sr-https (LISTEN)
k3s-serve 1290 root    9u  IPv4   7620      0t0  TCP localhost:33116->localhost:sun-sr-https (ESTABLISHED)
k3s-serve 1290 root   10u  IPv6   1885      0t0  TCP localhost:sun-sr-https->localhost:33116 (ESTABLISHED)
k3s-serve 1290 root   14u  IPv4   1900      0t0  TCP localhost:33206->localhost:sun-sr-https (ESTABLISHED)
k3s-serve 1290 root   15u  IPv6   9768      0t0  TCP localhost:sun-sr-https->localhost:33206 (ESTABLISHED)

[khuedoan]

Could you get the output of cat /etc/rancher/k3s/config.yaml on both nodes?

Side note: you'll need to add another master node (3 masters) or just use 1 master

HA embedded etcd cluster must be comprised of an odd number of server nodes for etcd to maintain quorum. For a cluster with n servers, quorum is (n/2)+1. For any odd-sized cluster, adding one node will always increase the number of nodes necessary for quorum. Although adding a node to an odd-sized cluster appears better since there are more machines, the fault tolerance is worse since exactly the same number of nodes may fail without losing quorum but there are more nodes that can fail.

https://docs.k3s.io/datastore/ha-embedded

[danghung-dev]

Here is output of config.yaml for both

Side note: you'll need to add another master node (3 masters) or just use 1 master

@khuedoan thanks for noting.

[khuedoan]

That appears correct, just to double check could you confirm that you are using the provided PXE boot setup? The node host name localhost.localdomain looks weird, it's supposed to be defined by the automatic kickstart config based on the name specified in the Ansible inventory.