etcd cluster is unhealthy (Rocky Linux 8)

[Muhtarhan27]

TASK [etcd : Enable and start etcd service] *********************************************************************************************************************************************************************
ok: [10.8.72.99]
ok: [10.8.72.98]
ok: [10.8.72.100]

TASK [etcd : Wait for port 2379 to become open on the host] *****************************************************************************************************************************************************
ok: [10.8.72.99]
ok: [10.8.72.98]
ok: [10.8.72.100]
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (10 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (10 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (10 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (9 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (9 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (9 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (8 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (8 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (8 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (7 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (7 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (7 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (6 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (6 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (6 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (5 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (5 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (5 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (4 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (4 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (4 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (3 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (3 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (3 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (2 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (2 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (2 retries left).
FAILED - RETRYING: [10.8.72.99]: Wait until the etcd cluster is healthy (1 retries left).
FAILED - RETRYING: [10.8.72.98]: Wait until the etcd cluster is healthy (1 retries left).
FAILED - RETRYING: [10.8.72.100]: Wait until the etcd cluster is healthy (1 retries left).

TASK [etcd : Wait until the etcd cluster is healthy] ************************************************************************************************************************************************************
fatal: [10.8.72.99]: FAILED! => {"attempts": 10, "changed": false, "cmd": ["/usr/local/bin/etcdctl", "endpoint", "health", "--endpoints=http://10.8.72.99:2379"], "delta": "0:00:05.015246", "end": "2023-10-07 09:01:01.460159", "msg": "non-zero return code", "rc": 1, "start": "2023-10-07 09:00:56.444913", "stderr": "{\"level\":\"warn\",\"ts\":\"2023-10-07T09:01:01.459411+0600\",\"logger\":\"client\",\"caller\":\"v3@v3.5.9/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc0000e4a80/10.8.72.99:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"}\nhttp://10.8.72.99:2379 is unhealthy: failed to commit proposal: context deadline exceeded\nError: unhealthy cluster", "stderr_lines": ["{\"level\":\"warn\",\"ts\":\"2023-10-07T09:01:01.459411+0600\",\"logger\":\"client\",\"caller\":\"v3@v3.5.9/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc0000e4a80/10.8.72.99:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"}", "http://10.8.72.99:2379 is unhealthy: failed to commit proposal: context deadline exceeded", "Error: unhealthy cluster"], "stdout": "", "stdout_lines": []}
fatal: [10.8.72.98]: FAILED! => {"attempts": 10, "changed": false, "cmd": ["/usr/local/bin/etcdctl", "endpoint", "health", "--endpoints=http://10.8.72.98:2379"], "delta": "0:00:05.016780", "end": "2023-10-07 09:01:01.951667", "msg": "non-zero return code", "rc": 1, "start": "2023-10-07 09:00:56.934887", "stderr": "{\"level\":\"warn\",\"ts\":\"2023-10-07T09:01:01.950982+0600\",\"logger\":\"client\",\"caller\":\"v3@v3.5.9/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc000314a80/10.8.72.98:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"}\nhttp://10.8.72.98:2379 is unhealthy: failed to commit proposal: context deadline exceeded\nError: unhealthy cluster", "stderr_lines": ["{\"level\":\"warn\",\"ts\":\"2023-10-07T09:01:01.950982+0600\",\"logger\":\"client\",\"caller\":\"v3@v3.5.9/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc000314a80/10.8.72.98:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"}", "http://10.8.72.98:2379 is unhealthy: failed to commit proposal: context deadline exceeded", "Error: unhealthy cluster"], "stdout": "", "stdout_lines": []}
fatal: [10.8.72.100]: FAILED! => {"attempts": 10, "changed": false, "cmd": ["/usr/local/bin/etcdctl", "endpoint", "health", "--endpoints=http://10.8.72.100:2379"], "delta": "0:00:05.018616", "end": "2023-10-07 09:00:58.474889", "msg": "non-zero return code", "rc": 1, "start": "2023-10-07 09:00:53.456273", "stderr": "{\"level\":\"warn\",\"ts\":\"2023-10-07T09:00:58.473938+0600\",\"logger\":\"client\",\"caller\":\"v3@v3.5.9/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc0003b6a80/10.8.72.100:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"}\nhttp://10.8.72.100:2379 is unhealthy: failed to commit proposal: context deadline exceeded\nError: unhealthy cluster", "stderr_lines": ["{\"level\":\"warn\",\"ts\":\"2023-10-07T09:00:58.473938+0600\",\"logger\":\"client\",\"caller\":\"v3@v3.5.9/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc0003b6a80/10.8.72.100:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"}", "http://10.8.72.100:2379 is unhealthy: failed to commit proposal: context deadline exceeded", "Error: unhealthy cluster"], "stdout": "", "stdout_lines": []}

[Muhtarhan27]

SELinux disabled, error still appears

[Muhtarhan27]

or from where I run the playbook should also be on Rocky Linux

[vitabaks]

Hi @Muhtarhan27

"context deadline exceeded": This error suggests that a request to the etcd cluster timed out. It's possible that network issues prevent communication.

1. Network Connectivity: Make sure there are no network problems. Try pinging nodes between each other to make sure there is a connection. And try telnet for port 2379 between nodes.

2. etcd Logs: View the etcd logs on each of the nodes (sudo journalctl -u etcd -n 100). This may provide additional information about the causes of the problem.

3. Execute the commands etcdctl member list , ETCDCTL_API=3 etcdctl endpoint health --cluster and provide the execution result

[redy01]

It is probably firewall, try to turn off firewalld.service
or open ports:
for _port in {2379,2380,5432,8008}; do firewall-cmd --add-port=${_port}/tcp --permanent; done && firewall-cmd --reload
on both hosts as root