Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcdctl false-negative "unhealthy cluster" when unauthenticated #13692

Closed
udf2457 opened this issue Feb 12, 2022 · 2 comments
Closed

etcdctl false-negative "unhealthy cluster" when unauthenticated #13692

udf2457 opened this issue Feb 12, 2022 · 2 comments
Labels

Comments

@udf2457
Copy link

udf2457 commented Feb 12, 2022

What happened?

etcdctl --write-out=table --endpoints=https://foo.example.com:2379 endpoint health gives a false-negative cluster health indication:

+------------------------------+--------+-------------+--------------------------------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+------------------------------+--------+-------------+--------------------------------+
| https://etc2.example.com:2379 | false | 47.552071ms | etcdserver: user name is empty |
| https://etc1.example.com:2379 | false | 48.137024ms | etcdserver: user name is empty |
| https://etc3.example.com:2379 | false | 48.444852ms | etcdserver: user name is empty |
+------------------------------+--------+-------------+--------------------------------+
Error: unhealthy cluster

This is unhelpful and misleading. If ectdctl was unable to determine the health of a cluster due to failed or missing authentication, it should not fallback to delivering an "unhealthy cluster" (or "false" in the node health).

What did you expect to happen?

Not report "false" on individual node health.
Not conclude "Error: unhealthy cluster"

How can we reproduce it (as minimally and precisely as possible)?

See above

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.2
Git SHA: 99018a77b
Go Version: go1.16.3
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.5.2
API version: 3.5

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

@udf2457 udf2457 changed the title etcdctl false-negative "unhealthy cluster" when unauthicated etcdctl false-negative "unhealthy cluster" when unauthenticated Feb 12, 2022
@serathius serathius mentioned this issue Apr 6, 2022
28 tasks
@spzala
Copy link
Member

spzala commented Apr 8, 2022

An endpoint is considered unhealthy when it cannot participate in consensus with the rest of the cluster. Inaccessible endpoints, for whatever reason, is considered unhealthy in general and a related log message is provide per it's documented https://github.com/etcd-io/etcd/tree/main/etcdctl#output-15.
Another example of unreachable endpoint,
+-----------------------------+--------+-------------+---------------------------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+-----------------------------+--------+-------------+---------------------------+
| http://foo.example.com:2379 | false | 5.00937926s | context deadline exceeded |
+-----------------------------+--------+-------------+---------------------------+
Error: unhealthy cluster

The cause of error is provided clearly, whether it's a missing credential, connection etc. Closing this issue as this logging behavior is documented and expected. Please reopen if needed. Thanks!
cc @serathius @ahrtr @ptabor

@spzala spzala closed this as completed Apr 8, 2022
@ahrtr
Copy link
Member

ahrtr commented Apr 8, 2022

Agreed or at least it's a minor one which doesn't need to be fixed at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

3 participants