2 cluster versions

[abezard]

I setup a new etcd cluster and for some reason each node is reporting 2 distinctetcd_cluster_version (v3.0 and v3.4) at the same time even though each node is reporting 1 etcd_server_version.

etcd_cluster_version:

3.4.0 | 10.10.10.33:2479 | Up
3.0.0 | 10.10.10.33:2479 | Up
3.4.0 | 10.10.10.92:2479 | Up
3.0.0 | 10.10.10.92:2479 | Up
3.4.0 | 10.10.10.15:2479 | Up
3.0.0 | 10.10.10.15:2479 | Up
3.4.0 | 10.10.10.241:2479 | Up
3.0.0 | 10.10.10.241:2479 | Up
3.4.0 | 10.10.10.190:2479 | Up
3.0.0 | 10.10.10.190:2479 | Up
[...]

etcd_server_version:

10.10.10.241:2479 | 3.4.0 | Up
10.10.10.190:2479 | 3.4.0 | Up
10.10.10.15:2479 | 3.4.0 | Up
10.10.10.247:2479 | 3.4.0 | Up
10.10.10.33:2479 | 3.4.0 | Up
10.10.10.92:2479 | 3.4.0 | Up
10.10.10.167:2479 | 3.4.0 | Up

I double checked on some hosts: it's not an issue with the metrics acquisition:

root@etcd-dasiuod:~# curl http://127.0.0.1:2479/metrics
# HELP etcd_cluster_version Which version is running. 1 for 'cluster_version' label with current cluster version
# TYPE etcd_cluster_version gauge
etcd_cluster_version{cluster_version="3.0.0"} 1
etcd_cluster_version{cluster_version="3.4.0"} 1
[...]

Note that this is a brand new cluster, not an upgrade. We have other 3.4 etcd clusters that don't behave like that. Any idea? Thanks!

[jaydhulia]

After investigation it seems that this is being caused by this function:
https://github.com/etcd-io/etcd/blob/release-3.4/etcdserver/api/membership/cluster.go#L541 .
When the cluster is initially started, it starts off at 3.0:
etcdserver/membership: set the initial cluster version to 3.0
And as a result the metric etcd_cluster_version{cluster_version="3.0.0"} is set to 1. But then, the cluster version is updated to 3.4:
etcdserver/membership: updated the cluster version from 3.0 to 3.4
So as a result the metric etcd_cluster_version{cluster_version="3.4.0"} is also set to 1. But in this logic, etcd_cluster_version{cluster_version="3.0.0"} is never set back to 0, so the metrics (etcd_cluster_version} show it as running 2 versions.