etcd_debugging_mvcc_keys_total inconsistency

[mwf]

This issue is connected with #8387

After the cluster stabilized we have inconsistency in "etcd_debugging_mvcc_keys_total" metric between hosts:

curl http://<host3>:2379/metrics | grep etcd_debugging_mvcc_keys_total  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3 35906    3  1328    0     0   1708      0  0:00:21 --:--:--  0:00:21  1706# HELP etcd_debugging_mvcc_keys_total Total number of keys.
# TYPE etcd_debugging_mvcc_keys_total gauge
etcd_debugging_mvcc_keys_total 15007
100 35906  100 35906    0     0  30216      0  0:00:01  0:00:01 --:--:-- 30223

One host reports it has 15007 keys.

Though range request ETCDCTL_API=3 etcdctl --endpoints="http://<host3>:2379" get --prefix --keys-only --limit=1 -w=json "" tells it has only "count":7671 keys, which is the same as other hosts report in metrics.

Maybe this issue was already fixed in newer etcd server versions? I just decided to report.

[mwf]

Maybe it's somehow connected with my old issue #8146, which is fixed now.

[heyitsanthony]

@mwf can this be reproduced with master?

[mwf]

Actually, I did not find a way to reproduce it, the issue is too strange to be true, but it's a fact :)

I'll try, if I manage to reproduce I will let you know. But if you could take a look at code producing this metric value and maybe find a bug - that would be great. Just as you did in #8146.