Log flood when node down

[Soulou]

After setting up a cluster ≥ 3 nodes, if one of these node is down, the leader is completely flooded by log messages (~20-30/sec)

[etcd] Feb 23 16:31:02.232 INFO      | node2: warning: heartbeat timed out: 'node1'

I don't think these messages are useful, at least the frequency should be different. How long does it need for a cluster to consider that a node is completely down/not part of the cluster anymore?

[xiang90]

@Soulou We do not remove a node automatically for now. But I do think we should clean up the logs.
For the heartbeat probing, we can do back-off too.

[Soulou]

Yes because it can be annoying, hundreds of MB per day, when a node is considered as down, it's too much.

So far, the only way to remove a node is to rebuild the cluster without it? It's a bit heavy oO

[xiang90]

@Soulou You can remove it by sent HTTP DELETE request to http://leader:7001/remove/[nodename]. We do not recommend this since there is a race issue we need to fix.

[philips]

We need to add exponential back off on this log entry.
On Feb 23, 2014 9:01 AM, "Xiang Li" notifications@github.com wrote:

@Soulou https://github.com/Soulou You can remove it by sent delete
request to http://leader:7001/remove/[nodename]. We do not recommend this
since there is a race issue we need to fix.

Reply to this email directly or view it on GitHubhttps://github.com//issues/595#issuecomment-35836425
.

[Soulou]

Great, thank you for these pieces of information. I'll keep looking for what will happen!

[Asmod4n]

Any news on this? Was planning to use a Laptop as a cluster node.

[yichengq]

fixed in #836.