Slow cluster startup with zen discovery.

[ghost]

When a cluster with a large number of nodes starts up, the joining of the nodes becomes slow, as the cluster state update is blocking. The master node adds the nodes one by one and waits after every join (zen-disco-receive) for the updated cluster state to be distributed.

This issue occurs in elasticsearch version 1.0 and is related to #3736 introducing the wait during the processing of the cluster state updates.

When setting discovery.zen.publish_timeout:0 the startup of the cluster is as fast as before, as the master node is not waiting for the individual updates to be acked by the client nodes.

A solution to the problem might be that the updates of the cluster state would be processed on the master and only distributed after all have been applied. Or that the master would not wait for the state to be acked by the client nodes during startup.

[bluelu]

Please note that this happens on an empty cluster with no indexes at all and slow means > 30 minutes

More details are here:
https://groups.google.com/forum/#!topic/elasticsearch/zxHeqW932f8

[dakrone]

@miccon @bluelu this should be helped by #6342 , are you still seeing this issue with Elasticsearch 1.2.1?

[bluelu]

Thanks. We will have time at the end of next week to test this with a seperate installation of elasticsearch. We will let you know then.

As this is a production cluster, we can not upgrade right away and easily (also because of #6372 (delaying the start) and #6295)

[kimchy]

@bluelu I suspect that this might happen because the join request was being to aggressive in its timeouts to wait for the cluster to ack the join, and got into continuous joining state. Now, the timeout is bigger (#6342), so I think it will help. If not, I would love to help in figuring out what takes too long.

[bluelu]

Thanks. We will observe both when we test and will let you know. Still the issue with the traffic remains (#6295) (e.g. when you have more than 500 nodes and a few thousand indexes...). Then also larger installations will run quite nicely :-)

[avleen]

@dakrone We see this with 1.2.1.
I just set discovery.zen.publish_timeout:0 and it's going much faster so far. Just restarted our cluster and 28 nodes joined in about 2 minutes rather than 10+ minutes. It does still slow down once it hit ~20-ish nodes in the cluster (the first ~20 joined really fast, now we're down to about 1 every 15 seconds).

Will test 1.3.0 when it lands.

[clintongormley]

@avleen have you had a chance to test this on 1.3?

[avleen]

Hey Clinton! We haven't upgraded to 1.3 yet. We plan on doing that in about
a week. I'll fill you in then!

On Fri, Aug 1, 2014 at 5:33 AM, Clinton Gormley notifications@github.com
wrote:

@avleen https://github.com/avleen have you had a chance to test this on
1.3?

Reply to this email directly or view it on GitHub
#5232 (comment)
.

[bluelu]

Unfortunately we haven't yet have time to update our cluster and test with the new version. We also tried this on a smaller cluster, and didn't have the issue that it was starting slowly, so we must test it with more servers.
@avleen, does the new version work for you? does it resolve the issue?

[avleen]

We upgraded to 1.3.
We found the first ~20-25 nodes join the cluster really fast (maybe less
than 2 minutes).
After that it starts to get extremely slow again.

It starts to take 15 or 30 seconds for each node to join again. Not 15 to
30 seconds, 15 or 30 seconds.
The nodes seemed to join one at a time, and close to those intervals.
In the end a full cluster restart still takes us at least 20 minutes.
On Aug 12, 2014 6:03 AM, "Thibaut" notifications@github.com wrote:

Unfortunately we haven't yet have time to update our cluster and test with
the new version. We also tried this on a smaller cluster, and didn't have
the issue that it was starting slowly, so we must test it with more servers.
@avleen https://github.com/avleen, does the new version work for you?
does it resolve the issue?

Reply to this email directly or view it on GitHub
#5232 (comment)
.

[avleen]

Further update.
Just did another full cluster restart.
It took 15 minutes for all 64 instances of elasticsearch to finish joining
the cluster.
After the server joins started slowing down, there is a very exact pause of
15 or 30 seconds between nodes joining.

Is there a 15s timeout here which might be coming into play?

On Tue, Aug 12, 2014 at 10:07 AM, Avleen Vig avleen@gmail.com wrote:

We upgraded to 1.3.
We found the first ~20-25 nodes join the cluster really fast (maybe less
than 2 minutes).
After that it starts to get extremely slow again.

It starts to take 15 or 30 seconds for each node to join again. Not 15 to
30 seconds, 15 or 30 seconds.
The nodes seemed to join one at a time, and close to those intervals.
In the end a full cluster restart still takes us at least 20 minutes.
On Aug 12, 2014 6:03 AM, "Thibaut" notifications@github.com wrote:

Unfortunately we haven't yet have time to update our cluster and test
with the new version. We also tried this on a smaller cluster, and didn't
have the issue that it was starting slowly, so we must test it with more
servers.
@avleen https://github.com/avleen, does the new version work for you?
does it resolve the issue?

Reply to this email directly or view it on GitHub
#5232 (comment)
.

[clintongormley]

/cc @kimchy

[kimchy]

improve_zen branch just landed in master/1.x, this includes a lot of improvements when it comes to forming a cluster. We ran 100s nodes test (with not data in the one I refer to, just to see how quickly a cluster is formed) and the results were very good (less than 30s to form 100s nodes clusters).

Even with improve_zen, the logic is still similar though, when a node joins, a full cluster state cycle is needed (more lightweight, batched, but still). Maybe the 30s come from the publish timeout, thats the one that can explain it, but I then don't understand why one of the nodes that are part of the cluster not answering in the proper time. Maybe next time you can set discovery.zen.publish logging to DEBUG, and see if we see messages like "timed out waiting for all nodes to process published".

[clintongormley]

@miccon @avleen just pinging to hear about your experiences with cluster startup on v1.4.

[bluelu]

I'm working together with @miccon. Except for the slow allocation of unallocated shards (#6372) , the joining of nodes is now very fast. This issue can be closed.

[clintongormley]

thanks for letting us know @bluelu

[avleen]

This happened a while ago but we just upgraded to 1.4.2 and found that joining is pretty much instant now. Thanks everyone!

[bleskes]

@avleen happy to hear.