Courier Fetch: Bad Gateway

[pip00]

I been getting this error lately as soon as fire up Kibana. Any ideas why this is happening ?
As soon as I hit the Go Back button my queries would populate.

[epixa]

This will happen whenever Kibana receives a 502 error from Elasticsearch. Do you see any errors in your Elasticsearch logs?

[pip00]

Here are my es logs

[2016-06-15 11:32:38,512][DEBUG][action.admin.cluster.node.stats] [elastic-kibana] failed to execute on node [Cr1WzqKXTAmtVy15adTRbg]
ReceiveTimeoutTransportException[[elasticsearch-1][192.168.1.102:9300][cluster:monitor/nodes/stats[n]] request_id [189677] timed out after [15099ms]]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-06-15 11:32:38,899][DEBUG][action.admin.cluster.node.stats] [elastic-kibana] failed to execute on node [CIqJQI_SQlK0HbP4naWhkA]
ReceiveTimeoutTransportException[[elasticsearch-2][192.168.1.103:9300][cluster:monitor/nodes/stats[n]] request_id [189678] timed out after [15499ms]]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-06-15 11:32:59,304][WARN ][transport ] [elastic-kibana] Received response for a request that has timed out, sent [35904ms] ago, timed out [20805ms] ago, action [cluster:monitor/nodes/stats[n]], node [{elasticsearch-1}{Cr1WzqKXTAmtVy15adTRbg}{192.168.1.102}{192.168.1.102:9300}{master=false}], id [189677]
[2016-06-15 11:33:10,304][WARN ][transport ] [elastic-kibana] Received response for a request that has timed out, sent [46904ms] ago, timed out [31405ms] ago, action [cluster:monitor/nodes/stats[n]], node [{elasticsearch-2}{CIqJQI_SQlK0HbP4naWhkA}{192.168.1.103}{192.168.1.103:9300}{master=false}], id [189678]

[epixa]

It looks like one of your Elasticsearch nodes is timing out on some requests. I'd recommend asking on the Elasticsearch forums or IRC channel, or opening an issue on https://github.com/elastic/elasticsearch

[jasontedor]

This will happen whenever Kibana receives a 502 error from Elasticsearch.

Elasticsearch does not send 502s.

[epixa]

@jasontedor Any idea what's going on in those logs pasted above? It looks like ES is encountering a timeout, and it would be helpful to know what the behavior should be in that scenario, since it seems like the Kibana proxy isn't handling it correctly and thus just triggers a 502 (my best guess).

[jasontedor]

It looks like ES is encountering a timeout

@epixa Maybe the node is GCing or otherwise starved of resources, I don't know. Those are some massively long timeouts. I don't see an Elasticsearch bug here though, perhaps just a misconfiguration or under-provisioning of resources.

it seems like the Kibana proxy isn't handling it correctly and thus just triggers a 502 (my best guess).

That's what I think too.

[epixa]

@jasontedor Based on the little we know from those logs, do you have any idea what behavior the consumer would encounter with ES when those GC issues happen? I'd like to get Kibana to handle this better so it's at least not obfuscated by an erroneous 502, but I don't know the behavior Kibana is encountering in the first place. Perhaps there's just not enough to go on at this point.

[jasontedor]

Perhaps there's just not enough to go on at this point.

@epixa It's not clear to me exactly what is happening here from the client perspective. This is because these log messages are from timing out while collecting node-level responses during a node stats request. Yet, even when that happens, Elasticsearch returns a status code 200 with the response body containing the node-level exceptions, if any.

[epixa]

Alright, thanks.

[Bargs]

Is there a proxy between Kibana and ES?