-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem with increasing connections while getting metrics from Elasticsearch in client mode #452
Comments
My guess is this is caused by not calling
I had the similar issue in Python with requests. |
thanks for the report, if @zstyblik is correct then that is the issue, we should be always be closing the response body. |
@sparrc - I don't know architecture of Telegraf, so it's a "guess" based on reading through the Elasticsearch plugin. But I doubt such functionality, read body of response and close connection, is implemented somewhere inside. Anyway, that part is copy-paste from documentation of |
I think you're right, we should be closing this: https://github.com/influxdb/telegraf/blob/master/plugins/elasticsearch/elasticsearch.go#L201 |
@sparrc indeed :) It could be something like ...
? And import |
why does it require the |
Because you're supposed to read the body of the response even if you don't care about it. Whether it's just a good practice or optional, that I don't know. I'm always doing it regardless :) |
@jipperinbham do it on master, the changes I'm making are on the 0.3.0 branch and will remain there until sometime in January |
@zstyblik re: whether or not the body should be read, from what I've read you are right in that the underlying transport will not reuse the connection if In this situation, I tend to think it's ok to not reuse the underlying connection because we received something other than thoughts? |
@jipperinbham sorry, but I don't know that well. I'd expect this to be related to connection pool rather than connection being reused, resp. another request being sent over the exactly the same and already open connection and freeing up that connection early. shrug |
Hello.
I'm using Telegraf (ver. 0.2.4) to monitor few Elasticsearch (ver. 1.7.3) instances. One of them is configured in client mode (has no data, essentially it's a load balancer). When I'm starting Telegraf, metrics are gathered and send to Influx database. But Telegraf process keeps opening network connections to Elasticsearch, never closing them. It keeps increasing until limit of open files is reached.
22994 is PID of Telegraf process:
In 2 hours number of open files is reaching limit, causing:
Then Telegraf restarts (or clear active connections) or sometimes even stops getting metrics.
When Elasticsearch plugin is not loaded, everything is ok.
I've tried all possible Elasticsearch plugin configuration options (local = true/false), without any positive results.
Number of open files is increasing in Elasticsearch process, too, so I'm sure it's connected.
Process file descriptors (output shortened for brevity):
I can help with further testing, for now my solution is to remove Elasticsearch plugin from Telegraf.
The text was updated successfully, but these errors were encountered: