Fluentd stopped sending data to ES for somewhile. #525
Comments
|
Could you provide your Fluentd docker log?
The above settings is very dangerous. |
|
Hi @cosmo0920 , |
|
Umm..., could you share fluentd error log between from 2019-01-10 2:00 to 2019-01-10 11:00 ? Shared log is booting log. It just says that Fluentd was launched normally. |
|
@cosmo0920 I find something like this |
|
It seems that ES plugin cannot push events due to |
|
@cosmo0920 My ES is setup with AWS EC2, and the networking should be fine, without disconnect or DNS issue. |
|
@cosmo0920 Here is more logs from ES and actually, I have two nodes/host with same configuration that collect logs from my application server, do you think that should be a concern for this issue? If it is true, is there any other way in the fluentd configuration to distinguish the logs collected from which node? like hostname or host ip as the metadata? |
It should check Docker networking. The above issue is also only occurred within docker not bare metal environment.
|
|
@cosmo0920 Thanks for you for your advice. But I have to use the fluentd in docker, and it looks like the issue still there. The services in my docker is always running well. It probably not the docker networking issue. |
|
Met similar issue, but I have the fluend deployed as a daemonset under And I can confirm ES is running well all the time, since fluentd is only one of my logging sources, and other sources can work well and showing logs correctly in ES. |
|
@emmayang Same issue on my |
|
Hmmm..., could you try |
|
I'm also seeing this same issue when running fluentd with ES plugin in Kubernetes. I tried both backends and typhoeus didn't work at all, while the default backend would work on initial connection (fresh deploy) and then stop sending data almost immediately. EDIT: I believe my issues were not from the ES plugin but performance tuning that I needed to do on Fluentd. |
|
I have similar problems.I also have huge number for warnings as below: |
|
Same problem here. I'm using fluentd-kubernetes-daemonset. Only solution is to restart the pod. But this isnt' an acceptable solution, |
|
Set |
@cosmo0920, I'm afraid so...In my case, the hits reach 100000+ then the issue happens. In fluentd, here's error info: I'll try 'reconnect_on_error true' and give feedback. |
Maybe this is the solution for me. Set reload_connection to false, now it's working for about 18h without troubles. I will monitor it for the next few hours / days. |
|
@bidiudiu @ChSch3000 Thank you for your issue confirmations and clarifications! fluentd-kubernates-daemonset provides the following environment variable:
This should be specified:
|
|
I've added FAQ for this situation. #564 Any lack of information to solve this issue? |
|
Thanks @cosmo0920. I add settings below and it works fine: |
OK. Thanks for confirming, @bidiudiu ! |
This is reported in uken/fluent-plugin-elasticsearch#525. Invalid sniffer information is obtained by default, but we can avoid the following configuration: ```aconf reload_connections false reconnect_on_error true reload_on_failure true ``` To specify reload_on_failure on fluentd-kubernetes-daemonset, we should introduce a new envver to specify it. Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
…e sniffering This is reported in uken/fluent-plugin-elasticsearch#525. Invalid sniffer information is obtained by default, but we can avoid the following configuration: ```aconf reload_connections false reconnect_on_error true reload_on_failure true ``` To specify reload_on_failure on fluentd-kubernetes-daemonset, we should introduce a new envver to specify it. Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
fluent-plugin-elasticsearch reloads connection after 10000 requests. (Not correspond to events counts because ES plugin uses bulk API.) This functionality which is originated from elasticsearch-ruby gem is enabled by default. Sometimes this reloading functionality bothers users to send events with ES plugin. On k8s platform, users sometimes shall specify the following settings: ```aconf reload_connections false reconnect_on_error true reload_on_failure true ``` This is originally reported at uken/fluent-plugin-elasticsearch#525. On k8s, Fluentd sometimes handles flood of events. This is a pitfall to use fluent-plugin-elasticsearch on k8s. So, this parameter set should be default. Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
|
Can we change the default value of those settings for fluentd-kubernetes-daemonset ? I think everyone who uses fluentd-kubernetes-daemonset will encounter this issue easily ? |
|
@dogzzdogzz if you are using helm to install, eg. Part of my snippet looks like this, |
|
@dogzzdogzz The latest fluentd-kubernetes-daemonset includes the above settings by default. |
|
Tried using the exact same config as #525 (comment) but the issue still persists. Fluentd stops shipping logs to Elasticsearch after some time. |
|
@cosmo0920 |
|
You should add simple sniffer loading code and specify loaded simple sniffer class: |
did this work , to solve the "failed to flush the buffer" error , if so could you post the configuration , Thanks, |
|
Yes me too, I've loaded the sniffer class and it's still giving me that error. I'm using version 4.0.5, but I'm getting that error as soon as the fluentd pods restart, there's no grace period where this is succeeding at sending logs. |
|
Same issue here. Did anyone found a concrete solution? Also the sniffer_class solution doesn't work for me at all and throws an error. |
|
So I found the solution 4days ago and I've been testing it ever since. So after the change I made, my fluentd didn't stopped or crashed sending logs to elasticsearch. My solution was to change the buffer path in a way I saw in fluentd documentation.
instead of
This worked for me. |
@mokhos , please could you let us know the version of fluentd / fluentd-plugin-elasticsearch , you were using to test this configuration ? |
I have used below versions: |
|
Hi @cosmo0920, I am also facing the same issue , it will be helpful if you share your solution with me If i restarted my td-agent.service the logs are coming for sometime in the elasticsearch after 3-6 mins they are getting stopped automatically and no error is showing in td-agent logs. Here is my configuration: also tried. Please help !!!!! Note : The above configuration is not copy pasted ignore indentation |
It works for me |
Problem
I used the fluentd with your plugin to collect logs from docker containers and send to ES. It works at the very begining. But later, the ES unable to recieve the logs from fluentd. The ES is always running fine. And I find there is no indices of the new day(eg.
fluentd-20190110, only the old indice20190109exist) in the ES.However, if I restart my docker containers with fluentd, it can start sending logs to ES.
...
Steps to replicate
The fluentd config
Expected Behavior or What you need to ask
The fluentd should keep sending logs to ES.
Using Fluentd and ES plugin versions
Docker
fluentd --versionortd-agent --versionv1.3.2-1.0fluent-gem list,td-agent-gem listor your Gemfile.lock6.5.4
The text was updated successfully, but these errors were encountered: