Monitoring API for Logstash Forwarder

[kiranos]

Hi I'm still testing this and would really like to have some sort of way to know and easily setup a check with icinga/nagios if logstash-forwarder is shipping logs or if it somehow has halted.

I dont know how to solve it, there could be a threshold in the config file which says to generete an error in a logfile if no logs has been processed under this threshold or some other way for a third party script to check when the last time was that logstash-forwarder processed a logfile.

Might be good to brainstorm if this feature seems feasible, I would love to have some sort of security by knowing that I will get an alarm if the logs stops to pour in.

Thanks

[avleen]

Some kind of stats interface to logstash-forwarder would be great.
A TCP port - you connect, it sends back stats:

    <filename> <current position> <current size>
    <filename> <current position> <current size>
    <filename> <current position> <current size>

etc.

[driskell]

The state file, .logstash-forwarder, contains that info in json. I know there are json plugins for nagios but not sure it could handle the "check increasing"

At the moment i just use a check_file_age plugin on .logstash-forwarder to make sure it don't get more than 60m old and that works a treat.

Jason

[jordansissel]

My recommendation is to do behavioral tests. Want to know if logs are going
through? Ask the final destination if logs are going in!

Perhaps a heartbeat log that you measure latency on? This is how I have
done it before, if that helps.

-Jordan

On Sunday, April 27, 2014, driskell notifications@github.com wrote:

The state file, .logstash-forwarder, contains that info in json. I know
there are json plugins for nagios but not sure it could handle the "check
increasing"

At the moment i just use a check_file_age plugin on .logstash-forwarder to
make sure it don't get more than 60m old and that works a treat.

Jason

Reply to this email directly or view it on GitHubhttps://github.com//issues/183#issuecomment-41491372
.

[avleen]

I have logstash add an extra date field called index_timestamp, which is
the timestamp at which logstash processed the event.

This can then be measured with a query for each host you're expecting logs
from.
On Apr 27, 2014 3:23 PM, "Jordan Sissel" notifications@github.com wrote:

My recommendation is to do behavioral tests. Want to know if logs are going
through? Ask the final destination if logs are going in!

Perhaps a heartbeat log that you measure latency on? This is how I have
done it before, if that helps.

-Jordan

On Sunday, April 27, 2014, driskell notifications@github.com wrote:

The state file, .logstash-forwarder, contains that info in json. I know
there are json plugins for nagios but not sure it could handle the "check
increasing"

At the moment i just use a check_file_age plugin on .logstash-forwarder
to
make sure it don't get more than 60m old and that works a treat.

Jason

Reply to this email directly or view it on GitHub<
#183 (comment)

.

Reply to this email directly or view it on GitHubhttps://github.com//issues/183#issuecomment-41506107
.

[cognusion]

FWIW: Since all (most) of my logstash'd logs are going into ElasticSearch, I have Nagios running a simple query to ES to determine the most recent items grouped by sending host, and if the date falls out of spec (1 minute) it trips.

[chjohnst]

I know syslog-ng and rsyslog support putting a mark ever X seconds, I have used this in the past for monitoring to ensure the system is actually logging. You can perhaps us that as a monitor by checking ES for it.

[jordansissel]

lsf is feature-frozen until the experiment described here is concluded. In general, I agree with @avleen and others: there needs to be an API to ask lsf how it's doing.

In the meantime, using a logfile as a heartbeat can help.

[jordansissel]

We will add an API for filebeat for monitoring

[elvarb]

Then think about how well this goes together with the logstash http poller input :-)

[jordansissel]

Thanks for helping make logstash-forwarder better!

Logstash-forwarder is going away and is replaced by filebeat and its friend, libbeat. If this is still an issue, would you mind opening a ticket there?