Skip to content

Latest commit

 

History

History
141 lines (89 loc) · 6.52 KB

README.md

File metadata and controls

141 lines (89 loc) · 6.52 KB
Raw

Various documentation and example apps for sending logs to Graylog via syslog.

Note that in most situations it would be preferable to send logs to Graylog as Graylog Extended Log Format (GELF) messages; this repository is only for when the logs must be sent as syslog messsages.

Testing

See logs sent by applications

If you want to see the log messages that are being sent, a quick way is to use netcat to listen on a local port and dump the logs to the console; this is also helpful for troubleshooting if, for example, the timestamp in the log is improperly formatted (which may make it difficult to find the log in Graylog):

nc -klu 5140

Or for TCP:

nc -kl 5140

Alternatively, see below for setting up a test Graylog server.

Setting up a local Graylog server for testing

  1. Clone this repo: https://github.com/Graylog2/docker-compose/

  2. Create .env file

    cd open-core
cp .env.example .env

  3. Set GRAYLOG_PASSWORD_SECRET and GRAYLOG_ROOT_PASSWORD_SHA2 in the .env file according to the instructions in the file

  4. Set the time zone for the admin user

    This will allow you to do searches in Graylog in your local timezone as the admin user

    1. Add this line to the .env file you created:

      GRAYLOG_ROOT_TIMEZONE: '...'

      Replace ... with a valid timezone from http://joda-time.sourceforge.net/timezones.html; copy the value from Canonical ID, e.g. America/New_York

  5. Start the server

    docker-compose up

  6. Login using a user name of admin and whatever value you used when you set GRAYLOG_ROOT_PASSWORD_SHA2

  7. Add a new syslog input

    1. System / Inputs (click the hamburger menu if you don't see it) > Inputs

    2. Select input > Syslog UDP > Launch new input

    3. Set the Port to 5140 (this port is already pre-configured in the docker-compose file)

    4. Check Store full message (this will help troubleshooting)

    5. Save

  8. Send some syslog logs to localhost on port 5140/udp (see one of the provided test apps)

  9. Go to Search to see any logs you send to the server

    • You can see the full message under full_message as needed

Graylog and syslog

TCP vs UDP

TCP offers a lot of advantages over UDP, however there's a catch when sending syslog messages to Graylog over TCP: by default, the Graylog TCP input will split logs on the newline ('\n') character, splitting any multi-line messages into separate logs.

This is controlled by the use_null_delimiter setting in the Syslog TCP input. When set to false (the default), Graylog will split messages on the newline character. When set to true, Graylog will split messages on the null character (\0). However, in my testing, setting use_null_delimiter: true caused logs that weren't terminated with a null character to not show in Graylog at all (regardless of the number of lines in the log).

Here are a few possible options to deal with this:

  • Send logs to the Syslog UDP input

  • Or set use_null_delimiter: true in the Syslog TCP input and make sure they're null-terminated

    • e.g. Winston can do this by using this setting in the syslog transport:

      eol: '\0',

  • Alternatively, set use_null_delimiter: false in the Syslog TCP input and use a different separator between lines. Unfortunately, I haven't found a separator that shows up as proper newlines in the Graylog UI as it does when sending over UDP. Separators such as \r and \u2028 get converted to spaces.

BSD (RFC 3164) vs RFC 5424

Some syslog clients may give the option of sending logs formatted as BSD (RFC 3164) or RFC 5424 messages. Always prefer RFC 5424 when possible, because it has the following advantages:

  • More accurate timestamps, including milliseconds and timezone
  • More fields, including application name, process ID, message ID, structured data (custom fields)

The biggest painpoint when sending BSD-formatted messages to Graylog is the timestamp:

  • Because there's no time zone, Graylog will always assume logs are sent with a UTC time zone
  • Because there are no milliseconds, logs will likely be out of order

Here is an example of a BSD-formatted syslog message (generated using log4j2 without format):

<131>Sep 27 11:33:14 localhost Test error message, without stack trace

Here is an example of the same message formatted according to RFC 5424 (generated using log4j2 with format="RFC5424"):

<131>1 2021-09-27T11:33:14.564-04:00 localhost testlog4j2 56903 - [mdc@18060 category="App" exception="" priority="ERROR" thread="main"] Test error message, without stack trace

How Graylog parses syslog messages

Given the sample RFC 5424 above, here is how Graylog will parse it:

  • The first part (in angle brackets) is the syslog facility, which gets set to these fields in Graylog: facility, facility_num, and level
  • After that should be a 1, which tells Graylog that the message adheres to syslog protocol version 1 (RFC 5424)
  • The next field is the timestamp, which Graylog parses providing that it's properly formatted
  • Next field is the hostname, set to the source field in Graylog
  • Next field is the application name, set to the application_name field in Graylog
  • Next field is the process ID, set to the process_id field in Graylog
  • Next field is the message ID, which above is empty (- is an empty value in syslog; see NILVALUE)
  • If structured data is present, each item will each get parsed as fields in Graylog
  • Everything else gets set to the message field in Graylog