[libbeat] Add Syslog parser and processor

[taylor-swanson]

What does this PR do?

• Add Syslog parser
• Add Syslog processor
• Add unit tests and benchmarks
• Add processor documentation

Why is it important?

This change allows us to detach syslog message parsing from a specific filebeat input. A new processor and parser have been added to libbeat, each providing the ability to parse RFC 3164 and RFC 5424-formatted syslog messages.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run unit tests in these packages:

libbeat/processors/syslog
libbeat/reader/syslog

Benchmark tests are also available.

Example config for a filebeat processor:

filebeat.inputs:
- type: udp
  host: "localhost:9000"
  processors:
    - syslog:
        field: message
        format: auto
        timezone: Local

Example config for a filebeat parser:

filebeat.inputs:
- type: filestream
  paths:
    - /tmp/syslog.txt
  parsers:
    - syslog:
        format: auto
        timezone: Local

Related issues

• Closes Create Syslog Processor #30139

[mergify]

This pull request does not have a backport label. Could you fix it @taylor-swanson? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-21T13:45:23.423+0000

• Duration: 81 min 47 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22523
Skipped	1940
Total	24463

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

Leaving some high-level comments. I did not dive into any of the parser.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b syslog-parser upstream/syslog-parser
git merge upstream/main
git push upstream syslog-parser

[andrewkroh]

👍

[andrewkroh]

LGTM. I recommend running golangci-lint before merging.

It might be worthwhile to explain the behavior of dates without years in the docs. Sometimes people ingesting logs from the previous year are surprised when those logs appear to be in the future (e.g. logs from Dec 31 ingested on Jan 1 get the wrong year).

[cmacknz]

Didn't do a thorough review but LGTM.

[taylor-swanson]

LGTM. I recommend running golangci-lint before merging.

It might be worthwhile to explain the behavior of dates without years in the docs. Sometimes people ingesting logs from the previous year are surprised when those logs appear to be in the future (e.g. logs from Dec 31 ingested on Jan 1 get the wrong year).

What are your thoughts on something like this:

func (m *message) setTimestampBSD(v string, ref time.Time) {
	if parsed, err := time.ParseInLocation(time.Stamp, v, ref.Location()); err == nil {
		year := ref.Year()
		if parsed.Month() == time.December && ref.Month() == time.January {
			year--
		}
		m.timestamp = parsed.AddDate(year, 0, 0)
	}
}

If the parsed month is December and the current month is now January, we decrement the year by 1. This gives a month on either side of the year boundary to make sure the year is correct. If we're outside of this window, all bets are off. Another benefit with this is the function now accepts a time.Time. We can write unit tests to explicitly test the year boundary and any other cases we can think of.

Either way, I'll document this situation and suggest that if logs are meant to be stored more long term and ingested at a later date, then it is critical that they switch to a timestamp that provides more information (ISO 8601/RFC 3339). Heck, the RFC even says this:

It has been found that some network administrators like to archive their syslog messages over long periods of time. [...] Implementers may wish to utilize the ISO 8601 [7] date and time formats if they want to include more explicit date and time information.

[andrewkroh]

What are your thoughts on something like this:

I would avoid the heuristic based solution (or at least make it optional behavior). There are a lot of edge cases. Logstash has some discussion on the topic in logstash-plugins/logstash-filter-date#51 (comment).

[taylor-swanson]

What are your thoughts on something like this:

I would avoid the heuristic based solution (or at least make it optional behavior). There are a lot of edge cases. Logstash has some discussion on the topic in logstash-plugins/logstash-filter-date#51 (comment).

That's fair. In that case, I think this is something that should be handled in a separate issue. The current implementation behaves just like the existing input, so behavior hasn't changed.

Having configuration for this that could handle several different scenarios would provide the better options for customers. What I proposed would not help a customer ingest logs older than a year, for example. It was only meant to handle year boundaries, but even at that there are still edge cases.