-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Syslog Input: expecting a MSGLEN #4482
Comments
I'll double check the way docker syslog works, but I imagine we'll just need a new format to parse logs as |
Thanks for bringing this up. It appears we are always parsing syslogs as if they were sent TLS style (RFC5425) with a telegraf.conf[[inputs.syslog]]
server = "tcp://localhost:6514"
tls_cert = "/tmp/thing.crt"
tls_key = "/tmp/thing.key" daemon.json{
"log-opts": {
"syslog-tls-skip-verify": "true",
"syslog-address": "tcp+tls://localhost:6514",
"syslog-format": "rfc5424micro"
},
"log-driver": "syslog"
} Last note: you won't have to include |
The UDP version works without requiring any certificates: telegraf.conf[[inputs.syslog]]
server = "udp://localhost:6514" daemon.json{
"log-driver": "syslog",
"log-opts": {
"syslog-address": "udp://localhost:6514",
"syslog-format": "rfc5424micro"
}
} @leodido Should we select between the 5424 and 5425 parsers depending on if the TCP socket is configured for TLS? |
RFC5424 does not explicitly address the type of framing. Unlike RFC5425, which explicitly provides octet counting - ie., So my answer is no, we should not select the parsers depending on the protocol. RFC5425 does not contemplate another framing technique. It has been done exactly for this reason, since RFC5424 was missing this part and various client started, in the past, to make custom framing techniques. Please notice that octet counting allows syslog message to be split across different packets, among other things. Therefore, since we are providing RFC5425 parsing respecting this RFC, ideally, we should not modify this behavior. SolutionWhat could be done, imho, is to provide another parser respecting the RFC5424 (we already have it and we use it internally in RFC5425) leaving the choice of the framing technique (eg., old one like new lines and so on) to the user. But imho it should be enabled via options by the user, not switched depending on the protocol in use. This way users can also use clients not respecting RFC5425 and implementing custom framing techniques. |
I agree that we could expose some options to configure the spec followed as I think we would run into issues by simply checking if the plugin is configured with tls prior to parsing. For example Examples:Config with no
Produces (RFC5424 output):
Config with
Produces (RFC5425 output):
|
That's exactly what I am saying. It is not protocol dependant.
RFC5425 basically wraps RFC5424 with octet counting framing technique.
An user can utilise a syslog relay which emits RFC5424 (ef., rsyslog
default behavior).
…On Wed, Aug 1, 2018, 7:28 PM Greg ***@***.***> wrote:
I agree that we could expose some options to configure the spec followed
as I think we would run into issues by simply checking if the plugin is
configured with tls prior to parsing. For example rsyslog can prefix the
message length to the log by adding (o) to the entry in rsyslog.conf so
even though RSYSLOG_SyslogProtocol23Format (RFC5424) is specified, the
message can still be transmitted to match RFC5425.
------------------------------
Examples:
Config with no (o):
*.* @@127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format
Produces (RFC5424 output):
<30>1 2018-08-01T11:16:08.529324-06:00 hilldale systemd 1 - - Started System Logging Service.
Config with (o):
*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format
Produces (RFC5425 output):
95 <30>1 2018-08-01T11:12:29.276656-06:00 hilldale systemd 1 - - Started System Logging Service.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4482 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAHU80CclWi90XGjrBUq9bIz2vajTfmTks5uMeU1gaJpZM4Vj0SQ>
.
|
I mean the RFC5425 spec title and such could be more clear though; "Transport Layer Security (TLS) Transport Mapping for Syslog." Plus, it does state:
Took some digging into rsyslog configuration to realize why it was sending the |
If we add a boolean option: This would also mean one additional option that must be set before using the plugin, perhaps we should it as a multi value option instead: |
Hi @danielnelson @glinton, |
@JulienChampseix We haven't had time to work on this yet, it is probably something for 1.9.0. I think we should do the 3-state option:
Default would be |
blocked by influxdata/go-syslog#17 |
So can someone actually concisely TL;DR this whole mess in how to configure rsyslog and telegraf to play nice with each other? |
@FrankyBoy were you able to follow the steps in the README? https://github.com/influxdata/telegraf/tree/master/plugins/inputs/syslog#rsyslog-integration If something is missing there, let's open an issue to update the docs. |
Yes, and the udp based path definitely doesn't work right now for me. TCP is not an option for various reasons. |
I have the same problem with the Keycloak Syslog plugin, which follows RFC 5424 but also leads to
Telegraf/influxdb work if I manually compose a test log entry with MSGLEN:
I cannot change the way the Keycloak syslog sends these entries. [edit] Doh! It is writting in the docs:
Keycloak syslog requires without the octet counting framing. And in the telegraf plugin it says:
I only had to enable
|
is this issue #4335 fixed?
I am seeing same errors in latest version of telegraf(1.7.2). I am trying to send my docker daemon logs via syslog drivers.
018/07/27 16:09:10 I! Using config file: /etc/telegraf/telegraf.conf
2018-07-27T16:09:10Z I! Starting Telegraf v1.7.2
2018-07-27T16:09:10Z I! Loaded inputs: inputs.syslog
2018-07-27T16:09:10Z I! Loaded aggregators:
2018-07-27T16:09:10Z I! Loaded processors:
2018-07-27T16:09:10Z I! Loaded outputs: file
2018-07-27T16:09:10Z I! Tags enabled:
2018-07-27T16:09:10Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"", Flush Interval:10s
2018-07-27T16:09:11Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:16Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
Here is my docker daemon json config:
The text was updated successfully, but these errors were encountered: