docs(inputs.syslog): Document change of default read timeout to infinite

[srebhan]

Summary

When setting a read-timeout on stream connections such as TCP, the server (in this case the syslog plugin) reads from an initiated connection. In case no new message arrives during the set period, the read times out and the plugin expects the connection to be dead. As a consequence, the connection is closed.

In case of syslog, this means that the remote logger needs to send logs with a maximum time-distance of the set timeout. By default we set a timeout of 5 seconds which forces the logger to send log every 5 seconds as otherwise the connection times-out and is closed. This causes errors such as

Feb  4 22:18:17 logserver rsyslogd: omfwd: remote server at 127.0.0.1:6514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2302.0 try https://www.rsyslog.com/e/2027 ]
Feb  4 22:18:17 logserver rsyslogd: action 'action-0-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2302.0 try https://www.rsyslog.com/e/2007 ]
Feb  4 22:18:17 logserver rsyslogd: action 'action-0-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2302.0 try https://www.rsyslog.com/e/2359 ]

on the remote side. This is unexpected by the average user and a default of 5 seconds seems unreasonable for logging with potentially long periods of low or no activity.

This PR changes the default read-timeout to zero resulting in connection to never timeout and thus to never be closed from the plugin side. In case a user sets the timeout, we now print a warning to make the user aware of the behavior.
Furthermore, the PR adds a unit-test for issue #10121.

PR #14837 modified the default read timeout.

Checklist

• No AI generated code was used in this PR

Related issues

[powersj]

Thanks! I did update the message slightly.

Can we not land this in 1.29.5 and instead punt to 1.30?

[srebhan]

@powersj yeah we can do this, want me to change this to a fix!(...) and mark it a breaking change?

[powersj]

yes please

[srebhan]

@powersj why did you lowercase the message? This is a log-warn and thus should start with a capital letter!

[powersj]

@powersj why did you lowercase the message? This is a log-warn and thus should start with a capital letter!

That is habbit sorry about that; the bigger issue is the overuse of exclamation marks ;)

[powersj]

@srebhan I've added the capital letter back, can you rebase to resolve the conflict?

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz