[rsyslog]: Use RELP instead of UDP for forwarding from container to host #18113

saiarcot895 · 2024-02-16T23:09:41Z

Why I did it

When the host's rsyslog is restarted (for example, to regenerate the config after some changes, or as part of some automated script), there is a chance that some syslog messages from the containers are lost. Most of the time, this isn't an issue. However, if there are test cases that expect all syslogs to be present (such as the advanced-reboot test case), then this can cause a problem. Additionally, this could affect debuggability of issues where a rsyslog restart happens in the middle.

There are two options for reliable message transport in rsyslog: TCP and RELP. With TCP, while the protocol knows whether a syslog message has been delivered or not, the application doesn't know, because there is no feedback from the remote side saying the message was received. This means that there is still a chance that messages could be lost when the connection is broken (if, for example, the host rsyslog gets restarted), because after the connection is established, the sender rsyslog (in the
container) doesn't know if the message has been received or not.

RELP builds on top of TCP, and adds a feedback mechanism where the remote side notifies the sender whether the message has actually been received or not. This makes it much less likely to lose a message. There is one known possible case where a message (or messages) could be lost: the network is down, and rsyslog gets restarted. This at least requires both the network and rsyslog to have an issue, rather than just one. There is also a slim possibility where a message could get duplicated; this should be mostly fine (hopefully).

RELP does require that both sides are using a recent version of rsyslogd (at least 7.3.16, which looks like it was released more than 10 years ago), but since we use Debian on both the container and the host, it should be fine.

Therefore, switch to using RELP when sending syslog messages from the container to the host.

Fixes #17792.

Work item tracking

Microsoft ADO (number only): 28314311

How I did it

Modify the rsyslog.conf file on the host and the container to use RELP instead of UDP.

In addition, update the syntax used for the config files to the (newer) RainierScript format, which, among other things, makes it easier to set settings for specific outputs.

How to verify it

Stop rsyslogd on the host, make sure that the containers generate some syslogs, restart rsyslogd on the host, and verify no logs were lost.

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

When the host's rsyslog is restarted (for example, to regenerate the config after some changes, or as part of some automated script), there is a chance that some syslog messages from the containers are lost. Most of the time, this isn't an issue. However, if there are test cases that expect all syslogs to be present (such as the advanced-reboot test case), then this can cause a problem. Additionally, this could affect debuggability of issues where a rsyslog restart happens in the middle. There are two options for reliable message transport in rsyslog: TCP and RELP. With TCP, while the protocol knows whether a syslog message has been delivered or not, the application doesn't know, because there is no feedback from the remote side saying the message was received. This means that there is still a chance that messages could be lost when the connection is broken (if, for example, the host rsyslog gets restarted), because after the connection is established, the sender rsyslog (in the container) doesn't know if the message has been received or not. RELP instead adds a feedback mechanism where the remote side notifies the sender whether the message has actually been received or not. This makes it much less likely to lose a message. There is one known possible case where a message (or messages) could be lost: the network is down, and rsyslog gets restarted. This at least requires both the network and rsyslog to have an issue, rather than just one. There is also a slim possibility where a message could get duplicated; this should be mostly fine (hopefully). RELP does require that both sides are using a recent version of rsyslogd (at least 7.3.16, which looks like it was released more than 10 years ago), but since we use Debian on both the container and the host, it should be fine. Therefore, switch to using RELP when sending syslog messages from the container to the host. Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

…urst not being defined $SystemLogRateLimitInterval and $SystemLogRateLimitBurst both come from the imuxsock module. Specify them as module parameters (and also remove the legacy syntax). Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

By default, just using omrelp doesn't hold log messages if the server happens to be unavailable. This needs to be configured manually. Configure an in-memory storage (of a linked list) that by default will store up to 1000 messages (this appears to be a default value that can be bumped up) if the server is unavailable. I'm assuming this will be sufficient for most cases. Assuming each message is 512 bytes (many of our messages will be smaller than this), this will take up an additional 512kB of memory if 1000 messages are queues. If there are no messages queued, then no additional space is taken up. Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

If rsyslogd on the host goes down, and rsyslogd on the containers is configured to use librelp to forward messages to the host rsyslogd (instead of UDP), then there will be error messages from the container rsyslogd about not being able to forward messages. Ignore these error messages as they are expected when running tests which may restart rsyslogd. This is in preparation for sonic-net/sonic-buildimage#18113 Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

saiarcot895 · 2024-06-24T16:05:17Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2024-06-24T16:05:19Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2024-06-24T16:05:31Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

* Ignore errors about rsyslogd w/ librelp not being able to send syslogs If rsyslogd on the host goes down, and rsyslogd on the containers is configured to use librelp to forward messages to the host rsyslogd (instead of UDP), then there will be error messages from the container rsyslogd about not being able to forward messages. Ignore these error messages as they are expected when running tests which may restart rsyslogd. This is in preparation for sonic-net/sonic-buildimage#18113 Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

saiarcot895 · 2024-08-01T20:47:16Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2024-08-01T20:47:18Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2024-08-01T20:47:32Z

Azure Pipelines successfully started running 1 pipeline(s).

saiarcot895 · 2024-08-30T15:28:35Z

/azpw ms_checker

saiarcot895 · 2024-08-30T15:28:49Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2024-08-30T15:28:52Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2024-08-30T15:29:06Z

Azure Pipelines successfully started running 1 pipeline(s).

prgeor · 2024-08-31T19:25:26Z

dockers/docker-base-bookworm/etc/rsyslog.conf

+module(load="imuxsock" SysSock.RateLimit.Interval="300" SysSock.RateLimit.Burst="20000") # provides support for local system logging
+#module(load="imklog")   # provides kernel logging support
+#module(load="immark")  # provides --MARK-- message capability


@saiarcot895 can you mention this syntax change in PR description

prgeor · 2024-08-31T19:26:32Z

dockers/docker-base-bookworm/etc/rsyslog.conf

@@ -37,7 +33,8 @@ set $.CONTAINER_NAME=getenv("CONTAINER_NAME");

 # Set remote syslog server
 template (name="ForwardFormatInContainer" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %$.CONTAINER_NAME%#%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
-*.* action(type="omfwd" target=`echo $SYSLOG_TARGET_IP` port="514" protocol="udp" Template="ForwardFormatInContainer")
+module(load="omrelp")
+*.* action(type="omrelp" target=`echo $SYSLOG_TARGET_IP` port="2514" action.resumeRetryCount="-1" queue.type="LinkedList" Template="ForwardFormatInContainer")


@saiarcot895 2514 is the port used by relp?

There's no standard port used by RELP. The default port that rsyslog uses is 514, but that can conflict with regular syslog forwarding over TCP. A couple of the examples in the documentation for this feature uses 2514.

prgeor · 2024-08-31T19:27:03Z

dockers/docker-base-bullseye/etc/rsyslog.conf

@saiarcot895 are we still using bullseye?

Yes, bullseye is still being used for a couple containers.

prgeor

what about platform/vs/docker-sonic-vs/etc/rsyslog.conf don't need this change?

prgeor · 2024-08-31T20:06:53Z

files/image_config/rsyslog/rsyslog.conf.j2

-#$ModLoad immark  # provides --MARK-- message capability
+module(load="imuxsock" {% if rate_limit_interval is not none %}SysSock.RateLimit.Interval="{{ rate_limit_interval }}"{% endif %} {% if rate_limit_burst is not none %}SysSock.RateLimit.Burst="{{ rate_limit_burst }}"{% endif %}) # provides support for local system logging
+module(load="imklog")   # provides kernel logging support
+#module(load="immark")  # provides --MARK-- message capability

 # provides UDP syslog reception


@saiarcot895 This UDP syslog is for remote server?

Yes, in the case of a remote syslog server sending over UDP.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

saiarcot895 · 2024-09-25T23:16:08Z

what about platform/vs/docker-sonic-vs/etc/rsyslog.conf don't need this change?

Strictly speaking, it doesn't need this change, because the logs aren't actually being forwarded anywhere. It'll forward it to localhost port 514, but there likely won't be anything listening on this port. That container doesn't end up on the device.

It would be nice to update the syntax there to have it use the new syntax, but I'll keep that separate for now.

* Ignore errors about rsyslogd w/ librelp not being able to send syslogs If rsyslogd on the host goes down, and rsyslogd on the containers is configured to use librelp to forward messages to the host rsyslogd (instead of UDP), then there will be error messages from the container rsyslogd about not being able to forward messages. Ignore these error messages as they are expected when running tests which may restart rsyslogd. This is in preparation for sonic-net/sonic-buildimage#18113 Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

saiarcot895 · 2024-10-14T21:19:14Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2024-10-14T21:19:16Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2024-10-14T21:19:30Z

Azure Pipelines successfully started running 1 pipeline(s).

* Ignore errors about rsyslogd w/ librelp not being able to send syslogs If rsyslogd on the host goes down, and rsyslogd on the containers is configured to use librelp to forward messages to the host rsyslogd (instead of UDP), then there will be error messages from the container rsyslogd about not being able to forward messages. Ignore these error messages as they are expected when running tests which may restart rsyslogd. This is in preparation for sonic-net/sonic-buildimage#18113 Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

In case rsyslog can't forward messages to the host's rsyslog server, messages will be queued so that they can be sent out later. For this queue, set a limit of 20000 messages so that rsyslog doesn't take too much memory. Assuming each message is 512 bytes, the approximate maximum additional memory usage is 10MB. Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

qiluo-msft · 2024-10-23T22:14:58Z

This is kind of new feature. Is it possible to config for new behavior or old behavior?

saiarcot895 added 2 commits February 4, 2024 22:57

Try to fix repeated messages being logged, and clean up templates

e275fd2

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

saiarcot895 requested review from qiluo-msft and lguohan as code owners February 16, 2024 23:09

saiarcot895 added 2 commits February 16, 2024 18:28

saiarcot895 marked this pull request as draft June 7, 2024 18:29

saiarcot895 added 7 commits June 7, 2024 11:33

Merge remote-tracking branch 'origin/master' into rsyslog-use-relp

7c937cc

Update rsyslog.conf file for Bookworm

dd7af0a

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Merge remote-tracking branch 'origin/master' into rsyslog-use-relp

6ebca5f

Install rsyslog-relp for Bookworm

ed01b28

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Use new syntax for specifying rate limiting interval and burst

bbe163b

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Revert template change to match what is currently on master branch

e030511

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Restore the RepeatedMsgReduction config option

ab995b0

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

saiarcot895 mentioned this pull request Jul 26, 2024

Ignore loganalyzer errors for rsyslog RELP sonic-net/sonic-mgmt#13853

Merged

8 tasks

saiarcot895 added 4 commits July 26, 2024 17:05

Merge remote-tracking branch 'origin/master' into rsyslog-use-relp

d57c5ba

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Update rsyslog.conf file for pmon docker

4d8056b

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Remove extra newlines

bf9cda9

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Add (or re-add) retry support into the default /etc/rsyslog.conf files

d5edef8

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

saiarcot895 marked this pull request as ready for review August 7, 2024 16:42

saiarcot895 requested a review from prgeor August 7, 2024 16:42

prgeor reviewed Aug 31, 2024

View reviewed changes

If tests fail, print the diff as the error message

20e8621

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Merge remote-tracking branch 'origin/master' into rsyslog-use-relp

5310d94

saiarcot895 requested a review from prgeor October 14, 2024 21:19

Merge remote-tracking branch 'origin/master' into rsyslog-use-relp

5444c45

prgeor previously approved these changes Oct 23, 2024

View reviewed changes

saiarcot895 dismissed prgeor’s stale review via dce24d6 October 23, 2024 20:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rsyslog]: Use RELP instead of UDP for forwarding from container to host #18113

[rsyslog]: Use RELP instead of UDP for forwarding from container to host #18113

saiarcot895 commented Feb 16, 2024 •

edited

Loading

saiarcot895 commented Jun 24, 2024

mssonicbld commented Jun 24, 2024

azure-pipelines bot commented Jun 24, 2024

saiarcot895 commented Aug 1, 2024

mssonicbld commented Aug 1, 2024

azure-pipelines bot commented Aug 1, 2024

saiarcot895 commented Aug 30, 2024

saiarcot895 commented Aug 30, 2024

mssonicbld commented Aug 30, 2024

azure-pipelines bot commented Aug 30, 2024

prgeor Aug 31, 2024

saiarcot895 Sep 13, 2024

prgeor Aug 31, 2024 •

edited

Loading

saiarcot895 Sep 13, 2024

prgeor Aug 31, 2024

saiarcot895 Sep 13, 2024

prgeor left a comment

prgeor Aug 31, 2024

saiarcot895 Sep 13, 2024

saiarcot895 commented Sep 25, 2024

saiarcot895 commented Oct 14, 2024

mssonicbld commented Oct 14, 2024

azure-pipelines bot commented Oct 14, 2024

qiluo-msft commented Oct 23, 2024

[rsyslog]: Use RELP instead of UDP for forwarding from container to host #18113

Are you sure you want to change the base?

[rsyslog]: Use RELP instead of UDP for forwarding from container to host #18113

Conversation

saiarcot895 commented Feb 16, 2024 • edited Loading

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

saiarcot895 commented Jun 24, 2024

mssonicbld commented Jun 24, 2024

azure-pipelines bot commented Jun 24, 2024

saiarcot895 commented Aug 1, 2024

mssonicbld commented Aug 1, 2024

azure-pipelines bot commented Aug 1, 2024

saiarcot895 commented Aug 30, 2024

saiarcot895 commented Aug 30, 2024

mssonicbld commented Aug 30, 2024

azure-pipelines bot commented Aug 30, 2024

prgeor Aug 31, 2024

Choose a reason for hiding this comment

saiarcot895 Sep 13, 2024

Choose a reason for hiding this comment

prgeor Aug 31, 2024 • edited Loading

Choose a reason for hiding this comment

saiarcot895 Sep 13, 2024

Choose a reason for hiding this comment

prgeor Aug 31, 2024

Choose a reason for hiding this comment

saiarcot895 Sep 13, 2024

Choose a reason for hiding this comment

prgeor left a comment

Choose a reason for hiding this comment

prgeor Aug 31, 2024

Choose a reason for hiding this comment

saiarcot895 Sep 13, 2024

Choose a reason for hiding this comment

saiarcot895 commented Sep 25, 2024

saiarcot895 commented Oct 14, 2024

mssonicbld commented Oct 14, 2024

azure-pipelines bot commented Oct 14, 2024

qiluo-msft commented Oct 23, 2024

saiarcot895 commented Feb 16, 2024 •

edited

Loading

prgeor Aug 31, 2024 •

edited

Loading