Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Received gossip status" message is always logged by DistributedData #6074

Closed
object opened this issue Aug 26, 2022 · 5 comments · Fixed by #6080
Closed

"Received gossip status" message is always logged by DistributedData #6074

object opened this issue Aug 26, 2022 · 5 comments · Fixed by #6080

Comments

@object
Copy link
Contributor

object commented Aug 26, 2022

We are going through log information recorded by our Akka services to reduce costs associated with extensive logging. Large portion of log message comes from continuous gossip status logging, that occurs in Akka.DistributedData/Replicator.cs:

private void ReceiveStatus(IImmutableDictionary<string, ByteString> otherDigests, int chunk, int totChunks)
{
            if (_log.IsDebugEnabled)
                _log.Debug("Received gossip status from [{0}], chunk {1}/{2} containing [{3}]", Sender.Path.Address, chunk + 1, totChunks, string.Join(", ", otherDigests.Keys));
            // Skipped the rest
}

The problem with this message is that it's generated with a high frequency, so large part of debug logs are filled with these messages. In our case we are getting about 7.2 million log messages every 24 hours, and about 6.1 million of them are "Received gossip" messages from Akka DistributedData.

I propose turning these messages off by default because they are helpful only under special circumstances. DistributedData uses configuration described here:

https://github.com/akkadotnet/akka.net/blob/dev/src/contrib/cluster/Akka.DistributedData/reference.conf

Adding a debug section that would control gossip logging will solve the issue.

@object
Copy link
Contributor Author

object commented Aug 26, 2022

I need to further investigate this issue, closing it for now. Will open if I confirm it.

@Aaronontheweb
Copy link
Member

So it doesn't appear as though these logs are always on after all? Please let us know what you find in your investigation - these are easy fixes for us to make if there's indeed a bug here.

@object
Copy link
Contributor Author

object commented Aug 29, 2022

@Aaronontheweb I have written the issue and proposed extending DistributedData configuration and turn "Received gossip" off by default. I can send a PR if you want.

@object object reopened this Aug 29, 2022
@Aaronontheweb
Copy link
Member

@object A PR would be very welcome here - we can do it ourselves but we'd always prefer to let a community member take a stab at it

@object
Copy link
Contributor Author

object commented Aug 29, 2022

Great, then I will start looking into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants