Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configurable Log Level for Agones controllers #1218

Closed
aLekSer opened this issue Dec 9, 2019 · 4 comments
Closed

Configurable Log Level for Agones controllers #1218

aLekSer opened this issue Dec 9, 2019 · 4 comments
Labels
area/operations Installation, updating, metrics etc kind/feature New features for Agones
Milestone
1.5.0

Comments

@aLekSer
Copy link
Collaborator

aLekSer commented Dec 9, 2019

Is your feature request related to a problem? Please describe.
There is an additional billing cost caused by excessive Agones controller logging.

Describe the solution you'd like
We need to prioritise the logs, some should be moved to debug level.
Add Helm variables to configure log level.

Describe alternatives you've considered

Additional context
Similar work we have done for Agones SDK server:
#971
@aLekSer aLekSer added the kind/feature New features for Agones label Dec 9, 2019
@aLekSer aLekSer mentioned this issue Dec 9, 2019
Merged
@roberthbailey
Copy link
Member

@KamiMay

@markmandel
Copy link
Member

@KamiMay would love to know which logs you are seeing the most - see if we can narrow down some culprits.

One note I made on the PR #1220 - do we need some rules of thumbs for which logs should be debug vs info ? (Which we can then document).

@markmandel markmandel added the area/operations Installation, updating, metrics etc label Dec 9, 2019
@KamiMay
Copy link

KamiMay commented Dec 10, 2019 •
edited
Loading

These are my observations on an active cluster at about 400 CCU during off peak time in EUW in 1 minute. Size specified under each log is of a single entry.

Log type: Syncing
Variations: Syncing Shutdown State, Syncing Port Allocation GameServerState, Syncing Create State, Syncing RequestReady State, Syncing with Deletion Timestamp, syncing deleted GameServer (strange capitalisation compared to others)
Occurs in 1 minute: ~120 times
Size: 10KB

Log type: syncGameServerSet
Variations: syncGameServerSet, syncGameServerSet finished
Occurs in 1 minute: ~650 times
Size: 1.1KB

Log type: Processing
Variations: Processing
Occurs in 1 minute: ~560 times
Size: 1.1KB

Log type: Reconciling GameServerSet
Variations: Reconciling GameServerSet
Occurs in 1 minute: ~330 times
Size: 8KB

Log type: Enqueuing
Variations: Enqueuing, Enqueuing immediately
Occurs in 1 minute: ~600 times
Size: 1.1KB

Log type: Synchronising
Variations: Synchronising
Occurs in 1 minute: ~240 times
Size: 1.1KB

Log type: Update Fleet Status
Variations: Update Fleet Status
Occurs in 1 minute: ~90 times
Size: 13KB

Log type: running webhook
Variations: running webhook
Occurs in 1 minute: ~30 times
Size: 1KB

Log type: Adding more gameservers
Variations: Adding more gameservers
Occurs in 1 minute: ~10 times
Size: 8KB

Log type: patch created!
Variations: patch created!
Occurs in 1 minute: ~10 times
Size: 8KB

Log type: creationValidationHandler
Variations: creationValidationHandler
Occurs in 1 minute: ~15 times
Size: 20KB

Log type: No pods found, removing finalizer agones.dev
Variations: No pods found, removing finalizer agones.dev
Occurs in 1 minute: ~15 times
Size: 8KB

Log type: Event
Variations: Example: Event(v1.ObjectReference{Kind:"GameServer", Namespace:"default", Name:"royale-server-6zz97", UID:"9409fb88-1b43-42010a840174", APIVersion:"agones.dev/v1", ResourceVersion:"90763796", FieldPath:""}): type: 'Normal' reason: 'PortAllocation' Port allocated
Occurs in 1 minute: ~90 times
Size: 1.3KB

Concerns: These are the only ones that were easy to detect as they are constantly happening, just sum of the listed ones come up to almost 10MB every minute and that is on one cluster (we are running 9 of them) during off peak.

@aLekSer aLekSer mentioned this issue Dec 10, 2019
Closed
markmandel added a commit to markmandel/agones that referenced this issue Dec 13, 2019
@markmandel
Move conflict messaged to Debug Logging
dc57487 
Conflicts when attempting to update Kubernetes resources is expected and
will happen frequently in Agones.

Currently they are reported as errors, and (a) produce a lot of noise,
and (b) are reported as errors, which can often be seen as potential
issues by end users, as well as red-herring reasons for actual issues.

We may want to wait until googleforgames#1220 is complete before merging.

Will help with issues also raised in googleforgames#1218.
@markmandel markmandel mentioned this issue Dec 13, 2019
Merged
markmandel added a commit to markmandel/agones that referenced this issue Dec 13, 2019
@markmandel
Move conflict messaged to Debug Logging
81f1cc6 
Conflicts when attempting to update Kubernetes resources is expected and
will happen frequently in Agones.

Currently they are reported as errors, and (a) produce a lot of noise,
and (b) are reported as errors, which can often be seen as potential
issues by end users, as well as red-herring reasons for actual issues.

We may want to wait until googleforgames#1220 is complete before merging.

Will help with issues also raised in googleforgames#1218.
markmandel added a commit to markmandel/agones that referenced this issue Dec 13, 2019
@markmandel
Move conflict messaged to Debug Logging
a16bf28 
Conflicts when attempting to update Kubernetes resources is expected and
will happen frequently in Agones.

Currently they are reported as errors, and (a) produce a lot of noise,
and (b) are reported as errors, which can often be seen as potential
issues by end users, as well as red-herring reasons for actual issues.

We may want to wait until googleforgames#1220 is complete before merging.

Will help with issues also raised in googleforgames#1218.
markmandel added a commit that referenced this issue Dec 13, 2019
@markmandel
Move conflict messaged to Debug Logging (#1232)
9770f3a 
Conflicts when attempting to update Kubernetes resources is expected and
will happen frequently in Agones.

Currently they are reported as errors, and (a) produce a lot of noise,
and (b) are reported as errors, which can often be seen as potential
issues by end users, as well as red-herring reasons for actual issues.

We may want to wait until #1220 is complete before merging.

Will help with issues also raised in #1218.
@markmandel markmandel mentioned this issue Dec 30, 2019
Merged
markmandel added a commit to markmandel/agones that referenced this issue Feb 1, 2020
@markmandel
Move all "Synchronising" logs to debug
33a4439 
Just more work on googleforgames#1218 to reduce logging verbosity.
@markmandel markmandel mentioned this issue Feb 1, 2020
Merged
pooneh-m pushed a commit that referenced this issue Feb 3, 2020
@markmandel
Move all "Synchronising" logs to debug (#1311)
be51a18 
Just more work on #1218 to reduce logging verbosity.
@aLekSer
Copy link
Collaborator Author

aLekSer commented Mar 16, 2020

As from the above list only these types would occur with the Info log level:

Log type: patch created!
Variations: patch created!
Occurs in 1 minute: ~10 times
Size: 8KB

Log type: Event
Variations: Example: Event(v1.ObjectReference{Kind:"GameServer", Namespace:"default", Name:"royale-server-6zz97", UID:"9409fb88-1b43-42010a840174", APIVersion:"agones.dev/v1", ResourceVersion:"90763796", FieldPath:""}): type: 'Normal' reason: 'PortAllocation' Port allocated
Occurs in 1 minute: ~90 times
Size: 1.3KB

Log type: No pods found, removing finalizer agones.dev
Variations: No pods found, removing finalizer agones.dev
Occurs in 1 minute: ~15 times
Size: 8KB

Log type: Adding more gameservers
Variations: Adding more gameservers
Occurs in 1 minute: ~10 times
Size: 8KB

Other messages were moved into Debug log level.

I am closing this issue, because most noisy log messages were moved to Debug log level.

@aLekSer aLekSer closed this as completed Mar 16, 2020
@markmandel markmandel added this to the 1.5.0 milestone Apr 7, 2020
ilkercelikyilmaz pushed a commit to ilkercelikyilmaz/agones that referenced this issue Oct 23, 2020
@markmandel @ilkercelikyilmaz
Move conflict messaged to Debug Logging (googleforgames#1232)
2db6e63 
Conflicts when attempting to update Kubernetes resources is expected and
will happen frequently in Agones.

Currently they are reported as errors, and (a) produce a lot of noise,
and (b) are reported as errors, which can often be seen as potential
issues by end users, as well as red-herring reasons for actual issues.

We may want to wait until googleforgames#1220 is complete before merging.

Will help with issues also raised in googleforgames#1218.
ilkercelikyilmaz pushed a commit to ilkercelikyilmaz/agones that referenced this issue Oct 23, 2020
@markmandel @ilkercelikyilmaz
Move all "Synchronising" logs to debug (googleforgames#1311)
ad6424e 
Just more work on googleforgames#1218 to reduce logging verbosity.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/operations Installation, updating, metrics etc kind/feature New features for Agones
Projects
None yet
Milestone
1.5.0
Development

No branches or pull requests
4 participants
@markmandel @KamiMay @aLekSer @roberthbailey