Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support for persisting logs in 'emptyDir' volume attached to agones controller. #620

Merged
merged 2 commits into from
Feb 26, 2019
Merged

Added support for persisting logs in 'emptyDir' volume attached to agones controller. #620

merged 2 commits into from
Feb 26, 2019

Conversation

jkowalski
Copy link
Contributor

@jkowalski jkowalski commented Feb 22, 2019 •
edited
Loading

(This creates a secondary place to look at logs for debugging when controller is logging too fast and fluentd can't sync to stackdriver fast enough)

The logs are saved in 100MB chunks under /home/agones/logs up to a specified limit (10000 MB by default).

Example:

$  kubectl exec -n agones-system -it agones-controller-7d85b5c996-8c22b -- ls -lh /home/agones/logs
total 757764
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:22 agones-controller-20190222_212001-2019-02-22T21-22-37.499.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:26 agones-controller-20190222_212001-2019-02-22T21-26-31.904.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:27 agones-controller-20190222_212001-2019-02-22T21-27-01.546.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:27 agones-controller-20190222_212001-2019-02-22T21-27-42.934.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:28 agones-controller-20190222_212001-2019-02-22T21-28-46.574.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:32 agones-controller-20190222_212001-2019-02-22T21-32-31.224.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:33 agones-controller-20190222_212001-2019-02-22T21-33-30.350.log
-rw-r--r--    1 agones   agones     40.0M Feb 22 21:41 agones-controller-20190222_212001.log

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 7cabd1cf-51be-4a31-af82-6e3f2564cff7

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@jkowalski
Added support for persisting logs in 'emptyDir' volume attached to ag…
8285096 
…ones controller.

The logs are saved in 100MB chunks under `/home/agones/logs` up to a specified limit (10000 MB by default).

Example:

```shell
$ ls -lh /home/agones/logs
total 757764
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:22 agones-controller-20190222_212001-2019-02-22T21-22-37.499.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:26 agones-controller-20190222_212001-2019-02-22T21-26-31.904.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:27 agones-controller-20190222_212001-2019-02-22T21-27-01.546.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:27 agones-controller-20190222_212001-2019-02-22T21-27-42.934.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:28 agones-controller-20190222_212001-2019-02-22T21-28-46.574.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:32 agones-controller-20190222_212001-2019-02-22T21-32-31.224.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:33 agones-controller-20190222_212001-2019-02-22T21-33-30.350.log
-rw-r--r--    1 agones   agones     40.0M Feb 22 21:41 agones-controller-20190222_212001.log

```
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 999cd8c7-3476-4ece-9881-eb20aba98a66

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/620/head:pr_620 && git checkout pr_620
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.9.0-8285096

@jkowalski jkowalski marked this pull request as ready for review February 22, 2019 22:15
cyriltovena
cyriltovena approved these changes Feb 24, 2019
View reviewed changes
Copy link
Collaborator

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

markmandel
markmandel reviewed Feb 24, 2019
View reviewed changes
Copy link
Member

@markmandel markmandel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'couple o' thoughts 😄

install/helm/agones/templates/controller.yaml Show resolved Hide resolved
install/helm/agones/values.yaml
@@ -49,6 +49,8 @@ agones:
operator: Exists
generateTLS: true
safeToEvict: false
persistentLogs: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought: Should this be on by default? I'm thinking most people won't be doing load tests, and stackdriver/fluentd are likely fast enough. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not just load testing, we currently emit enough logs to overwhelm fluentd. For any fleets of significant size there are dropped messages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. In that case, yes - let's keep this on by default. I didn't realise we were getting dropped messages, that's unfortunate.

Sounds like something we should push upstream? (or should we be logging less?)

But otherwise, this makes sense, and looks good to me.

Only other thing I can think of - should we add a section in (to be released in 0.9.0) https://agones.dev/site/docs/guides/troubleshooting/ to let people know that this is an option?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, I'll submit this one now and make next PR with doc change and ability to use PVC for durable logs

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 7b48d1a6-aa76-402d-a719-8a8db012f765

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/620/head:pr_620 && git checkout pr_620
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.9.0-7b7ec63

@jkowalski jkowalski merged commit d46947c into googleforgames:master Feb 26, 2019
@markmandel markmandel added this to the 0.9.0 milestone Mar 7, 2019
@markmandel markmandel added kind/feature New features for Agones area/user-experience Pertaining to developers trying to use Agones, e.g. SDK, installation, etc labels Mar 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@markmandel markmandel markmandel left review comments

@cyriltovena cyriltovena cyriltovena approved these changes

Assignees
No one assigned
Labels
area/user-experience Pertaining to developers trying to use Agones, e.g. SDK, installation, etc kind/feature New features for Agones
Projects
None yet
Milestone
0.9.0
Development

Successfully merging this pull request may close these issues.
4 participants
@jkowalski @agones-bot @markmandel @cyriltovena