[11.0-stable] Configure Go GC based on pillar memory limit or global config #4274
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rouming
requested review from
deitch,
yash-zededa,
OhmSpectator,
rene,
milan-zededa,
zedi-pramodh and
eriknordmark
as code owners
OhmSpectator
approved these changes
Sep 23, 2024
rouming
force-pushed
the
11.0-stable-gogc-configuration
branch
3 times, most recently
from
262a2c3 to
65fac58
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
OhmSpectator
approved these changes
Sep 23, 2024
Patch introduces two settings for Golang runtime which
impacts garbage collector behavior:
1. `gogc.memory.limit.bytes` provides the runtime with a soft memory
limit. The runtime undertakes several processes to try to respect
this memory limit, including adjustments to the frequency of
garbage collections and returning memory to the underlying system
more aggressively. The Go API call is described here:
https://pkg.go.dev/runtime/debug#SetMemoryLimit
By default, EVE setting is disabled (set to 0), meaning the Golang
runtime memory limit will be set according to the following
equation based on the `memory.limit_in_bytes` hard memory limit
provided by the pillar `cgroups`:
`limit = memory.limit_in_bytes * 0.6`
The constant 0.6 was chosen empirically and is explained by simple
logic: `memory.limit_in_bytes` is a hard limit for the whole
pillar cgroup, meaning when reached, likely one of the processes
will be killed by OOM. In turn Golang runtime memory limit is a
soft limit, so the difference must be significant to ensure that
after the soft limit is reached, there will be enough memory for
the Go garbage collector to do its job and, fortunately, not to
hit the hard limit.
2. `gogc.percent` sets the garbage collection target percentage: a
collection is triggered when the ratio of freshly allocated data
to live data remaining after the previous collection reaches this
percentage. The Go API call is described here:
https://pkg.go.dev/runtime/debug#SetGCPercent
The patch is motivated by a frequently observed bloated `zedbox`
application (up to 500MB) that causes an OOM kill call to the /eve or
/pillar cgroups. It is assumed that the bloated `zedbox` application
is not caused by memory leaks, but by a delayed GC sweep cycle and a
unconditionally growing runtime heap size. An explicit memory limit
set for the Golang runtime (~400MB in the current version of EVE)
should make the GC more aggressive when the soft memory limit is hit,
which should result in a significant reduction in allocated but unused
memory.
Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
(cherry picked from commit 40feb16)
rouming
force-pushed
the
11.0-stable-gogc-configuration
branch
from
65fac58 to
a96c7b2
Compare
|
Added the "cherry picked from commit" line. No other changes. |
OhmSpectator
approved these changes
Sep 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of the #4273.
PR introduces two settings for Golang runtime which impacts garbage collector behavior:
The PR is motivated by a frequently observed bloated zedbox application (up to 500MB) that causes an OOM kill call to the /eve or /pillar cgroups. It is assumed that the bloated zedbox application is not caused by memory leaks, but by a delayed GC sweep cycle and a unconditionally growing runtime heap size. An explicit memory limit set for the Golang runtime (~400MB in the current version of EVE) should make the GC more aggressive when the soft memory limit is hit, which should result in a significant reduction in allocated but unused memory.
Cc: @rene @OhmSpectator