-
Notifications
You must be signed in to change notification settings - Fork 371
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enforce memory usage for agent #2671
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #2671 +/- ##
===========================================
+ Coverage 71.98% 72.02% +0.04%
===========================================
Files 103 103
Lines 15698 15740 +42
Branches 2245 2254 +9
===========================================
+ Hits 11300 11337 +37
Misses 3881 3881
- Partials 517 522 +5
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
|
||
NOTE: This option is experimental and may be removed in later versions of the Agent. | ||
""" | ||
return conf.get_int("Debug.AgentMemoryQuota", 30 * 1024 ** 2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just a placeholder value. We will update once we have concrete value.
|
||
NOTE: This option is experimental and may be removed in later versions of the Agent. | ||
""" | ||
return conf.get_switch("Debug.EnableAgentMemoryUsageCheck", False) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
feature set to false for now
""" | ||
try: | ||
if conf.get_enable_agent_memory_usage_check() and self._extensions_summary.converged: | ||
if self._last_check_memory_usage == datetime.min or datetime.utcnow() >= (self._last_check_memory_usage + UpdateHandler.CHECK_MEMORY_USAGE_PERIOD): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would check if goal state completed and also, cgroup monitor period.
Note: I'm using same cgroup monitor period to run this check assuming that we evaluate and calculate memory limit based on the values we get from monitoring thread.
msg = "Check on agent memory usage:\n{0}".format(ustr(exception)) | ||
logger.info(msg) | ||
add_event(AGENT_NAME, op=WALAEventOperation.CGroupsInfo, is_success=True, message=msg) | ||
raise ExitException("Agent {0} is reached memory limit -- exiting".format(CURRENT_AGENT)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This ExitException and rest of the logic of sys.exit already handled in main thread for safe exit.
add_event(AGENT_NAME, op=WALAEventOperation.CGroupsInfo, is_success=True, message=msg) | ||
raise ExitException("Agent {0} is reached memory limit -- exiting".format(CURRENT_AGENT)) | ||
except Exception as exception: | ||
if self._check_memory_usage_last_error_report == datetime.min or (self._check_memory_usage_last_error_report + timedelta(hours=6)) > datetime.now(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is to avoid flooding error msgs in agent log
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool. Could also add a short string such as "[Won't report the same error for 6 hours]"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the long, long delay. Added a couple of comments/requests.
current_usage += metric.value | ||
|
||
if current_usage > conf.get_agent_memory_quota(): | ||
raise CGroupsException("The agent memory limit {0} bytes exceeded. The current reported usage is {1} bytes.".format(conf.get_agent_memory_quota(), current_usage)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use a new exception type for this (AgentMemoryExceededException?). CGroupsException is too generic, and also in general the code handles cgroups issues in such a way that the correspoding cgroup task becomes a no-op.
azurelinuxagent/common/conf.py
Outdated
@@ -186,6 +187,7 @@ def load_conf_from_file(conf_file_path, conf=__conf__): | |||
"Debug.CgroupCheckPeriod": 300, | |||
"Debug.AgentCpuQuota": 50, | |||
"Debug.AgentCpuThrottledTimeThreshold": 120, | |||
"Debug.AgentMemoryQuota": 31457280, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we use the same notation as below? (30 * 1024 ** 2)
azurelinuxagent/ga/update.py
Outdated
except CGroupsException as exception: | ||
msg = "Check on agent memory usage:\n{0}".format(ustr(exception)) | ||
logger.info(msg) | ||
add_event(AGENT_NAME, op=WALAEventOperation.CGroupsInfo, is_success=True, message=msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should have a new WALAEventOperation for this event
add_event(AGENT_NAME, op=WALAEventOperation.CGroupsInfo, is_success=True, message=msg) | ||
raise ExitException("Agent {0} is reached memory limit -- exiting".format(CURRENT_AGENT)) | ||
except Exception as exception: | ||
if self._check_memory_usage_last_error_report == datetime.min or (self._check_memory_usage_last_error_report + timedelta(hours=6)) > datetime.now(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool. Could also add a short string such as "[Won't report the same error for 6 hours]"?
Description
Instead enforcing memory with cgroup, agent itself monitor the current memory usage at regular intervals and do the safe exit when memory reaches the limit. This approach avoids OOM kills on system.
By default, memory usage check set to false for now. We will enable once we know the agent memory limit value.
Issue #
PR information
Quality of Code and Contribution Guidelines