enforce memory usage for agent #2671
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #2671 +/- ##
===========================================
+ Coverage 71.98% 72.02% +0.04%
===========================================
Files 103 103
Lines 15698 15740 +42
Branches 2245 2254 +9
===========================================
+ Hits 11300 11337 +37
Misses 3881 3881
- Partials 517 522 +5
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
|
|
||
| NOTE: This option is experimental and may be removed in later versions of the Agent. | ||
| """ | ||
| return conf.get_int("Debug.AgentMemoryQuota", 30 * 1024 ** 2) |
There was a problem hiding this comment.
This is just a placeholder value. We will update once we have concrete value.
|
|
||
| NOTE: This option is experimental and may be removed in later versions of the Agent. | ||
| """ | ||
| return conf.get_switch("Debug.EnableAgentMemoryUsageCheck", False) |
There was a problem hiding this comment.
feature set to false for now
| """ | ||
| try: | ||
| if conf.get_enable_agent_memory_usage_check() and self._extensions_summary.converged: | ||
| if self._last_check_memory_usage == datetime.min or datetime.utcnow() >= (self._last_check_memory_usage + UpdateHandler.CHECK_MEMORY_USAGE_PERIOD): |
There was a problem hiding this comment.
This would check if goal state completed and also, cgroup monitor period.
Note: I'm using same cgroup monitor period to run this check assuming that we evaluate and calculate memory limit based on the values we get from monitoring thread.
| msg = "Check on agent memory usage:\n{0}".format(ustr(exception)) | ||
| logger.info(msg) | ||
| add_event(AGENT_NAME, op=WALAEventOperation.CGroupsInfo, is_success=True, message=msg) | ||
| raise ExitException("Agent {0} is reached memory limit -- exiting".format(CURRENT_AGENT)) |
There was a problem hiding this comment.
This ExitException and rest of the logic of sys.exit already handled in main thread for safe exit.
| add_event(AGENT_NAME, op=WALAEventOperation.CGroupsInfo, is_success=True, message=msg) | ||
| raise ExitException("Agent {0} is reached memory limit -- exiting".format(CURRENT_AGENT)) | ||
| except Exception as exception: | ||
| if self._check_memory_usage_last_error_report == datetime.min or (self._check_memory_usage_last_error_report + timedelta(hours=6)) > datetime.now(): |
There was a problem hiding this comment.
This is to avoid flooding error msgs in agent log
There was a problem hiding this comment.
Cool. Could also add a short string such as "[Won't report the same error for 6 hours]"?
| current_usage += metric.value | ||
|
|
||
| if current_usage > conf.get_agent_memory_quota(): | ||
| raise CGroupsException("The agent memory limit {0} bytes exceeded. The current reported usage is {1} bytes.".format(conf.get_agent_memory_quota(), current_usage)) |
There was a problem hiding this comment.
Let's use a new exception type for this (AgentMemoryExceededException?). CGroupsException is too generic, and also in general the code handles cgroups issues in such a way that the correspoding cgroup task becomes a no-op.
azurelinuxagent/common/conf.py
Outdated
| @@ -186,6 +187,7 @@ def load_conf_from_file(conf_file_path, conf=__conf__): | |||
| "Debug.CgroupCheckPeriod": 300, | |||
| "Debug.AgentCpuQuota": 50, | |||
| "Debug.AgentCpuThrottledTimeThreshold": 120, | |||
| "Debug.AgentMemoryQuota": 31457280, | |||
There was a problem hiding this comment.
can we use the same notation as below? (30 * 1024 ** 2)
azurelinuxagent/ga/update.py
Outdated
| except CGroupsException as exception: | ||
| msg = "Check on agent memory usage:\n{0}".format(ustr(exception)) | ||
| logger.info(msg) | ||
| add_event(AGENT_NAME, op=WALAEventOperation.CGroupsInfo, is_success=True, message=msg) |
There was a problem hiding this comment.
I think we should have a new WALAEventOperation for this event
| add_event(AGENT_NAME, op=WALAEventOperation.CGroupsInfo, is_success=True, message=msg) | ||
| raise ExitException("Agent {0} is reached memory limit -- exiting".format(CURRENT_AGENT)) | ||
| except Exception as exception: | ||
| if self._check_memory_usage_last_error_report == datetime.min or (self._check_memory_usage_last_error_report + timedelta(hours=6)) > datetime.now(): |
There was a problem hiding this comment.
Cool. Could also add a short string such as "[Won't report the same error for 6 hours]"?
Description
Instead enforcing memory with cgroup, agent itself monitor the current memory usage at regular intervals and do the safe exit when memory reaches the limit. This approach avoids OOM kills on system.
By default, memory usage check set to false for now. We will enable once we know the agent memory limit value.
Issue #
PR information
Quality of Code and Contribution Guidelines