cmd/scollector: add a kill switch for total memory used by scollector #1866
Conversation
| @@ -245,17 +245,27 @@ func main() { | |||
| } | |||
| collect.MaxQueueLen = conf.MaxQueueLen | |||
| } | |||
| maxMemMegaBytes := uint64(500) | |||
| maxMemMB := uint64(500) | |||
There was a problem hiding this comment.
I'm not sure I love the default of having the kill switch active. May surprise some people.
There was a problem hiding this comment.
It has always been active, but only for runtime memory. This adds a second check for total memory usage.
We were using over 1GB on many systems (around 80GB total on all systems) due to a WMI leak, which could have caused other systems to fail if we hadn't noticed.
|
very nice. LGTM |
|
This only works if the process monitoring collectors are enabled, but it matches based on the os.Getpid call, so I don't think you need to be explicitly monitoring scollector. I'm going to manually deploy this to a few systems including ny-bosun01 (largest scollector memory usage on linux in our environment) to make sure there aren't any issues. Can then merge it on Monday. |
gbrayut
force-pushed
the
scollector_mem
branch
from
722b493 to
ccfad48
Compare
|
So it appears we don't log panics on Windows so I changed the error from a panic to a fatal. I also added #1867 to find a way to log panics. The fatal error will look like this:
|
gbrayut
force-pushed
the
scollector_mem
branch
from
ccfad48 to
1764425
Compare
|
^ was just a rebase and a missing multiplier: Merging to master after travisci builds |
… leaks outside the runtime
gbrayut
force-pushed
the
scollector_mem
branch
from
1764425 to
aad1661
Compare
This should help prevent memory leaks in CGO or WMI from causing scollector to consume too much memory.