-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python.d.plugin: use separate process for initial module checking #5552
Conversation
ok, i tested it with on Manjaro ( |
Hey @Ferroin if you could find a time and test the PR a bit it would be very nice |
ok, i don't think we need to run a job if |
@ktsaou if you are ok with the changes this is ready. |
@cakrit please approve |
ok, here we go, i am on a high alert ⏰ |
…tdata#5552) ##### Summary This PR adds (major) changes only to `python.d.plugin` file. Fixes: netdata#5525 `pyhton.d.plugin` imports a lot of additional packages during initial module initialization/job creating/checking and there is no way to unimport them, even if they arn't needed. It consumes relatively a lot of ram. ___ Memory utilization comparing before/after the PR (one job `example` module, py3.7.2): > 21.1 => 8.8 MiB ![screenshot_20190305_111837](https://user-images.githubusercontent.com/22274335/53791147-c27a6e00-3f39-11e9-8eaf-8ac3809a3b6e.png) ##### Component Name [`collectors/python.d.plugin`](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/python.d.plugin.in) ##### Additional Information This PR adds separate process for initial module checking. Logic: - main process spawns checker process - checker process loads every module, loads module config, creates jobs and runs job.check() for every job, if check success it adds the job to the list. - checker process returns list of modules and jobs. - main process loads only active modules, etc.
Summary
This PR adds (major) changes only to
python.d.plugin
file.Fixes: #5525
pyhton.d.plugin
imports a lot of additional packages during initial module initialization/job creating/checking and there is no way to unimport them, even if they arn't needed. It consumes relatively a lot of ram.Memory utilization comparing before/after the PR (one job
example
module, py3.7.2):Component Name
collectors/python.d.plugin
Additional Information
This PR adds separate process for initial module checking.
Logic: