Cherry-pick #16349 to 7.x: Fix leaks with metadata processors #21739
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a closer interface for processors so their resources can be released when the processor is not needed anymore. Explicitly close publisher pipelines so their processors are closed. Add closers for add_docker_metadata, add_kubernetes_metadata and add_process_metadata. Script processor will fail if a processor that needs to be closed is used. (cherry picked from commit a3fe796)
jsoriano
added
[zube]: In Review
backport
Team:Platforms
|
Pinging @elastic/integrations-platforms (Team:Platforms) |
|
Pinging @elastic/integrations-services (Team:Services) |
botelastic
bot
added
needs_team
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errors
Expand to view the tests failures
Steps errors
Expand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
|
Failing tests are #21748 |
kaiyan-sheng
approved these changes
Oct 13, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #16349 to 7.x branch. Original message:
What does this PR do?
Add a closer interface for processors so their resources can be released when the processor is not needed anymore.
Explicitly close publisher pipelines so their processors are closed.
Add closers for
add_docker_metadata,add_kubernetes_metadataandadd_process_metadata.Why is it important?
Processors can be defined in dynamic configurations, as in autodiscover templates. Some processors open connections, files, or start goroutines that need to be closed or stopped.
Processors can be dynamically created at least when using autodiscover and when used in the
scriptprocessor. When this issue appears it can be usually solved by configuration (as described in #14389). So this case is probably not so serious.Processors can include an on-disk cache since #20775, files open by this cache should be properly closed on shutdown.
Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesAuthor's Checklist
Check that resources are released in:
Processors at the module level are open multiple times?(One per metricset, to be reviewed as a different issue, only a real problem when a processor cannnot be instantiated twice with the same config, what only happens withadd_cloudfoundry_metadata)Somehow call close on all the instantiated processors once the script is finished.Add tests:
Related issues
Use cases
add_cloudfoundry_metadata).