[libbeat] Enable WriteAheadLimit in the disk queue #21391
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
faec
added
enhancement
libbeat
needs_backport
|
Pinging @elastic/integrations (Team:Integrations) |
botelastic
bot
added
needs_team
kvch
approved these changes
Sep 29, 2020
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
|
what is the behaviour when the memory queue is full? blocking, dropping, configurable? |
Configurable. Blocking vs dropping is unchanged by this PR (in general it is configured by the beat / input producing the events). This PR only changes the conditions under which the queue is considered "full": previously this only meant that it had reached its maximum global byte capacity, whereas now it can also mean that the memory-based intake queue is at capacity. |
faec
added a commit
to faec/beats
that referenced
this pull request
Sep 29, 2020
(cherry picked from commit ef6274d)
faec
added
v7.10.0
and removed
needs_backport
v1v
added a commit
to v1v/beats
that referenced
this pull request
Sep 30, 2020
…ci-build-label-support * upstream/master: [JJBB] Set shallow cloning to 10 (elastic#21409) docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419) docs: Prepare Changelog for 7.9.2 (elastic#21229) (elastic#21403) fix: mark flaky tests (elastic#21300) fix: use a fixed version of setuptools (elastic#21393) Move Kubernetes events metricset to its own block in reference config (elastic#21407) [libbeat] Enable WriteAheadLimit in the disk queue (elastic#21391) docs: fix apt/yum formatting (elastic#21362) Fix shutdown tracking in s3 input (elastic#21380) [libbeat] Fix position writing in the disk queue Add UBI 8 image to the dependencies report (elastic#21374) Fix debug message to show actual SQS message ID (elastic#20614) [Elastic Agent] Rename *ConfigChange to PolicyChange (elastic#20779) [Elastic Agent] Add install/uninstall sub-command (elastic#21206) [Filebeat][httpjson] Make httpjson use cursor input when using date cursor (elastic#20751) feat: prepare release pipelines (elastic#21238) Add IP validation to Security module (elastic#21325)
faec
added a commit
that referenced
this pull request
Sep 30, 2020
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 2, 2020
* upstream/master: (27 commits) [Ingest Manager] Split index restrictions into type,dataset, namespace parts (elastic#21406) Update Filebeat module expected logs files (elastic#21454) Edit SQL module docs and fix broken doc structure (elastic#21233) [Ingest Manager] Send snapshot flag together with metadata (elastic#21285) Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447) [JJBB] Use reference repo for fast checkouts (elastic#21410) Add initial skeleton of filestream input (elastic#21427) Initial spec file for apm-server (elastic#21225) [Ingest Manager] Upgrade Action: make source URI optional (elastic#21372) Add field limit check for AWS Cloudtrail flattened fields (elastic#21388) [Winlogbeat] Move winlogbeat javascript processor to libbeat (elastic#21402) ci: pipeline to generate the changelog (elastic#21426) [JJBB] Set shallow cloning to 10 (elastic#21409) docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419) docs: Prepare Changelog for 7.9.2 (elastic#21229) (elastic#21403) fix: mark flaky tests (elastic#21300) fix: use a fixed version of setuptools (elastic#21393) Move Kubernetes events metricset to its own block in reference config (elastic#21407) [libbeat] Enable WriteAheadLimit in the disk queue (elastic#21391) docs: fix apt/yum formatting (elastic#21362) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR activates the previously inert setting
WriteAheadLimit, which limits how many events the disk queue will store in memory waiting to be written to disk (this is to prevent unbounded memory use / provide back-pressure when the beat is under extremely high load and is receiving inputs faster than they can be written to disk).