[winlogbeat] performance improvment; avoid rendering event message twice #39544
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
|
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
intxgo
force-pushed
the
winlogbeat-perf-improvement
branch
from
8ef40e1
to
58762e8
Compare
intxgo
added
Team:Security-Windows Platform
intxgo
added
backport-v8.13.0
|
Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform) |
intxgo
changed the title
intxgo
changed the title
intxgo
removed
backport-v8.8.0
|
The concept SGTM. I've used this pattern elsewhere with Win32/Native APIs - nice perf boost. I'm not qualified to review the golang, however. |
|
LGTM, I wonder if it is feasible to craft an .evtx with such an event to be included in tests |
We already have unit tests with evtx files, but they don't test these methods because these require publisher metadata. |
nfritts
approved these changes
May 15, 2024
v1v
added a commit
to v1v/beats
that referenced
this pull request
May 15, 2024
…-actions * upstream/main: (313 commits) github-action: delete opentelemetry workflow (elastic#39559) updatecli: move to the .github folder and support for signed commits (elastic#39472) Osquerybeat: Add action responses data stream (elastic#39143) [winlogbeat] performance improvment; avoid rendering event message twice (elastic#39544) Fix the AWS SDK dependencies issue causing the "not found, ResolveEndpointV2" error (elastic#39454) x-pack/filebeat/input/cel: add http metrics collection (elastic#39503) build(deps): bump github.com/elastic/elastic-agent-libs from 0.9.4 to 0.9.7 (elastic#39424) Remove unused env vars from pipelines (elastic#39534) [BK] - Remove osx steps from branch execution (elastic#39552) [BK] - Remove certain steps from running for Branches (elastic#39533) Allow dependabot report BK status checks (elastic#39540) Remove hardcoded module definitions in CI (elastic#39506) Explicitly set DOCKER_PULL, RACE_DETECTOR and TEST_COVERAGE for pipelines (elastic#39510) Fixed pipelines formatting (elastic#39513) Update filebeat pipeline to match Jenkins steps (elastic#39261) Add error check to groupToEvents so we don't blindly add error values (elastic#39404) Remove fields not needed for session view in add_session_view processor (elastic#39500) `aws-s3` input: Split S3 poller and SQS reader into explicit input objects (elastic#39353) ci(jenkins): remove post-build notifications (elastic#39483) [DOCS] Add the `read_pipeline` cluster privilege for winlogbeat and the `auto_configure` index privilege to beats documentation (elastic#38534) ...
intxgo
added a commit
that referenced
this pull request
May 16, 2024
…ice (#39544) (#39573) * wineventlog performance improvment; avoid rendering message twice * ignore missing or mismatched parameter values * add comment * changelog * actually increase the buffer (cherry picked from commit d2ebffe) Co-authored-by: Leszek Kubik <39905449+intxgo@users.noreply.github.com>
intxgo
added a commit
that referenced
this pull request
May 16, 2024
…ice (#39544) (#39572) * wineventlog performance improvment; avoid rendering message twice * ignore missing or mismatched parameter values * add comment * changelog * actually increase the buffer (cherry picked from commit d2ebffe) Co-authored-by: Leszek Kubik <39905449+intxgo@users.noreply.github.com>
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
The change increases events-per-second throughput by about 30%.
I have improved the throughput by using a fixed buffer size at first attempt of message parsing.
I thought about adding a config parameter to control the size of the initial buffer but after some research I believe it's not needed. I use the size 16KB already existing in our code base. It's relatively small, but the majority of windows event log messages are even much smaller. Historically the event log was designed to store only a message templates to be filled in with actual strings or values from external resources when viewing.
This PR does not pose a regression risk, related to #35437 The former code was prone to error, on certain Windows releases, by using the out parameter
BufferUsedinstead of relying only on C-style string format when handling the output. The documentation of this parameter is a bit vague, in practice it's only needed whenERROR_INSUFFICIENT_BUFFERis returned.Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Disruptive User Impact
Author's Checklist
How to test this PR locally
Setup a reasonably large VM, for example 8 CPU, 32 GB RAM. Prepare an event log with thousands of entries (or just create empty event log and prepare a tool to quickly produce events).
Prepare winlogbeat.yaml config pointing to the event log, use file output. Configure http statistics endpoint to periodically check the progress. Run winlogbeat.exe with the same config, on the same machine, before and after the change or run them side-by-side. Observe the increased events-per-second throughput.
Related issues
Use cases
Screenshots
Logs