[libbeat] Fix encoding and file offset issues in the disk queue #26484
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
faec
added
bug
Team:Elastic-Agent
|
Pinging @elastic/agent (Team:Agent) |
botelastic
bot
added
needs_team
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
fearful-symmetry
approved these changes
Jun 24, 2021
urso
reviewed
Jun 24, 2021
| // A test to make sure serialization works correctly on multi-byte characters. | ||
| func TestSerializeMultiByte(t *testing.T) { | ||
| asciiOnly := "{\"name\": \"Momotaro\"}" | ||
| multiBytes := "{\"name\": \"桃太郎\"}" |
There was a problem hiding this comment.
let's create a table driven test and separate the 2 cases into separate tests.
urso
reviewed
Jun 24, 2021
| @@ -173,6 +173,9 @@ outerLoop: | |||
| // This can only happen if the queue is being closed; abort. | |||
| break | |||
| } | |||
| // We're creating a new segment file, set the initial bytes written | |||
| // to the header size. | |||
| curSegmentResponse.bytesWritten = wl.currentSegment.headerSize() | |||
There was a problem hiding this comment.
This looks like a bug fix. Did we introduce the problem when moving to absolute adresses, or does it fix an issue we have had before? In the later case we might want to add a changelog entry (and maybe a separate PR).
There was a problem hiding this comment.
No, this was just an oversight in the previous (absolute address) PR, so it hasn't hit a release.
urso
approved these changes
Jun 28, 2021
mergify bot
pushed a commit
that referenced
this pull request
Jun 28, 2021
(cherry picked from commit a3b447a)
faec
added a commit
that referenced
this pull request
Jun 28, 2021
mdelapenya
added a commit
to mdelapenya/beats
that referenced
this pull request
Jun 29, 2021
* master: Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545) [Heartbeat] add screenshots config to synthetics (elastic#26455) [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474) Remove all docs about Beats central management (elastic#26399) update data.json for gcp billing (elastic#26506) Skip x-pack metricbeat tests (elastic#26537) [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529) Add changelog entry for elastic#26224 (elastic#26531) Add inttests for the x-pack/metricbeat on a PR/branches basis (elastic#26526) Suppress too many errors (elastic#26224) Fix master's linting issue (elastic#26517) [libbeat] Fix encoding and file offset issues in the disk queue (elastic#26484) Add log_group_name_prefix config option for aws-cloudwatch input (elastic#26187) Update shared-deduplication.asciidoc (elastic#26492) Add Recorded Future support to threatintel module (elastic#26481)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR fixes #26335 (incorrect decoding of multi-byte characters) by changing the disk queue serialization format to CBOR rather than JSON. It also fixes some file position calculations that weren't properly updated along with the schema change in #22970.
There are no user-visible changes in this PR other than the queue working correctly on new segment files; the underlying schema is altered, but files created by 7.13 or earlier will deserialize with JSON as before.
Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.