[7.x] Add initial support for indexing to data streams (#4409) #4517
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* idxmgmt: add support for data streams
Introduce `apm-server.data_streams.enabled` config,
which will be used by idxmgmt to route events to
data streams based on data_stream.* fields that are
expected to be in each published event.
* _meta/fields.common.yml: add data_stream.* fields
* model: add data_stream.{type,dataset} fields
When transforming model objects into beat.Events,
set the data_stream.{type,dataset} fields. We add
data_stream.namespace elsewhere, using an event
processor.
* beater: handle apm-server.data_streams.enabled
Handle the new apm-server.data_streams.enabled
config:
- when enabled, we add data_stream.namespace to
all published events
- when disabled, we remove data_stream.* fields
from all published events
For now we just set data_stream.namespace to "default".
Later this will be based on the config received from Fleet.
* processor/stream/package_tests: update tests
There is a hack in here to inject data_stream.namespace
in all published events, since the tests do not use the
standard libbeat pipeline code.
* Update approvals
* Add changelog entry
* datastreams: address review comments
* changelogs: clarify feature is experimental
* _meta: specify allowed values for data_stream.type
* model: update datasets
Use a common "apm." prefix, and place the service name last.
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Steps errors
Expand to view the steps failures
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backports the following commits to 7.x: