Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RFC] wip: data_stream - stage 2 #1145

Merged
merged 8 commits into from
Jan 4, 2021
Merged

[RFC] wip: data_stream - stage 2 #1145

Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
530cd15
wip: moving to stage 2
Nov 23, 2020
f64d01a
wip: added link to PR
Nov 23, 2020
be362a0
wip: added fields.yml
Nov 24, 2020
72f6db8
add file and check if I can push
ruflin Dec 22, 2020
ea69457
add more updates
ruflin Dec 22, 2020
0e6436d
add example event and adjust author
ruflin Dec 23, 2020
c440eff
fix indentation
ruflin Dec 23, 2020
b4f72b9
setting advancement date
ebeahan Jan 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions rfcs/text/0009-data_stream-fields.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# 0009: Data stream fields
<!-- Leave this ID at 0000. The ECS team will assign a unique, contiguous RFC number upon merging the initial stage of this RFC. -->

- Stage: **1 (proposal)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
- Stage: **2 (draft)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
- Date: **2020-11-11** <!-- The ECS team sets this date at merge time. This is the date of the latest stage advancement. -->

When introducing the new indexing strategy for Elastic Agent which uses data streams, we found that adding a few [constant_keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type) fields corresponding to the central components in the new indexing strategy would be advantageous.
Expand Down Expand Up @@ -31,6 +31,20 @@ data_stream.namespace | constant_keyword | A user defined namespace. Namespaces

In the new indexing strategy, the value of the data stream fields combine to the name of the actual data stream in the following manner `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. This means the fields can only contain characters that are valid as part of names of data streams.

The following is the field definitions as a `fields.yml`:

```yml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's instead add these as a separate YML file at rfcs/text/0009/data_stream.yml.

It will require a definition for the field set as well.

The definitions for the fields should instead start from what we had in the table above in the RFC. It's fine to be a bit more terse than in the table though, for example we should only cover the currently expected values for namespace.type, and not mention "we expect values X and Y". The field definitions should also cover the character limitations.

Here's a starting point for the full YML file:

- name: data_stream
  title: Data Stream
  short: TODO
  description: TODO
  fields:

    - name: type
      level: extended
      type: keyword
      example: logs
      description: >
        A description

        with multiple paragraphs

        requires a 'short' description as well.
      short: A short version of the description.
    - name: dataset
# ...

- name: data_stream.type
type: constant_keyword
description: Data stream type.
- name: data_stream.dataset
type: constant_keyword
description: Data stream dataset.
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
```

### Restrictions on values

Due to the fact that the values of the `data_stream` fields make up the data stream name, the restrictions on data stream names also apply to values for the `data_stream` fields. As an example, they cannot include \, /, *, ?, ", <, >, |, ` `. Please see the Elasticsearch reference for [restrictions on index/data stream names](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params). Here follows the _additional_ restrictions imposed on the data stream fields:
Expand Down Expand Up @@ -182,7 +196,7 @@ e.g.:
<!-- An RFC should link to the PRs for each of it stage advancements. -->

* Stage 1: https://github.com/elastic/ecs/pull/980

* Stage 2: https://github.com/elastic/ecs/pull/1145
<!--
* Stage 1: https://github.com/elastic/ecs/pull/NNN
...
Expand Down