Migrate beat.* to agent.* #8873

ruflin · 2018-11-01T09:54:05Z

The beat.* fields map to agent.* fields in ECS. This PR converts the following fileds:

beat.hostname -> agent.hostname
beat.version -> agent.version

agent.type is added to indicate which Beat the data comes from. The field beat.name is converted to agent.name but is only set when it is not the same as the hostname. We already feed the field at the moment into host.name. The main discussion is which field we use by default in our dashboards moving forward.

Changed:

Remove old fields from field.yml
Updated changelog
Updated tests and generated files

ruflin · 2018-11-01T09:58:49Z

dev-tools/ecs-migration.yml

+#
+
+
+- from: beat.name


@bleskes This is the file I have in mind to track the migration of the fields so it can be used later to do automatic migrations.

What's the meaning of this structure? I'm wondering about the direction of the alias this is suggesting:

Does it mean field offset is type alias, and points to the data in log.offset?

This will be appropriate for 7.x, for backwards compatibility

Does it mean field log.offset is type alias, and points to the data in offset?

This would be appropriate if we want to start preparing for ECS in 6.x

Since we may actually do both, I think we should rename the keys of the structures slightly.

I currently read them as "old field is in from" and "new field is in to". However when you read with the strategy in mind, it reads like "alias from offset to log.offset", which is confusing.

What do you think about the following?

- old: offset new: log.offset strategy: alias # or copy_to

I messed up the meaning of which strategy made sense in which version. Updated my comment above :)

Not a blocker for this PR, just to be clear. Just maybe a thing to rethink in a separate PR

The structure can still change, what is important right now is that we have a structure way of documenting it. We can focus on naming later.

ruflin · 2018-11-01T09:59:34Z

libbeat/publisher/pipeline/module.go

@@ -80,8 +80,8 @@ func Load(
 		Annotations: Annotations{
 			Event: config.EventMetadata,
 			Builtin: common.MapStr{
-				"beat": common.MapStr{
-					"name":     name,
+				"agent": common.MapStr{


@graphaelli I think we should must make it possible for the apm-server to overwrite these fields with the agent information. Meaning we should only set them if they are not already in the event.

Agreed, makes sense.

@graphaelli Will make this happen in a follow up PR. Marked it in the ECS tracking issue: #8655

This file should help to track the migration of all files to ECS and then have potential automation on top to convert dashboards and other files which use the fields. This file is extracted from elastic#8873 to make it available in multiple PRs.

This file should help to track the migration of all files to ECS and then have potential automation on top to convert dashboards and other files which use the fields. This file is extracted from #8873 to make it available in multiple PRs.

webmat

LGTM

webmat

Noticed a few discrepancies

webmat · 2018-11-05T18:44:50Z

CHANGELOG.asciidoc

@@ -18,6 +18,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]
 *Auditbeat*

 - Use `initial_scan` action for new paths. {pull}7954[7954]
+- Rename beat.name to beat.type, beat.hostname to agent.hostname, beat.version to agent.version.


Should "Rename beat.name to beat.type" read "Rename beat.name to agent.type"?

Fixed, thanks.

webmat · 2018-11-05T18:45:43Z

dev-tools/ecs-migration.yml

+
+
+- from: beat.name
+  to: agent.name


This is not aligned with the changelog. Shouldn't "beat.name" become "beat.type"?

yes, agent.type

Haha, I messed up my correction. Yes, "agent.type" :-)

webmat · 2018-11-05T18:47:02Z

filebeat/tests/system/test_fields.py

@@ -79,6 +79,6 @@ def test_beat_fields(self):

        output = self.read_output()
        doc = output[0]
-        assert doc["beat.name"] == "testShipperName"
-        assert doc["beat.hostname"] == socket.gethostname()
+        assert doc["host.name"] == "testShipperName"


This should be updated to "host.hostname", no?

We already have host.name in Beats today since 6.4 and is what is used at the moment everywhere.

webmat · 2018-11-05T18:47:27Z

filebeat/tests/system/test_modules.py

@@ -190,7 +190,7 @@ def _test_expected_events(self, test_file, objects):

 def clean_keys(obj):
    # These keys are host dependent
-    host_keys = ["host.name", "beat.hostname", "beat.name"]
+    host_keys = ["host.name", "agent.hostname", "agent.type"]


Same: "host.name" => "host.hostname"

So the intention is to stick to "host.name" in 6.x and move to "host.hostname" in 7.x?

We still need to figure that one out. This PR keeps the same behaviour as we had.

The beat.* fields map to agent.* fields in ECS. This PR converts the following fileds: * beat.hostname -> agent.hostname * beat.version -> agent.version `agent.type` is added to indicate which Beat the data comes from. The field `beat.name` is converted to `agent.name` but is only set when it is not the same as the hostname. We already feed the field at the moment into host.name. The main discussion is which field we use by default in our dashboards moving forward. Changed: * Remove old fields from field.yml * Updated changelog * Updated tests and generated files

andrewkroh · 2018-11-14T14:31:57Z

@ruflin @webmat We don't have agent.hostname listed in ECS. Is that an oversight? Should we add it?

webmat · 2018-11-14T18:16:23Z

Yeah it's an oversight. It was mentioned in #8873, but I promptly forgot about that :-) So I created issue elastic/ecs#178 to make sure it gets addressed.

This is not a concept that's only useful to Beats, I think it also belongs in ECS. See the ECS issue for details, or to discuss the idea.

The beat.* fields map to agent.* fields in ECS. This PR converts the following fileds: * beat.hostname -> agent.hostname * beat.version -> agent.version `agent.type` is added to indicate which Beat the data comes from. The field `beat.name` is converted to `agent.name` but is only set when it is not the same as the hostname. We already feed the field at the moment into host.name. The main discussion is which field we use by default in our dashboards moving forward. Changed: * Remove old fields from field.yml * Updated changelog * Updated tests and generated files

ruflin added in progress Pull request is currently in progress. ecs labels Nov 1, 2018

ruflin commented Nov 1, 2018

View reviewed changes

ruflin mentioned this pull request Nov 1, 2018

Add tracking file for ECS migration #8878

Merged

ruflin force-pushed the ecs-migration-kickoff branch 2 times, most recently from 2326942 to 92ad691 Compare November 5, 2018 13:00

webmat approved these changes Nov 5, 2018

View reviewed changes

webmat reviewed Nov 5, 2018

View reviewed changes

ruflin force-pushed the ecs-migration-kickoff branch from 92ad691 to 2d224c8 Compare November 6, 2018 08:11

ruflin changed the title [WIP] Migrate beat.* to agent.* Migrate beat.* to agent.* Nov 6, 2018

ruflin mentioned this pull request Nov 6, 2018

Transition Beats to ECS #8655

Closed

ruflin force-pushed the ecs-migration-kickoff branch from 2d224c8 to 5a9651a Compare November 6, 2018 08:27

ph approved these changes Nov 6, 2018

View reviewed changes

ruflin force-pushed the ecs-migration-kickoff branch from 865c7c5 to a4b28d9 Compare November 6, 2018 15:27

ruflin added review and removed in progress Pull request is currently in progress. labels Nov 6, 2018

ruflin merged commit 1767034 into elastic:master Nov 6, 2018

ruflin deleted the ecs-migration-kickoff branch November 6, 2018 18:26

webmat mentioned this pull request Nov 14, 2018

Add agent.hostname elastic/ecs#178

Closed

webmat mentioned this pull request Nov 19, 2018

[ML] Edits to fields used in auditbeat module configurations elastic/kibana#25866

Merged

cwurm mentioned this pull request Dec 15, 2018

[Auditbeat] Add system module #9546

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate beat.* to agent.* #8873

Migrate beat.* to agent.* #8873

ruflin commented Nov 1, 2018 •

edited

Loading

ruflin Nov 1, 2018

bleskes Nov 5, 2018

webmat Nov 6, 2018 •

edited

Loading

webmat Nov 6, 2018

webmat Nov 6, 2018

ruflin Nov 6, 2018

ruflin Nov 1, 2018

graphaelli Nov 1, 2018

ruflin Nov 5, 2018

webmat left a comment

webmat left a comment

webmat Nov 5, 2018

ruflin Nov 6, 2018

webmat Nov 5, 2018

ruflin Nov 6, 2018

webmat Nov 6, 2018

webmat Nov 5, 2018

ruflin Nov 6, 2018

webmat Nov 5, 2018

ruflin Nov 6, 2018

webmat Nov 6, 2018

ruflin Nov 6, 2018

andrewkroh commented Nov 14, 2018

webmat commented Nov 14, 2018

Migrate beat.* to agent.* #8873

Migrate beat.* to agent.* #8873

Conversation

ruflin commented Nov 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat Nov 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Nov 14, 2018

webmat commented Nov 14, 2018

ruflin commented Nov 1, 2018 •

edited

Loading

webmat Nov 6, 2018 •

edited

Loading