Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove agent.hostname field from libbeat #18328

Merged
merged 4 commits into from
May 12, 2020
Merged

Remove agent.hostname field from libbeat #18328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4af663b
Remove agent.hostname field from libbeat
andrewkroh May 6, 2020
4f17af4
Fix beat.hostname alias
andrewkroh May 7, 2020
112ae67
Unmark alias as migration
andrewkroh May 11, 2020
8bb5ff7
Keep deprecation field description
andrewkroh May 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Change aws_elb autodiscover provider field name from elb_listener.* to aws.elb.*. {issue}16219[16219] {pull}16402{16402}
- Remove `AddDockerMetadata` and `AddKubernetesMetadata` processors from the `script` processor. They can still be used as normal processors in the configuration. {issue}16349[16349] {pull}16514[16514]
- Introduce APM libbeat instrumentation, active when running the beat with ELASTIC_APM_ACTIVE=true. {pull}17938[17938]
- Remove the non-ECS `agent.hostname` field. Use the `agent.name` or `agent.id` fields for an identifier. {issue}16377[16377] {pull}18328[18328]

*Auditbeat*

Expand Down
8 changes: 5 additions & 3 deletions auditbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2458,10 +2458,12 @@ Contains common beat fields available in all event types.
*`agent.hostname`*::
+
--
Deprecated - use agent.name or agent.id to identify an agent. Hostname of the agent.
Deprecated - use agent.name or agent.id to identify an agent.


type: keyword
type: alias

alias to: agent.name

--

Expand Down Expand Up @@ -2498,7 +2500,7 @@ alias to: host.name
--
type: alias

alias to: agent.hostname
alias to: agent.name

--

Expand Down
2 changes: 1 addition & 1 deletion auditbeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion auditbeat/module/auditd/_meta/kibana/7/dashboard/auditbeat-kernel-executions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@
{
"attributes": {
"columns": [
"agent.hostname",
"agent.name",
"process.args",
"auditd.summary.actor.primary",
"auditd.summary.actor.secondary",
Expand Down
2 changes: 1 addition & 1 deletion auditbeat/module/auditd/_meta/kibana/7/dashboard/auditbeat-kernel-overview.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@
{
"attributes": {
"columns": [
"agent.hostname",
"agent.name",
"auditd.summary.actor.primary",
"auditd.summary.actor.secondary",
"event.action",
Expand Down
6 changes: 3 additions & 3 deletions auditbeat/module/auditd/_meta/kibana/7/dashboard/auditbeat-kernel-sockets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,7 +451,7 @@
{
"attributes": {
"columns": [
"agent.hostname",
"agent.name",
"auditd.summary.how",
"auditd.summary.object.primary",
"auditd.summary.object.secondary",
Expand Down Expand Up @@ -566,7 +566,7 @@
{
"attributes": {
"columns": [
"agent.hostname",
"agent.name",
"auditd.summary.how",
"auditd.summary.object.primary",
"auditd.summary.object.secondary",
Expand Down Expand Up @@ -673,7 +673,7 @@
{
"attributes": {
"columns": [
"agent.hostname",
"agent.name",
"auditd.summary.how",
"auditd.summary.object.primary",
"auditd.summary.object.secondary",
Expand Down
8 changes: 5 additions & 3 deletions filebeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3112,10 +3112,12 @@ Contains common beat fields available in all event types.
*`agent.hostname`*::
+
--
Deprecated - use agent.name or agent.id to identify an agent. Hostname of the agent.
Deprecated - use agent.name or agent.id to identify an agent.


type: keyword
type: alias

alias to: agent.name

--

Expand Down Expand Up @@ -3152,7 +3154,7 @@ alias to: host.name
--
type: alias

alias to: agent.hostname
alias to: agent.name

--

Expand Down
2 changes: 1 addition & 1 deletion filebeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions filebeat/module/osquery/_meta/kibana/7/dashboard/osquery-rootkit.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@
"id": "1",
"params": {
"customLabel": "Hosts",
"field": "agent.hostname"
"field": "agent.name"
},
"schema": "metric",
"type": "cardinality"
Expand Down Expand Up @@ -185,7 +185,7 @@
"columns": [
"osquery.result.name",
"osquery.result.columns.path",
"agent.hostname"
"agent.name"
],
"description": "",
"hits": 0,
Expand Down
4 changes: 2 additions & 2 deletions filebeat/module/santa/_meta/kibana/7/dashboard/filebeat-santa-log-overview.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@
"id": "1",
"params": {
"customLabel": "Hosts Reporting",
"field": "agent.hostname"
"field": "agent.name"
},
"schema": "metric",
"type": "cardinality"
Expand Down Expand Up @@ -373,7 +373,7 @@
{
"attributes": {
"columns": [
"agent.hostname",
"agent.name",
"process.executable",
"user.name",
"certificate.common_name"
Expand Down
25 changes: 21 additions & 4 deletions filebeat/tests/system/test_fields.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,9 @@ def test_custom_fields_under_root(self):
assert doc["timestamp"] == 2
assert "fields" not in doc

def test_beat_fields(self):
def test_agent_name_custom(self):
"""
Checks that it's possible to set a custom shipper name. Also
tests that agent.hostname has values.
Checks that it's possible to set a custom agent name.
"""
self.render_config_template(
path=os.path.abspath(self.working_dir) + "/test.log",
Expand All @@ -81,5 +80,23 @@ def test_beat_fields(self):
doc = output[0]
assert doc["host.name"] == "testShipperName"
assert doc["agent.name"] == "testShipperName"
assert doc["agent.hostname"] == socket.gethostname()
assert "fields" not in doc

def test_agent_name_default(self):
"""
Checks that agent.name defaults to the hostname.
"""
self.render_config_template(
path=os.path.abspath(self.working_dir) + "/test.log",
)

with open(self.working_dir + "/test.log", "w") as f:
f.write("test message\n")

filebeat = self.start_beat()
self.wait_until(lambda: self.output_has(lines=1))
filebeat.check_kill_and_wait()

output = self.read_output()
doc = output[0]
assert doc["agent.name"] == socket.gethostname()
2 changes: 1 addition & 1 deletion filebeat/tests/system/test_modules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ def _test_expected_events(self, test_file, objects):

def clean_keys(obj):
# These keys are host dependent
host_keys = ["host.name", "agent.name", "agent.hostname", "agent.type", "agent.ephemeral_id", "agent.id"]
host_keys = ["host.name", "agent.name", "agent.type", "agent.ephemeral_id", "agent.id"]
# The create timestamps area always new
time_keys = ["event.created"]
# source path and agent.version can be different for each run
Expand Down
8 changes: 5 additions & 3 deletions heartbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,12 @@ Contains common beat fields available in all event types.
*`agent.hostname`*::
+
--
Deprecated - use agent.name or agent.id to identify an agent. Hostname of the agent.
Deprecated - use agent.name or agent.id to identify an agent.


type: keyword
type: alias

alias to: agent.name

--

Expand Down Expand Up @@ -80,7 +82,7 @@ alias to: host.name
--
type: alias

alias to: agent.hostname
alias to: agent.name

--

Expand Down
2 changes: 1 addition & 1 deletion heartbeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 5 additions & 3 deletions journalbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,12 @@ Contains common beat fields available in all event types.
*`agent.hostname`*::
+
--
Deprecated - use agent.name or agent.id to identify an agent. Hostname of the agent.
Deprecated - use agent.name or agent.id to identify an agent.


type: keyword
type: alias

alias to: agent.name

--

Expand Down Expand Up @@ -73,7 +75,7 @@ alias to: host.name
--
type: alias

alias to: agent.hostname
alias to: agent.name

--

Expand Down
2 changes: 1 addition & 1 deletion journalbeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions libbeat/_meta/fields.common.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
Contains common beat fields available in all event types.
fields:
- name: agent.hostname
type: keyword
type: alias
path: agent.name
description: >
Deprecated - use agent.name or agent.id to identify an agent.
Hostname of the agent.

- name: beat.timezone
type: alias
Expand All @@ -28,7 +28,7 @@

- name: beat.hostname
type: alias
path: agent.hostname
path: agent.name
migration: true

- name: timeseries.instance
Expand Down
3 changes: 0 additions & 3 deletions libbeat/publisher/processing/default.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,9 +149,6 @@ func WithAgentMeta() modifier {
"name": info.Hostname,
"type": info.Beat,
"version": info.Version,
// hostname is deprecated. To be removed for 8.0. It's not in ECS.
// See https://github.com/elastic/beats/issues/16377.
"hostname": info.Hostname,
}
if info.Name != "" {
metadata["name"] = info.Name
Expand Down
2 changes: 0 additions & 2 deletions libbeat/publisher/processing/default_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,6 @@ func TestProcessorsConfigs(t *testing.T) {
},
"agent": common.MapStr{
"ephemeral_id": "123e4567-e89b-12d3-a456-426655440000",
"hostname": "test.host.name",
"name": "test.host.name",
"id": "123e4567-e89b-12d3-a456-426655440001",
"type": "test",
Expand All @@ -210,7 +209,6 @@ func TestProcessorsConfigs(t *testing.T) {
},
"agent": common.MapStr{
"ephemeral_id": "123e4567-e89b-12d3-a456-426655440000",
"hostname": "test.host.name",
"name": "other.test.host.name",
"id": "123e4567-e89b-12d3-a456-426655440001",
"type": "test",
Expand Down
2 changes: 1 addition & 1 deletion libbeat/tests/system/beat/beat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@


BEAT_REQUIRED_FIELDS = ["@timestamp",
"agent.type", "agent.hostname", "agent.version"]
"agent.type", "agent.name", "agent.version"]

INTEGRATION_TESTS = os.environ.get('INTEGRATION_TESTS', False)

Expand Down
8 changes: 5 additions & 3 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4631,10 +4631,12 @@ Contains common beat fields available in all event types.
*`agent.hostname`*::
+
--
Deprecated - use agent.name or agent.id to identify an agent. Hostname of the agent.
Deprecated - use agent.name or agent.id to identify an agent.


type: keyword
type: alias

alias to: agent.name

--

Expand Down Expand Up @@ -4671,7 +4673,7 @@ alias to: host.name
--
type: alias

alias to: agent.hostname
alias to: agent.name

--

Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/docker/_meta/kibana/7/dashboard/Metricbeat-docker-overview.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@
"id": "2",
"params": {
"customLabel": "Hosts",
"field": "agent.hostname",
"field": "agent.name",
"order": "desc",
"orderBy": "1",
"size": 5
Expand Down
8 changes: 5 additions & 3 deletions packetbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,10 +452,12 @@ Contains common beat fields available in all event types.
*`agent.hostname`*::
+
--
Deprecated - use agent.name or agent.id to identify an agent. Hostname of the agent.
Deprecated - use agent.name or agent.id to identify an agent.


type: keyword
type: alias

alias to: agent.name

--

Expand Down Expand Up @@ -492,7 +494,7 @@ alias to: host.name
--
type: alias

alias to: agent.hostname
alias to: agent.name

--

Expand Down
2 changes: 1 addition & 1 deletion packetbeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions packetbeat/tests/system/packetbeat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@
from beat.beat import Proc

TRANS_REQUIRED_FIELDS = ["@timestamp", "type", "status",
"agent.type", "agent.hostname", "agent.version",
"agent.type", "agent.name", "agent.version",
"event.kind", "event.category", "event.dataset", "event.start",
"source.ip", "destination.ip",
"client.ip", "server.ip",
"network.type", "network.transport", "network.community_id",
]

FLOWS_REQUIRED_FIELDS = ["@timestamp", "type",
"agent.type", "agent.hostname", "agent.version",
"agent.type", "agent.name", "agent.version",
"event.kind", "event.category", "event.dataset", "event.action", "event.start", "event.end", "event.duration",
"source.ip", "destination.ip",
"flow.id",
Expand Down
1 change: 0 additions & 1 deletion packetbeat/tests/system/test_0099_golden_files.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ def clean_keys(obj):
keys = [
"@metadata.version",
"agent.ephemeral_id",
"agent.hostname",
"agent.id",
"agent.name",
"agent.type",
Expand Down
17 changes: 17 additions & 0 deletions script/ecs-migration-8x.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# The ECS migration file contains the information about all the Beats fields
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

# which were changed between 7.x and 8.0.
#
# This file will be used to generate breaking change documentation and migrate
# Kibana dashboards. See script/renamed_fields.py and script/kibana-migration.py.
#
# The format of the file is as following:
#
# - from: <original-name-in-7x> # Field name in 7.x.
# to: <new-name-in-8x> # Field name in 8.0.
# alias: <true|false> # True if a migration alias exists in fields.yml.
# beat: <beatname> # Omit for libbeat fields.

## Libbeat
- from: agent.hostname
to: agent.name
alias: true
Loading