Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce service.type field for all Filebeat modules #10042

Merged
merged 5 commits into from
Jan 15, 2019
Merged

Introduce service.type field for all Filebeat modules #10042

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6887270
Introduce service.type field for all Filebeat modules
ruflin Jan 14, 2019
e0c2db5
add pr to changelog and fix docs
ruflin Jan 14, 2019
9d7b937
remove temporary target fix
ruflin Jan 14, 2019
175e557
Merge branch 'master' into service-type
ruflin Jan 15, 2019
eb3142c
update one more log file
ruflin Jan 15, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Rename a few `mysql.*` fields to map to ECS. {pull}10008[10008]
- Rename a few `nginx.error.*` fields to map to ECS. {pull}10007[10007]
- Filesets with multiple ingest pipelines added in {pull}8914[8914] only work with Elasticsearch >= 6.5.0 {pull}10001[10001]
- Remove service.name from Elastcsearch module. Replace by service.type. {pull}[]

*Heartbeat*

Expand Down Expand Up @@ -112,6 +113,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Support mysql 5.7.22 slowlog starting with time information. {issue}7892[7892] {pull}9647[9647]
- Add support for ssl_request_log in apache2 module. {issue}8088[8088] {pull}9833[9833]
- Add support for iis 7.5 log format. {issue}9753[9753] {pull}9967[9967]
- Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with `service.type` config. {pull}[]

*Heartbeat*

Expand Down
10 changes: 7 additions & 3 deletions filebeat/channel/factory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ type inputOutletConfig struct {
// Output meta data settings
Pipeline string `config:"pipeline"` // ES Ingest pipeline name

ServiceType string `config:"service.type"` // ES Ingest pipeline name
}

// NewOutletFactory creates a new outlet factory for
Expand Down Expand Up @@ -112,10 +113,13 @@ func (f *OutletFactory) Create(p beat.Pipeline, cfg *common.Config, dynFields *c
if config.Fileset != "" {
fields.Put("fileset.name", config.Fileset)
}
if config.ServiceType != "" {
fields.Put("service.type", config.ServiceType)
} else {
fields.Put("service.type", config.Module)
}
if config.Type != "" {
fields["input"] = common.MapStr{
"type": config.Type,
}
fields.Put("input.type", config.Type)
}

client, err := p.ConnectWith(beat.ClientConfig{
Expand Down
1 change: 1 addition & 0 deletions filebeat/module/apache/access/test/ssl-request.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"http.version": "1.1",
"input.type": "log",
"log.offset": 0,
"service.type": "apache",
"source.ip": "172.30.0.119",
"url.original": "/nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_admin_tasks_html%22%2C%22args%22%3A%22%22%7D&nsp=b5c7d5d4b6f7d0cf0c92f9cbdf737f6a5c838218425e6ae21"
}
Expand Down
5 changes: 5 additions & 0 deletions filebeat/module/apache/access/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"http.version": "1.1",
"input.type": "log",
"log.offset": 0,
"service.type": "apache",
"source.ip": "::1",
"url.original": "/favicon.ico",
"user.name": "-"
Expand All @@ -30,6 +31,7 @@
"http.version": "1.1",
"input.type": "log",
"log.offset": 73,
"service.type": "apache",
"source.ip": "192.168.33.1",
"url.original": "/hello",
"user.name": "-",
Expand All @@ -53,6 +55,7 @@
"http.response.status_code": 408,
"input.type": "log",
"log.offset": 238,
"service.type": "apache",
"source.ip": "::1",
"user.name": "-"
},
Expand All @@ -70,6 +73,7 @@
"http.version": "1.1",
"input.type": "log",
"log.offset": 285,
"service.type": "apache",
"source.ip": "172.17.0.1",
"url.original": "/stringpatch",
"user.name": "-",
Expand All @@ -96,6 +100,7 @@
"http.version": "1.1",
"input.type": "log",
"log.offset": 443,
"service.type": "apache",
"source.domain": "monitoring-server",
"url.original": "/status",
"user.name": "-",
Expand Down
5 changes: 4 additions & 1 deletion filebeat/module/apache/error/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"log.level": "error",
"log.offset": 0,
"message": "File does not exist: /var/www/favicon.ico",
"service.type": "apache",
"source.address": "192.168.33.1"
},
{
Expand All @@ -22,7 +23,8 @@
"log.level": "notice",
"log.offset": 99,
"message": "AH00094: Command line: '/usr/local/Cellar/httpd24/2.4.23_2/bin/httpd'",
"process.pid": 11379
"process.pid": 11379,
"service.type": "apache"
},
{
"@timestamp": "2011-09-09T10:42:29.902Z",
Expand All @@ -37,6 +39,7 @@
"message": "File does not exist: /usr/local/apache2/htdocs/favicon.ico",
"process.pid": 35708,
"process.thread.id": 4328636416,
"service.type": "apache",
"source.address": "72.15.99.187"
}
]
6 changes: 4 additions & 2 deletions filebeat/module/auditd/log/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@
"event.module": "auditd",
"fileset.name": "log",
"input.type": "log",
"log.offset": 0
"log.offset": 0,
"service.type": "auditd"
},
{
"@timestamp": "2017-01-31T20:17:14.891Z",
Expand Down Expand Up @@ -51,6 +52,7 @@
"event.module": "auditd",
"fileset.name": "log",
"input.type": "log",
"log.offset": 174
"log.offset": 174,
"service.type": "auditd"
}
]
4 changes: 0 additions & 4 deletions filebeat/module/elasticsearch/audit/config/audit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,3 @@ exclude_files: [".gz$"]
processors:
- add_locale: ~
{{ end }}

fields:
service.name: "elasticsearch"
fields_under_root: true
14 changes: 7 additions & 7 deletions filebeat/module/elasticsearch/audit/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"input.type": "log",
"log.offset": 0,
"message": "[2018-06-19T05:16:15,549] [rest] [authentication_failed] origin_address=[147.107.128.77], principal=[i030648], uri=[/_xpack/security/_authenticate]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:07:52.304Z",
Expand All @@ -30,7 +30,7 @@
"input.type": "log",
"log.offset": 155,
"message": "[2018-06-19T05:07:52,304] [v_VJhjV] [rest] [authentication_failed]\torigin_address=[172.22.0.3], principal=[rado], uri=[/_xpack/security/_authenticate]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:00:15.778Z",
Expand All @@ -48,7 +48,7 @@
"input.type": "log",
"log.offset": 306,
"message": "[2018-06-19T05:00:15,778] [transport] [access_granted] origin_type=[local_node], origin_address=[192.168.1.165], principal=[_xpack_security], action=[indices:data/read/scroll/clear], request=[ClearScrollRequest]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:07:45.544Z",
Expand All @@ -64,7 +64,7 @@
"input.type": "log",
"log.offset": 519,
"message": "[2018-06-19T05:07:45,544] [v_VJhjV] [rest] [anonymous_access_denied]\torigin_address=[172.22.0.3], uri=[/_xpack/security/_authenticate]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:26:27.268Z",
Expand All @@ -80,7 +80,7 @@
"input.type": "log",
"log.offset": 654,
"message": "[2018-06-19T05:26:27,268] [rest] [authentication_failed]\torigin_address=[147.107.128.77], principal=[N078801], uri=[/_xpack/security/_authenticate]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:55:26.898Z",
Expand All @@ -98,7 +98,7 @@
"input.type": "log",
"log.offset": 802,
"message": "[2018-06-19T05:55:26,898] [transport] [access_denied]\torigin_type=[rest], origin_address=[147.107.128.77], principal=[_anonymous], action=[cluster:monitor/main], request=[MainRequest]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-06-19T05:24:15.190Z",
Expand All @@ -116,6 +116,6 @@
"input.type": "log",
"log.offset": 986,
"message": "[2018-06-19T05:24:15,190] [v_VJhjV] [rest] [authentication_failed]\torigin_address=[172.18.0.3], principal=[elastic], uri=[/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip], request_body=[body]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
}
]
4 changes: 0 additions & 4 deletions filebeat/module/elasticsearch/deprecation/config/log.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,3 @@ processors:
- add_locale: ~
{{ end }}

fields:
service.name: "elasticsearch"

fields_under_root: true
8 changes: 4 additions & 4 deletions filebeat/module/elasticsearch/deprecation/test/elasticsearch_deprecation.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"log.level": "WARN",
"log.offset": 0,
"message": "Deprecated field [template] used, replaced by [index_patterns]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-04-23T16:40:13.862Z",
Expand All @@ -23,7 +23,7 @@
"log.level": "WARN",
"log.offset": 137,
"message": "Deprecated field [template] used, replaced by [index_patterns]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-04-23T16:40:14.792Z",
Expand All @@ -36,7 +36,7 @@
"log.level": "WARN",
"log.offset": 274,
"message": "Deprecated field [template] used, replaced by [index_patterns]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2018-04-23T16:40:15.127Z",
Expand All @@ -49,6 +49,6 @@
"log.level": "WARN",
"log.offset": 411,
"message": "Deprecated field [template] used, replaced by [index_patterns]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
}
]
32 changes: 16 additions & 16 deletions ...t/module/elasticsearch/deprecation/test/other_elasticsearch_deprecation.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"log.level": "WARN",
"log.offset": 0,
"message": "Deprecated field [inline] used, expected [source] instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-11-30T13:38:16.941Z",
Expand All @@ -23,7 +23,7 @@
"log.level": "WARN",
"log.offset": 118,
"message": "Deprecated field [inline] used, expected [source] instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-11-30T13:39:28.986Z",
Expand All @@ -36,7 +36,7 @@
"log.level": "WARN",
"log.offset": 236,
"message": "Fielddata access on the _uid field is deprecated, use _id instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-11-30T13:39:36.339Z",
Expand All @@ -49,7 +49,7 @@
"log.level": "WARN",
"log.offset": 362,
"message": "Fielddata access on the _uid field is deprecated, use _id instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-11-30T13:40:49.540Z",
Expand All @@ -62,7 +62,7 @@
"log.level": "WARN",
"log.offset": 488,
"message": "Fielddata access on the _uid field is deprecated, use _id instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-11-30T14:08:37.413Z",
Expand All @@ -75,7 +75,7 @@
"log.level": "WARN",
"log.offset": 614,
"message": "Fielddata access on the _uid field is deprecated, use _id instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-11-30T14:08:37.413Z",
Expand All @@ -88,7 +88,7 @@
"log.level": "WARN",
"log.offset": 740,
"message": "Fielddata access on the _uid field is deprecated, use _id instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-11-30T14:08:46.006Z",
Expand All @@ -101,7 +101,7 @@
"log.level": "WARN",
"log.offset": 866,
"message": "Fielddata access on the _uid field is deprecated, use _id instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-11-30T14:08:46.006Z",
Expand All @@ -114,7 +114,7 @@
"log.level": "WARN",
"log.offset": 992,
"message": "Fielddata access on the _uid field is deprecated, use _id instead",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-12-01T14:05:54.017Z",
Expand All @@ -127,7 +127,7 @@
"log.level": "WARN",
"log.offset": 1118,
"message": "[_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-12-01T14:05:54.019Z",
Expand All @@ -140,7 +140,7 @@
"log.level": "WARN",
"log.offset": 1329,
"message": "[_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-12-01T14:06:52.059Z",
Expand All @@ -153,7 +153,7 @@
"log.level": "WARN",
"log.offset": 1540,
"message": "[_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-12-01T14:46:10.428Z",
Expand All @@ -166,7 +166,7 @@
"log.level": "WARN",
"log.offset": 1751,
"message": "Deprecated aggregation order key [_term] used, replaced by [_key]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-12-04T16:17:18.271Z",
Expand All @@ -179,7 +179,7 @@
"log.level": "WARN",
"log.offset": 1882,
"message": "Deprecated field [template] used, replaced by [index_patterns]",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-12-04T16:17:18.282Z",
Expand All @@ -192,7 +192,7 @@
"log.level": "WARN",
"log.offset": 2019,
"message": "[_default_] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
},
{
"@timestamp": "2017-12-04T16:20:43.248Z",
Expand All @@ -205,6 +205,6 @@
"log.level": "WARN",
"log.offset": 2192,
"message": "[_default_] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type",
"service.name": "elasticsearch"
"service.type": "elasticsearch"
}
]
Loading