Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create (X-Pack Monitoring) stats metricset for Kibana module #7525

Merged
merged 66 commits into from
Jul 19, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
20c1e82
WIP: Starting to create (X-Pack Monitoring) stats metricset for Kiban…
ycombinator Jul 6, 2018
1dfb4da
Fleshing out some TODOs
ycombinator Jul 6, 2018
9bf3354
Marking optional fields
ycombinator Jul 7, 2018
33bc9f5
Rename variable to be more accurate
ycombinator Jul 7, 2018
95f975a
Simplifying schema
ycombinator Jul 7, 2018
5f18859
Updating schema
ycombinator Jul 7, 2018
a965a94
Refactoring out common conversions
ycombinator Jul 7, 2018
f9eafc2
Updating test fixture
ycombinator Jul 7, 2018
10d1c33
Fixing syntax error
ycombinator Jul 9, 2018
0d4ac92
Populating fields.yml
ycombinator Jul 10, 2018
83e5afa
Updating test fixture
ycombinator Jul 10, 2018
da32edf
Only pass in intervalMs instead of entire metricset struct
ycombinator Jul 10, 2018
9be7eff
Add unit test for data_xpack.go
ycombinator Jul 10, 2018
b347c68
Fixing collisions in function names
ycombinator Jul 11, 2018
6a56f54
Updating fields.go
ycombinator Jul 11, 2018
92e1617
Removing external_bytes field (for now)
ycombinator Jul 11, 2018
76f4ea9
Fixing schema conversion syntax
ycombinator Jul 11, 2018
c214ddf
Using the new xpack helper package
ycombinator Jul 13, 2018
50be6f4
Fixing type assertion + adding type conversion
ycombinator Jul 13, 2018
d0c7de9
Adapting to slight changes in Kibana API response
ycombinator Jul 13, 2018
e99d40c
Adjust usage structure to meet X-Pack Monitoring docs expectations
ycombinator Jul 16, 2018
b993ebe
Better naming of field
ycombinator Jul 16, 2018
7adc668
Removing sockets field
ycombinator Jul 17, 2018
5da8e34
Rename _all field to all
ycombinator Jul 17, 2018
6a23350
Updating test fixture
ycombinator Jul 17, 2018
a395b92
Updating field names
ycombinator Jul 17, 2018
a506aa9
Fixing config parsing + logging
ycombinator Jul 17, 2018
c2292bd
Adding file
ycombinator Jul 17, 2018
9315247
Fixing index name
ycombinator Jul 17, 2018
8760c91
Updating test fixture and corresponding code
ycombinator Jul 17, 2018
aa6ef8e
Making response_times.avg_ms optional as it is sometimes not reported
ycombinator Jul 17, 2018
6652ba5
Snake casing API response keys
ycombinator Jul 17, 2018
9417071
Removing os field (since they can be covered by the system module)
ycombinator Jul 17, 2018
55da81c
Removing usage fields from non-monitoring events
ycombinator Jul 17, 2018
24774e6
Regenerating data.json
ycombinator Jul 17, 2018
bcab559
Updating unit tests for xpack helper
ycombinator Jul 17, 2018
16b953d
Updating test fixture and corresponding code
ycombinator Jul 17, 2018
97f43a2
Adding config setting for base path
ycombinator Jul 17, 2018
ddb848e
More robust implementation of AddBasePath method
ycombinator Jul 17, 2018
c7c2cae
Updating docs
ycombinator Jul 17, 2018
3ee0678
Updating kibana module yaml file
ycombinator Jul 17, 2018
7e4a20d
Fixing up field names
ycombinator Jul 17, 2018
0f5f90b
Fixing field names
ycombinator Jul 18, 2018
ec62c12
Adding new reporting usage fields
ycombinator Jul 18, 2018
490b076
Forgot to update the test fixture
ycombinator Jul 18, 2018
43881aa
Alternative basepath implementation
ycombinator Jul 18, 2018
986fc61
Fixing type of response time fields
ycombinator Jul 18, 2018
60fa99d
Add error checking for type assertions
ycombinator Jul 18, 2018
03b3b35
Moving Kibana metadata fields into top-level
ycombinator Jul 18, 2018
8a3feee
Adding process.pid field
ycombinator Jul 18, 2018
97318e0
Making response_times.max_ms optional to match response_times.avg_ms
ycombinator Jul 18, 2018
b6ac522
Removing extraneous newline
ycombinator Jul 18, 2018
6b4e884
Making response_times.max_ms optional to match response_times.avg_ms
ycombinator Jul 18, 2018
d251761
Marking optional fields as optional
ycombinator Jul 18, 2018
0b8dc9a
Remove redundant else
ycombinator Jul 19, 2018
482f5c7
Remove BasePath from Kibana config struct
ycombinator Jul 19, 2018
d3e54c3
Use path.Join instead of hardcoding "/"
ycombinator Jul 19, 2018
a9d6295
Adding variations of basepath to unit test
ycombinator Jul 19, 2018
b75280d
Renaming field host -> host.name
ycombinator Jul 19, 2018
ec954b9
Cast to int64 instead of int
ycombinator Jul 19, 2018
d0a6df7
Renaming field cluster_uuid -> elasticsearch.cluster.id
ycombinator Jul 19, 2018
3bf76e5
Moving field elasticsearch.cluster.id to root level of event
ycombinator Jul 19, 2018
6398705
Moving field process.pid to root level of event
ycombinator Jul 19, 2018
8c54595
Adding unit test for ReportErrorForMissingField helper function
ycombinator Jul 19, 2018
a9b601f
Removing unused field
ycombinator Jul 19, 2018
f011ff7
Adding comment about hack
ycombinator Jul 19, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 57 additions & 100 deletions metricbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6681,259 +6681,216 @@ Kibana stats and run-time metrics.



*`kibana.stats.cluster_uuid`*::
*`kibana.stats.uuid`*::
+
--
type: keyword

UUID of the Elasticsearch cluster to which Kibana connects.
Kibana instance UUID


--

*`kibana.stats.name`*::
+
--
type: keyword
type: text

Kibana instance name.
Kibana instance name


--

*`kibana.stats.uuid`*::
*`kibana.stats.index`*::
+
--
type: keyword

Kibana instance uuid.
Name of Kibana's internal index


--

*`kibana.stats.version.number`*::
*`kibana.stats.host.name`*::
+
--
type: keyword

Kibana version number.
Kibana instance hostname


--

*`kibana.stats.status.overall.state`*::
*`kibana.stats.transport_address`*::
+
--
type: keyword

Kibana overall state.
Kibana server's hostname and port


--

[float]
== process fields

Kibana process metrics.



[float]
== mem fields

Memory usage metrics of the Kibana instance.



*`kibana.stats.process.mem.heap.max.bytes`*::
*`kibana.stats.version`*::
+
--
type: long

format: bytes
type: keyword

Total amount of heap memory used by V8.
Kibana version


--

*`kibana.stats.process.mem.heap.used.bytes`*::
*`kibana.stats.snapshot`*::
+
--
type: long

format: bytes
type: boolean

Amount of memory in use by V8.
Whether the Kibana build is a snapshot build


--

*`kibana.stats.process.mem.resident_set_size.bytes`*::
*`kibana.stats.status`*::
+
--
type: long

format: bytes
type: keyword

The amount of space occupied in main memory for the process. Includes heap, code segment, and stack.
Kibana instance's health status


--

*`kibana.stats.process.mem.external.bytes`*::
*`kibana.stats.concurrent_connections`*::
+
--
type: long

format: bytes

Memory usage of C++ objects bound to JavaScript objects managed by V8.
Number of client connections made to the server. Note that browsers can send multiple simultaneous connections to request mulitple server assets at once, and they can re-use established connections.


--

*`kibana.stats.process.pid`*::
+
--
type: long
[float]
== process fields

Process ID of the Kibana instance.
Process metrics


--

*`kibana.stats.process.uptime.ms`*::
*`kibana.stats.process.event_loop_delay.ms`*::
+
--
type: long
type: scaled_float

Amount of time that the Kibana process has been running in milliseconds.
Event loop delay in milliseconds


--

[float]
== response_times fields
== memory.heap fields

HTTP Server response time metrics
Process heap metrics



*`kibana.stats.response_times.avg.ms`*::
*`kibana.stats.process.memory.heap.total.bytes`*::
+
--
type: long

Accumulated averages for response times, for all responses in a 5-second time window.
format: bytes

Total heap allocated to process in bytes


--

*`kibana.stats.response_times.max.ms`*::
*`kibana.stats.process.memory.heap.used.bytes`*::
+
--
type: long

Accumulated maximums for response times, for all responses in a 5-second time window.


--

[float]
== requests fields

HTTP Server request metrics



*`kibana.stats.requests.status_codes`*::
+
--
type: object
format: bytes

Key-value pairs for each status code sent by the server, and the number of times it sent that code.
Heap used by process in bytes


--

*`kibana.stats.requests.total`*::
*`kibana.stats.process.memory.heap.size_limit.bytes`*::
+
--
type: long

Total number of requests sent by the server.


--

*`kibana.stats.requests.disconnects`*::
+
--
type: long
format: bytes

Total number of client disconnects encountered by the server.
Max. old space size allocated to Node.js process, in bytes


--

*`kibana.stats.concurrent_connections`*::
*`kibana.stats.process.memory.heap.uptime.ms`*::
+
--
type: long

Number of client connections made to the server. Note that browsers can send multiple simultaneous connections to request multiple server assets at once, and they can re-use established connections.
Uptime of process in milliseconds


--

[float]
== sockets fields
== request fields

HTTP Web Sockets metrics
Request count metrics



[float]
== http fields
*`kibana.stats.request.disconnects`*::
+
--
type: long

Web Sockets over plaintext HTTP
Number of requests that were disconnected


--

*`kibana.stats.sockets.http.total`*::
*`kibana.stats.request.total`*::
+
--
type: long

Number of HTTP web socket connections established
Total number of requests


--

[float]
== https fields
== response_time fields

Web Sockets over encrypted HTTPS
Response times metrics



*`kibana.stats.sockets.https.total`*::
*`kibana.stats.response_time.avg.ms`*::
+
--
type: long

Number of HTTPS web socket connections established
Average response time in milliseconds


--

*`kibana.stats.event_loop_delay`*::
*`kibana.stats.response_time.max.ms`*::
+
--
type: long

Node event loop delay calculated with internal benchmarking.
Maximum response time in milliseconds


--
Expand Down
1 change: 1 addition & 0 deletions metricbeat/docs/modules/kibana.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ metricbeat.modules:
metricsets: ["status"]
period: 10s
hosts: ["localhost:5601"]
basepath: ""
enabled: true
----

Expand Down
4 changes: 1 addition & 3 deletions metricbeat/helper/xpack/xpack.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ package xpack

import (
"fmt"
"time"
)

// Product supported by X-Pack Monitoring
Expand Down Expand Up @@ -57,8 +56,7 @@ func (p Product) String() string {
// MakeMonitoringIndexName method returns the name of the monitoring index for
// a given product { elasticsearch, kibana, logstash, beats }
func MakeMonitoringIndexName(product Product) string {
today := time.Now().UTC().Format("2006.01.02")
const version = "6"

return fmt.Sprintf(".monitoring-%v-%v-mb-%v", product, version, today)
return fmt.Sprintf(".monitoring-%v-%v-mb", product, version)
}
11 changes: 4 additions & 7 deletions metricbeat/helper/xpack/xpack_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,11 @@ package xpack
import (
"fmt"
"testing"
"time"

"github.com/stretchr/testify/assert"
)

func TestMakeMonitoringIndexName(t *testing.T) {
today := time.Now().UTC().Format("2006.01.02")

tests := []struct {
Name string
Product Product
Expand All @@ -36,22 +33,22 @@ func TestMakeMonitoringIndexName(t *testing.T) {
{
"Elasticsearch monitoring index",
Elasticsearch,
fmt.Sprintf(".monitoring-es-6-mb-%v", today),
".monitoring-es-6-mb",
},
{
"Kibana monitoring index",
Kibana,
fmt.Sprintf(".monitoring-kibana-6-mb-%v", today),
".monitoring-kibana-6-mb",
},
{
"Logstash monitoring index",
Logstash,
fmt.Sprintf(".monitoring-logstash-6-mb-%v", today),
".monitoring-logstash-6-mb",
},
{
"Beats monitoring index",
Beats,
fmt.Sprintf(".monitoring-beats-6-mb-%v", today),
".monitoring-beats-6-mb",
},
}

Expand Down
Loading