Adding version check to Logstash metricsets when xpack.enabled: true is set #12705

ycombinator · 2019-06-27T03:11:56Z

When the logstash module is used with xpack.enabled: true set, Logstash APIs called by the module's metricsets are expected to return Logstash pipeline graphs. Returning pipeline graphs is an enhancement to Logstash APIs that was implemented in 7.3.0 of Logstash. So using the logstash module with xpack.enabled: true set won't work against Logstash nodes < 7.3.0.

This PR adds a version check to the Logstash module code and emits an error message in the Metricbeat logs if the version check fails. This is similar to a version check in the Kibana module.

Testing this PR

Start up a Logstash node (running version < 7.3.0) running one or more pipelines.
Build Metricbeat with this PR:
```
cd metricbeat
mage build
```
Enable the logstash Metricbeat module for Stack Monitoring:
```
metricbeat modules enable logstash-xpack
```
Start Metricbeat:
```
metricbeat -e
```
Check that an error is shown in the Metricbeat logs about the Logstash module not working with Logstash < 7.3.0 when xpack.enabled: true is set.

elasticmachine · 2019-06-27T03:11:58Z

Pinging @elastic/stack-monitoring

cachedout · 2019-06-27T06:23:25Z

Using these instructions I can't seem to make this work.

The first problem is that there is no logstash-xack (I assume this is supposed to be logstash-xpack) but there is only logstash.yml.disabled. So, I used this and modified the file to read as follows:

# Module: logstash
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-logstash.html

- module: logstash
  metricsets:
    - node
    - node_stats
  period: 10s
  hosts: ["http://localhost:9600"]
  #username: "user"
  #password: "secret"
  xpack.enabled: true

I receive the following, no matter whether I set the host to localhost:9600 or http://localhost:9600 or https://localhost:9600:

08:22 $ ./metricbeat -e
2019-06-27T08:22:51.317+0200    INFO    instance/beat.go:606    Home path: [/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat] Config path: [/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat] Data path: [/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/data] Logs path: [/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/logs]
2019-06-27T08:22:51.318+0200    INFO    instance/beat.go:614    Beat ID: 86770118-2dba-44ec-95cd-cbd63561ce78
2019-06-27T08:22:51.318+0200    INFO    [beat]  instance/beat.go:902    Beat info       {"system_info": {"beat": {"path": {"config": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat", "data": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/data", "home": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat", "logs": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/logs"}, "type": "metricbeat", "uuid": "86770118-2dba-44ec-95cd-cbd63561ce78"}}}
2019-06-27T08:22:51.318+0200    INFO    [beat]  instance/beat.go:911    Build info      {"system_info": {"build": {"commit": "d8d98a3ba68bb4046e0896ba7de2d9b78ebccff7", "libbeat": "8.0.0", "time": "2019-06-27T05:45:43.000Z", "version": "8.0.0"}}}
2019-06-27T08:22:51.318+0200    INFO    [beat]  instance/beat.go:914    Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":12,"version":"go1.11.5"}}}
2019-06-27T08:22:51.319+0200    INFO    [beat]  instance/beat.go:918    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-06-19T18:27:43.756208+02:00","name":"Mikes-MacBook-Pro.local","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::14b0:2163:ea86:49ca/64","192.168.0.14/24","fe80::d05d:e5ff:fe25:9dce/64","fe80::4662:a553:1efe:ec38/64","10.0.0.1/24","fe80::a1e:38c9:922a:d512/64","fe80::aede:48ff:fe00:1122/64"],"kernel_version":"18.6.0","mac":["3a:f9:d3:1f:27:0d","38:f9:d3:1f:27:0d","0a:f9:d3:1f:27:0d","d2:5d:e5:25:9d:ce","5e:00:80:f0:b6:01","5e:00:80:f0:b6:00","5e:00:80:f0:b6:05","5e:00:80:f0:b6:04","5e:00:80:f0:b6:01","0a:00:27:00:00:00","0a:00:27:00:00:01","ac:de:48:00:11:22"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.14.5","major":10,"minor":14,"patch":5,"build":"18F203"},"timezone":"CEST","timezone_offset_sec":7200,"id":"819AEC04-3C58-5647-A308-DD04BB70BB76"}}}
2019-06-27T08:22:51.321+0200    INFO    [beat]  instance/beat.go:947    Process info    {"system_info": {"process": {"cwd": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat", "exe": "./metricbeat", "name": "metricbeat", "pid": 52712, "ppid": 5142, "start_time": "2019-06-27T08:22:51.275+0200"}}}
2019-06-27T08:22:51.321+0200    INFO    instance/beat.go:292    Setup Beat: metricbeat; Version: 8.0.0
2019-06-27T08:22:51.321+0200    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'metricbeat-8.0.0' as ILM is enabled.
2019-06-27T08:22:51.321+0200    INFO    elasticsearch/client.go:170     Elasticsearch url: http://localhost:9200
2019-06-27T08:22:51.322+0200    INFO    [publisher]     pipeline/module.go:97   Beat name: Mikes-MacBook-Pro.local
2019-06-27T08:22:51.322+0200    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-06-27T08:22:51.322+0200    INFO    instance/beat.go:421    metricbeat start running.
2019-06-27T08:22:51.324+0200    INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":21,"time":{"ms":21}},"total":{"ticks":64,"time":{"ms":64},"value":0},"user":{"ticks":43,"time":{"ms":43}}},"info":{"ephemeral_id":"8363fb4f-298a-4217-acc4-ef6ac6b2e720","uptime":{"ms":28}},"memstats":{"gc_next":5815696,"memory_alloc":3786760,"memory_total":11329856,"rss":30232576},"runtime":{"goroutines":44}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":12},"load":{"1":1.8247,"15":2.1895,"5":1.9268,"norm":{"1":0.1521,"15":0.1825,"5":0.1606}}}}}}
2019-06-27T08:22:51.324+0200    INFO    [monitoring]    log/log.go:154  Uptime: 29.172855ms
2019-06-27T08:22:51.324+0200    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2019-06-27T08:22:51.324+0200    INFO    instance/beat.go:431    metricbeat stopped.
2019-06-27T08:22:51.324+0200    ERROR   instance/beat.go:877    Exiting: 2 errors: error making http request: Get /: unsupported protocol scheme ""; error making http request: Get /: unsupported protocol scheme ""
Exiting: 2 errors: error making http request: Get /: unsupported protocol scheme ""; error making http request: Get /: unsupported protocol scheme ""```

cachedout · 2019-06-27T07:50:52Z

Update: @ruflin pointed out that the config file is just checked into master. I'm grabbing it and testing again.

cachedout · 2019-06-27T07:54:05Z

Using the supplied file in master which is shown below, I get the same protocol error as above:

# Module: logstash
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-logstash.html

- module: logstash
  metricsets:
    - node
    - node_stats
  period: 10s
  hosts: ["localhost:9600"]
  #username: "user"
  #password: "secret"
  xpack.enabled: true

09:52 $ ./metricbeat -e
2019-06-27T09:53:02.438+0200    INFO    instance/beat.go:606    Home path: [/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat] Config path: [/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat] Data path: [/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/data] Logs path: [/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/logs]
2019-06-27T09:53:02.438+0200    INFO    instance/beat.go:614    Beat ID: 86770118-2dba-44ec-95cd-cbd63561ce78
2019-06-27T09:53:02.440+0200    INFO    [beat]  instance/beat.go:902    Beat info       {"system_info": {"beat": {"path": {"config": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat", "data": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/data", "home": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat", "logs": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/logs"}, "type": "metricbeat", "uuid": "86770118-2dba-44ec-95cd-cbd63561ce78"}}}
2019-06-27T09:53:02.440+0200    INFO    [beat]  instance/beat.go:911    Build info      {"system_info": {"build": {"commit": "d8d98a3ba68bb4046e0896ba7de2d9b78ebccff7", "libbeat": "8.0.0", "time": "2019-06-27T07:51:40.000Z", "version": "8.0.0"}}}
2019-06-27T09:53:02.440+0200    INFO    [beat]  instance/beat.go:914    Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":12,"version":"go1.11.5"}}}
2019-06-27T09:53:02.441+0200    INFO    [beat]  instance/beat.go:918    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-06-19T18:27:43.850488+02:00","name":"Mikes-MacBook-Pro.local","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::14b0:2163:ea86:49ca/64","192.168.0.14/24","fe80::788d:99ff:fe61:53bb/64","fe80::4662:a553:1efe:ec38/64","10.0.0.1/24","fe80::a1e:38c9:922a:d512/64","fe80::aede:48ff:fe00:1122/64"],"kernel_version":"18.6.0","mac":["3a:f9:d3:1f:27:0d","38:f9:d3:1f:27:0d","0a:f9:d3:1f:27:0d","7a:8d:99:61:53:bb","5e:00:80:f0:b6:01","5e:00:80:f0:b6:00","5e:00:80:f0:b6:05","5e:00:80:f0:b6:04","5e:00:80:f0:b6:01","0a:00:27:00:00:00","0a:00:27:00:00:01","ac:de:48:00:11:22"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.14.5","major":10,"minor":14,"patch":5,"build":"18F203"},"timezone":"CEST","timezone_offset_sec":7200,"id":"819AEC04-3C58-5647-A308-DD04BB70BB76"}}}
2019-06-27T09:53:02.441+0200    INFO    [beat]  instance/beat.go:947    Process info    {"system_info": {"process": {"cwd": "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat", "exe": "./metricbeat", "name": "metricbeat", "pid": 39751, "ppid": 24791, "start_time": "2019-06-27T09:53:02.370+0200"}}}
2019-06-27T09:53:02.441+0200    INFO    instance/beat.go:292    Setup Beat: metricbeat; Version: 8.0.0
2019-06-27T09:53:02.441+0200    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'metricbeat-8.0.0' as ILM is enabled.
2019-06-27T09:53:02.441+0200    INFO    elasticsearch/client.go:170     Elasticsearch url: http://localhost:9200
2019-06-27T09:53:02.442+0200    INFO    [publisher]     pipeline/module.go:97   Beat name: Mikes-MacBook-Pro.local
2019-06-27T09:53:02.442+0200    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-06-27T09:53:02.442+0200    INFO    instance/beat.go:421    metricbeat start running.
2019-06-27T09:53:02.446+0200    INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":41,"time":{"ms":41}},"total":{"ticks":114,"time":{"ms":115},"value":114},"user":{"ticks":73,"time":{"ms":74}}},"info":{"ephemeral_id":"b0d9edd6-5f25-4818-b925-0b957595392e","uptime":{"ms":44}},"memstats":{"gc_next":5776864,"memory_alloc":2928336,"memory_total":10526544,"rss":29896704},"runtime":{"goroutines":24}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":12},"load":{"1":3.4819,"15":2.2275,"5":2.4541,"norm":{"1":0.2902,"15":0.1856,"5":0.2045}}}}}}
2019-06-27T09:53:02.447+0200    INFO    [monitoring]    log/log.go:154  Uptime: 46.281538ms
2019-06-27T09:53:02.447+0200    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2019-06-27T09:53:02.447+0200    INFO    instance/beat.go:431    metricbeat stopped.
2019-06-27T09:53:02.447+0200    ERROR   instance/beat.go:877    Exiting: 2 errors: error making http request: Get /: unsupported protocol scheme ""; error making http request: Get /: unsupported protocol scheme ""
Exiting: 2 errors: error making http request: Get /: unsupported protocol scheme ""; error making http request: Get /: unsupported protocol scheme ""

cachedout · 2019-06-27T08:02:03Z

Disabling the logstash-xpack configuration and running with only the system module configuration enabled does not produce errors.

cachedout · 2019-06-27T08:04:09Z

Commenting out xpack.enabled: true in logstash-xpack.yml removes the error and the Beat does not exit with an error. As soon as the flag is re-enabled, the error returns.

cachedout · 2019-06-27T08:05:29Z

Additionally, I can query the Logstash endpoint (running on 7.2 per test instructions) by hand without problems:

10:04 $ curl http://localhost:9600/
{"host":"Mikes-MacBook-Pro.local","version":"7.2.0","http_address":"127.0.0.1:9600","id":"2dd34e9d-b048-4540-946e-3b5f78a9dc6b","name":"Mikes-MacBook-Pro.local","ephemeral_id":"4206c4c5-216a-4cc1-9ceb-1266fc1b390a","status":"green","snapshot":null}

cachedout · 2019-06-27T08:08:15Z

Since you don't mention what the intended error should be, I can't conclude whether this is the expected behavior or not. Once you can confirm, I can come back and re-review. Thanks!

…is set

ycombinator · 2019-06-27T09:50:24Z

@cachedout I can confirm the following error locally:

Exiting: 2 errors: error making http request: Get /: unsupported protocol scheme ""; error making http request: Get /: unsupported protocol scheme ""

Good news is that it's only happening in this PR, not in master 😅. I'm investigating what's wrong in this PR and will fix shortly.

ycombinator · 2019-06-27T09:54:52Z

@cachedout The issue is fixed in 1409c55. Please re-review this PR when you get a chance. Thanks!

cachedout

I have tested this and I receive the error as expected. Good to go!

ycombinator · 2019-06-27T11:40:16Z

jenkins, test this

ycombinator added module review Metricbeat Metricbeat needs_backport PR is waiting to be backported to other branches. Feature:Stack Monitoring v8.0.0 v7.3.0 labels Jun 27, 2019

ycombinator requested review from cachedout and ruflin June 27, 2019 03:11

ycombinator requested a review from a team as a code owner June 27, 2019 03:11

ycombinator mentioned this pull request Jun 27, 2019

System test for logstash module, xpack code path #12676

Merged

Adding version check to Logstash metricsets when xpack.enabled: true …

6a9324a

…is set

ycombinator force-pushed the mb-ls-xp-version-check branch from d8d98a3 to 6a9324a Compare June 27, 2019 09:22

Fixiing URI in call to logstash.GetVersion

1409c55

cachedout approved these changes Jun 27, 2019

View reviewed changes

ycombinator merged commit 2f69549 into elastic:master Jun 27, 2019

ycombinator deleted the mb-ls-xp-version-check branch June 27, 2019 12:48

ycombinator removed the needs_backport PR is waiting to be backported to other branches. label Jan 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding version check to Logstash metricsets when xpack.enabled: true is set #12705

Adding version check to Logstash metricsets when xpack.enabled: true is set #12705

ycombinator commented Jun 27, 2019 •

edited

Loading

elasticmachine commented Jun 27, 2019

cachedout commented Jun 27, 2019

cachedout commented Jun 27, 2019

cachedout commented Jun 27, 2019 •

edited

Loading

cachedout commented Jun 27, 2019

cachedout commented Jun 27, 2019 •

edited

Loading

cachedout commented Jun 27, 2019

cachedout commented Jun 27, 2019

ycombinator commented Jun 27, 2019

ycombinator commented Jun 27, 2019

cachedout left a comment

ycombinator commented Jun 27, 2019

Adding version check to Logstash metricsets when xpack.enabled: true is set #12705

Adding version check to Logstash metricsets when xpack.enabled: true is set #12705

Conversation

ycombinator commented Jun 27, 2019 • edited Loading

Testing this PR

elasticmachine commented Jun 27, 2019

cachedout commented Jun 27, 2019

cachedout commented Jun 27, 2019

cachedout commented Jun 27, 2019 • edited Loading

cachedout commented Jun 27, 2019

cachedout commented Jun 27, 2019 • edited Loading

cachedout commented Jun 27, 2019

cachedout commented Jun 27, 2019

ycombinator commented Jun 27, 2019

ycombinator commented Jun 27, 2019

cachedout left a comment

Choose a reason for hiding this comment

ycombinator commented Jun 27, 2019

ycombinator commented Jun 27, 2019 •

edited

Loading

cachedout commented Jun 27, 2019 •

edited

Loading

cachedout commented Jun 27, 2019 •

edited

Loading