[6.8] Difficulty configuring Elasticsearch module in Metricbeat

[cachedout]

I observed the following behavior on 6.8.6 and ES with a basic license:

20:08 $ ./metricbeat version
metricbeat version 6.8.6 (amd64), libbeat 6.8.6 [ed8710b437606e8e2915efbb1bef0764fc4c350c built 2019-12-05 19:00:42 +0000 UTC]

1. Enable the Elasticsearch module:

./metricbeat modules enable elasticsearch

2. Set xpack.enabled: true

3. Start Metricbeat and observe an error:

2019-12-05T20:04:24.296+0100    ERROR   instance/beat.go:906    Exiting: 1 error: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr cluster_stats index index_recovery index_summary
ml_job node_stats shard]

4. Modify the elasticsearch.yml file to include the requested metricsets:

20:06 $ cat modules.d/elasticsearch.yml
# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/6.8/metricbeat-module-elasticsearch.html

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["http://localhost:9200"]
  username: "elastic"
  password: "changeme"
  xpack.enabled: true

5. Restart Elasticsearch:

2019-12-05T20:04:44.383+0100    ERROR   [elasticsearch] elastic/elastic.go:107  the CCR feature is available with a platinum Elasticsearch license. You currently have a basic license. Either upgrade your lice
nse or remove the ccr metricset from your Elasticsearch module configuration.
panic: value method time.Time.MarshalText called using nil *Time pointer

goroutine 290 [running]:
time.(*Time).MarshalText(0x0, 0x5784e40, 0x0, 0xbbf51e8, 0x0, 0x1)
        <autogenerated>:1 +0x93
github.com/elastic/beats/libbeat/common.normalizeValue(0x5784e40, 0x0, 0xc0006ddaa8, 0x2, 0xa, 0x537a6a0, 0xc0002ca5e0, 0x0, 0x0, 0x0)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/common/event.go:161 +0x193b
github.com/elastic/beats/libbeat/common.normalizeMap(0xc0002864b0, 0xc0006ddaa8, 0x1, 0xa, 0x0, 0x0, 0x0, 0x0)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/common/event.go:64 +0x1a4
github.com/elastic/beats/libbeat/common.normalizeValue(0x56bde80, 0xc0002864b0, 0xc0006ddaa8, 0x1, 0xa, 0x53757e0, 0xc000384490, 0x0, 0x0, 0x0)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/common/event.go:181 +0x149a
github.com/elastic/beats/libbeat/common.normalizeMap(0xc000286540, 0xc0006ddaa8, 0x0, 0xa, 0x746e69203a657079, 0x2020200a72656765, 0x2020202020202020, 0x7069726373656420)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/common/event.go:64 +0x1a4
github.com/elastic/beats/libbeat/common.ConvertToGenericEvent(0xc000286540, 0x82d19b0)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/common/event.go:48 +0x84
github.com/elastic/beats/libbeat/publisher/pipeline.glob..func2(0xc000830040, 0x38, 0x5628640, 0x1)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/processor.go:207 +0x48
github.com/elastic/beats/libbeat/publisher/pipeline.(*processorFn).Run(0xc00000eee0, 0xc000830040, 0x0, 0x0, 0xc00052e000)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/processor.go:198 +0x34
github.com/elastic/beats/libbeat/publisher/pipeline.(*program).Run(0xc000277410, 0xc000830040, 0xc000323e48, 0x0, 0x0)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/processor.go:168 +0x94
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).publish(0xc0000adb60, 0xbf7273131479a360, 0x1f8f03f, 0x6a0cb40, 0xc0001f1650, 0xc000286540, 0x0, 0x0)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:84 +0x597
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).Publish(0xc0000adb60, 0xbf7273131479a360, 0x1f8f03f, 0x6a0cb40, 0xc0001f1650, 0xc000286540, 0x0, 0x0)
        /Users/mp/devel/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:63 +0xc2
github.com/elastic/beats/metricbeat/mb/module.PublishChannels.func1(0xc0000adc20)
        /Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/mb/module/publish.go:41 +0x155
created by github.com/elastic/beats/metricbeat/mb/module.PublishChannels
        /Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/mb/module/publish.go:48 +0xd0

6. Remove the ccr metricset from the module configuration as suggested in the output of step 5:

20:06 $ cat modules.d/elasticsearch.yml
# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/6.8/metricbeat-module-elasticsearch.html

- module: elasticsearch
  metricsets:
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["http://localhost:9200"]
  username: "elastic"
  password: "changeme"
  xpack.enabled: true

7. Restart Metricbeat and observe this error:

2019-12-05T20:05:10.507+0100    ERROR   instance/beat.go:906    Exiting: 1 error: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr cluster_stats index index_recovery index_summary
ml_job node_stats shard]
Exiting: 1 error: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr cluster_stats index index_recovery index_summary ml_job node_stats shard]

Per the 6.8 docs, the xpack.enabled flag seems to have been supported back then, so I'm finding this curious. :) I have not tested this in other versions.

[zz9pzza]

I am seeing this on a 6.8.5 source cluster as well

[ycombinator]

This has been fixed by #15175.

[Ahmadhashem86]

this isssue appears in 6.8.22