Report kibana_settings to X-Pack Monitoring (#7664)

[ycombinator]

Resolves #7621.

Depends on elastic/kibana#21100.

X-Pack Monitoring of Kibana requires two types of documents in the .monitoring-kibana-* indices: kibana_stats and kibana_settings. We made Metricbeat's kibana/stats metricset index kibana_stats documents into .monitoring-kibana-* in #7525. This PR makes the same metricset index kibana_settings documents into .monitoring-kibana-*.

To test this PR

The idea is that metricbeat (specifically the kibana/stats metricset with xpack.enabled: true) will create exactly the same documents in .monitoring-kibana-* indices as Kibana's bulk uploader does today.

1. Start up Elasticsearch (with the default Basic license is okay)

2. Start up Kibana (with the default Basic license is okay), checked out from master.

3. Enable Monitoring in Elasticsearch (via the cluster setting xpack.monitoring.collection.enabled: true). You can do this via Elasticsearch's Cluster Update Settings API or by clicking the "Turn on Monitoring" button in the Monitoring UI in Kibana.

4. Let Kibana run for ~20 seconds so a few documents are indexed into .monitoring-kibana-*.

5. Stop Kibana

6. From .monitoring-kibana-*, retrieve a document for type = kibana_settings

7. Delete the .monitoring-kibana-* indices.

8. In kibana.yml, set xpack.monitoring.kibana.collection.enabled: false. This will ensure that Kibana is no longer directly shipping its monitoring metrics to Elasticsearch.

9. Start up Kibana again, this time checked out from [Monitoring] Kibana settings api kibana#21100.

10. Enable the kibana module in metricbeat: ./metricbeat modules enable kibana.

11. In modules.d/kibana.yml, add the stats metricset and set xpack.enabled: true. Concretely, your modules.d/kibana.yml should look something like this:

    - module: kibana
       metricsets:
       - stats
       period: 10s
       hosts: ["localhost:5601"]
       #basepath: ""
       #username: "user"
       #password: "secret"
       xpack.enabled: true

    Additionally, if your Kibana URLs have a basepath prefix, say foo, make sure to un-comment the basepath setting in the kibana module YAML file and set it's value to "foo".

12. Start metricbeat.

13. Let metricbeat run for ~20 seconds so a few documents are indexed into .monitoring-kibana-*.

14. Stop metricbeat

15. From .monitoring-kibana-*, retrieve a document for type = kibana_settings.

16. Using a tool such as http://www.jsondiff.com, compare the documents indexed by Kibana's with those indexed by metricbeat. Verify that their structures are identical (same fields, not necessarily same values), except for these known and expected differences:

    1. Only Metricbeat-indexed documents are expected to contain the fields @timestamp, beat, host, and metricset. These are "standard" fields added by beats and metricbeat and don't have an adverse impact since they are additive.
    2. Only Kibana-indexed documents are expected to contain the field source_node. This field is used for debugging purposes only and not actually consumed by either the Monitoring UI or Telemetry feature in Kibana.

(cherry picked from commit 2af5ab9)