[Telemetry] Have Telemetry automatically get all the Kibana usage stats

[tsullivan]

Closes #21239

This change gets the telemetry data payload to automatically include any usage stat data that has been collected in the .monitoring-kibana-* indices.

Note that Kibana telemetry still only works when Kibana Monitoring is turned on.

[tsullivan]

@pickypg this PR has the stats generated by Kibana's usage collectors generically added into a kibana.plugins object. For example:

  "kibana": {
    "count": 1,
    "dashboard": { "total": 1 },
    "visualization": { "total": 1 },
    "search": { "total": 0 },
    "index_pattern": { "total": 3 },
    ...,
    "plugins": {
     "canvas": {
      "workpads": { "total": 2 },
      ...
     },
     "xpack": {
      "reporting": {
       "available": true,
       "enabled": true,
       "browser_type": "phantom",
       ...
      }
     }
    }
   }

Previously we had a module to collect reporting stats, and there is an overall combiner that fetches reporting stats and plugs it into the payload in an area we decided on. This setup isn't scalable as we need to add another module for each new integration.

The obvious issue is this breaks the expected data that the telemetry web service receives. Can something be changed there so we can go with these changes?

[pickypg]

The obvious issue is this breaks the expected data that the telemetry web service receives. Can something be changed there so we can go with these changes?

We can adjust the other side to handle the breaking changes if we must. Let's not get too harsh of an issue, but we should avoid continuing down this path for future telemetry and move to the new model in the future (for others reading this).

[tsullivan]

retest

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[tsullivan]

@pickypg able to take a look for review?

[tsullivan]

cc @elastic/kibana-platform anyone have some time to review this change to simplify the payload-generation side of telemetry?

[tsullivan]

We can adjust the other side to handle the breaking changes if we must. Let's not get too harsh of an issue, but we should avoid continuing down this path for future telemetry and move to the new model in the future (for others reading this).

Maybe we can keep this conversation going. I get that it is a pain to add more mappings in the telemetry cluster for new features added into the data. But that's only half the pain, this PR addresses the caller's half of the pain of wiring new features into the data.

I think I understand the new model better than I initially did. Am I right in thinking that the data could be fit into that model in a post-query phase? I'm pretty ok with how the usage collectors represent the data they fetch, but perhaps that data could be mapped right before getting transmitted to have an expressive structure where mapping on the other side would be automatic.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[tsullivan]

We can adjust the other side to handle the breaking changes if we must. Let's not get too harsh of an issue, but we should avoid continuing down this path for future telemetry and move to the new model in the future (for others reading this).

I'm coming back to this as other work targeted for 6.5 depends on this. The update on this comment is that this PR is planned to be the last breaking thing we'll do for the telemetry upload data.

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[tsullivan]

re: #22336 (comment)

    "xpack": {
      "reporting": {

Correction/update: there will be no plugins nested under xpack. That was a side-effect of having the internal monitoring collection keep the structure that was in place for reporting. Since this breaks the data model already (moving reporting into kibana.plugins from making the stats merged together in a generic way, let's make it really generic and not differentiate xpack.

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[pickypg]

LGTM with latest tweak