[Telemetry][Usage Collection] Finish migration to the new elasticsearch client

[TinaHeiligers]

Support for the legacy Elasticsearch client will be removed from 7.15. We'll remove it by mid-7.13.

Full migration to the new client involves several steps.

In #75384, we added the new client to collectors, allowing plugin owners to migrate over time.

This issue is to finish the migration and remove support for the legacy client in all telemetry and usage collection related plugins.

Items:

• Add migration Guide ([Elasticsearch Migration] Update docs re UsageCollection #84322)
• List all the collectors to migrate
• Follow up with the teams and help them ([Meta] Migrate telemetry usage_collectors to the new ES Client #86358)

[elasticmachine]

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

[afharo]

Maybe we should send an email to Kibana Contributors so every collector owner updates their implementation to the new esClient or soClient clients provided in the fetch context?

[TinaHeiligers]

@afharo that's a very good idea! We can't fully migrate to the new client until collector owners switch over.
cc @joshdover

[joshdover]

Let's discuss this in our next weekly. We need to start coordinating this migration for all ES client usages, and it's worth asking teams to do this specific usage specifically.

[afharo]

I added some documentation for these changes in the Migration Examples guide: #84322

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[TinaHeiligers]

Blocked by #86358

[TinaHeiligers]

@joshdover
There is tight coupling between the usage_collection plugin, the telemetry plugin and the telemetry_collection_manager and we cannot migrate one without migrating the others.
More specifically, removing the legacy ES Client from usage_collection also involves removing the legacy client from the bulkFetch method.
bulkFetch is passed down through registerCollection in the telemetry plugin to stats getters (e.g. getLocalStats) that then passes callCluster on to getKibana, where that is used in the fetch context for kibana collectors.
The StatsCollectionConfig type is declared in the telemetry_collection_manager and also needs to be updated (i.e. the legacy client needs to be removed).

In other words, the migration effort will require migrating all three of these plugins and will be a larger effort than we originally thought it would be.

I've started the work anyway and am migrating collectors as needed to unblock tests (e.g. api integration tests).

If I do end up migrating all the collectors, I'll do those as separate PR's to this work.

cc @afharo @Bamieh

[joshdover]

In other words, the migration effort will require migrating all three of these plugins and will be a larger effort than we originally thought it would be.

Got it, thanks for the update. Any reason to think we should delay this? I suspect not since we need to do it anyways and I don't believe any other 7.12 work you have is needed more urgently.