[Status] /api/stats?extended=true throttling/caching

[afharo]

Describe the feature:

It would be beneficial to apply some sort of throttling or caching of the requests in the API /api/stats?extended=true.

Describe a specific use case for the feature:

We've noticed some clusters are suffering from timeouts and Kibana slowness when requesting /api/stats?extended=true[&exclude_usage=false].

Looking at Metricbeat, it forces a timeout to all HTTP requests after 10s and polls every 10s. It requests the usage (exclude_usage=false) only once every 24h but, if the request fails, it will attempt again on the next poll.

This leads to a situation where Kibana might take a bit longer to reply and Metricbeat will issue a new request after 10s, doubling the load on retrieving the usage and, potentially, taking longer to respond. So the process repeats endlessly.

On Metricbeat, we are studying a backoff mechanism to not request the usage again for a while if it fails.

But I think we should also apply some throttling/caching/memoization to avoid overloading due to something repeating the request too eagerly.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[afharo]

In this comment, there are some suggestions about the possibility of returning 429 Too Many Requests responses or even using the Retry-After header when the service is overloaded to gently ask the consumer to backoff the frequency of their requests.

[afharo]

Closing this issue because we implemented a caching mechanism to the piece of logic gathering data from the collectors.