[Monitoring] Check for timeouts to ES requests

[pickypg]

Now that the Monitoring UI defaults to using Cross Cluster Search (CCS), there is the potential for requests to ES to timeout when a remote cluster goes down -- but it is not marked as skip_unavailable or it has not been recognized as unavailable yet -- and this can cause timeouts that create a bad experience.

In one such case, CCS was in use and all CCS searches (e.g., GET /*:.monitoring-6-*/_search) were timing out. This resulted in the cluster listing page being redirected to #no-data and then presenting a completely blank screen and a toast notification indicating a timeout (pictured below).

Technically, we are telling the user what happened, but the experience is poor to say the least and there's no hint in the timeout to check CCS. Worse, having the blank screen leaves the UI in a completely unusable state that leaves the user thinking that it is reasonably broken rather than it has nothing to show.

For searchability, this is the error:

Monitoring Request Error

Request Timeout after 30000ms: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes.

HTTP 503

[elasticmachine]

Pinging @elastic/kibana-monitoring

[johntdyer]

Any thoughts on this ? \

[cachedout]

@igoristic Could you please investigate this and propose a solution?

[cachedout]

@igoristic Polite bump. Did you ever get a chance to look at this?