[Monitoring] telemetry fetchers use broken pagination logic

[afharo]

As @robbavey pointed out, this is following the same approach as the get_beats_stats piece of logic.

@chrisronline, do you know if .monitoring-* indices have a custom index.max_result_window setting that allows pagination for more than 10k docs? Or if it's even possible that we reach 10k docs on these types of queries?

Originally posted by @afharo in #90850 (comment)

Let's revisit the pagination logics for Beats and Logstash to make sure they work for 10k+ docs. We can use either the Scroll or the Search After approaches.

[elasticmachine]

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

[afharo]

Ping @elastic/stack-monitoring-ui

[chrisronline]

We don't have anything like that, AFAIK. It seems like it'd come up more but I honestly can't recall the last time I encountered someone with an issue around this

[afharo]

This is about collecting telemetry, so any errors will silently fail and simply not send telemetry. I don't think anyone would actually notice.

I also think it's an edge case. That would mean we are dealing with massive clusters with more than 10k Beasts instances, Logstash instances or Logstash's pipelines ephemeral IDs.

[chrisronline]

Makes sense. I meant we don't hear issues from customers that they aren't able to see their 10k+ beats in the SM UI, so I'm not sure how often this scenario comes up for anyone (where they exceed 10k instances of anything) so it seems like an isolated use case but maybe something worth noting in docs somewhere.

[simianhacker]

I'm not sure what the issue is here. Are we trying to figure out if the UI can handle 10K+ logstash instances?

[Bamieh]

@simianhacker telemetry is running queries against the .monitoring-* indices to report them back to our cluster.

The current logic to paginate over the indices does not work because the max_window (from + size) can't exceed 10k docs by default in ES. This means with the current implementation no pagination will happen even if there are over 10k documents reported by beats/logstash.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[jasonrhodes]

@Bamieh @afharo @robbavey we just re-discovered this ticket and we aren't sure if this is a "bug" or not? We are trying to determine if we need to pull this in...

[afharo]

@jasonrhodes in #127273 I removed the "broken" pagination logic: it basically doesn't paginate anymore.

However, ideally, the requests in there should use PIT pagination to retrieve all the available data in a less expensive way. For more context of why we prefer PIT requests vs. size: 1000 searches, please refer to #93770

[smith]

Closing. If it's something we want to do we can start with:

However, ideally, the requests in there should use PIT pagination to retrieve all the available data in a less expensive way. For more context of why we prefer PIT requests vs. size: 1000 searches, please refer to #93770