[Lens] Provide better guidance for dimension cardinality (especially for top values)

[flash1293]

Splitting this out from #93912 because I think it's a longer term project.

We discussed this topic offline and had the following ideas:

• In some cases, it makes sense to go beyond 1000, in some cases it doesn't make sense (like break down by in xy chart)
  • A smart system should take the context of the dimension into account
• The real problem is when there are nested top values (because in the worst case the sizes are multiplied)
  • A smart system should take the full configuration into account and limit this full configuration in a meaningful way

However, there are a lot of open questions how this would work exactly:

• The nesting order of x axis and break down by is configurable, even if the user is setting the size to "just" 1000, they can still end up with many more series - is this OK? If not, what can we do about it?
• In what way do we try to guide the user if they configure too high cardinality configurations?
• How do we communicate the sum/product of the nested aggregation cardinality matters? Does the user have to balance the configuration of all dimensions, going back and forth to be allowed to increase cardinality? How do we keep this a nice experience?
• How does this fit into the architecture? Right now the visualization is not communicating these kinds of things to the datasource at all
• What are sensible limits for the different situations?

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[yuvielastic]

+1 Need the functionality to show more than 1000 top values in Lens.

We were determining why we are seeing high capacity usage in {months} and one of the ways to determine was to identify the capacity used per deployment. As we have more than 1000 deployments on specific days, I wasn't able to get the required data from Table vis in Lens and had to use Data table vis to get the results.

[ghudgins]

related discuss post - https://discuss.elastic.co/t/limitation-on-the-value-of-top-values/307731

[drewdaemon]

Related change to 10,000 limit #136399

[archon810]

The top limit is now 10,000, but what if I wanted to go higher? Why does it need to be hardcoded - can it be a setting in kibana.yml? For example, we set index.max_result_window to 65535 and would like to match it in kibana.

The use case is downloading a large list of IPs that participated in a DDoS from the logs powered by Elastic/kibana.

[markov00]

In order to provide better transparency of priorities, issues that will not be prioritized within the next 24 months are being closed.

Tracking request in Lens general improvements ice box #184459