[APM] Many environments are not displayed in dropdown filter

[ogupte]

When exploring an instance with a lot of environments, not all environments will display in the environment filter. This might be because we query for environments using a terms agg with a size specified (https://github.com/elastic/kibana/blob/master/x-pack/plugins/apm/server/lib/ui_filters/get_environments.ts#L56-L59). The solution might be more involved than specifying a large enough size because a large size would greatly impact performance. Another solution would be to use a composite aggregration to get the distinct environment values. This would introduce pagination to this query which might need changes to the UI selector to trigger loading new page of environments.

Related: #77573

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[henrikno]

Just noticed it stops at 10 (including Not defined), the default size for aggregations, so this is likely the cause.

[alex-fedotyev]

It looks like fixing the query would be important here.

@ogupte - would it make sense to redesign the environment selector and make it a dropdown with a typeahead for better usability?

[sorenlouv]

I don't think we've seen customers with 100s of environments (they are often less than 10), so I'm not sure if this is a scenario we should invest in improving.
If we add support for spaces would that alleviate the problem?

For the time being, instead of registering a separate environment per dev, could they set a separate service name? I know it seems like a hack but would probably work better.

[nehaduggal]

@sqren don't think adding support for spaces would exactly solve the issue and creating service names with environments is pretty hacky(also won't work for things like maps where you want to view maps of specific environments only). Lets make the query configurable for now to unblock folks and consider re-designing the filter as per @alex-fedotyev suggestion?