[APM] Make UI indices space aware (support for spaces)

[sorenlouv]

It should be possible to limit which services are displayed in a given space. This will help users organize services into separate spaces.

This could be accomplished by letting users select which services are part of a given space. If no services are selected all services will be displayed.

Related
https://discuss.elastic.co/t/define-apm-permission-for-services/276663/2

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

The ability to customize apm indices will be removed starting in 8.0.

[vxavictor513]

The ability to customize apm indices will be removed starting in 8.0.

@sqren: Is there any particular reason?

[simitt]

@sqren there will still be the option to set a custom namespace, leading to dedicated data streams.
The datastreams are built "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}" where data_stream.namespace will be configurable. Not certain if that impacts the decision to close this issue.

[sorenlouv]

Good point @simitt . I think I'll re-open this issue but rephrase it to be about supporting spaces without mentioning index patterns.

[dgieselaar]

Allowing the user to set the index target per space, could give them the option of using filtered aliases to restrict access to specific services or an environment in a space. Data streams don't support aliases yet but it is a work-in-progress, and support for filters could be added if needed. Example:

POST /_aliases
{
  "actions": [
    {
      "add": {
        "indices": [
          "apm-*"
        ],
        "alias": "production-services-apm",
        "filter": {
          "term": { "service.environment": "production" }
        }
      }
    }
  ]
}

or:

POST /_aliases
{
  "actions": [
    {
      "add": {
        "indices": [
          "logs-apm*",
          "traces-apm*",
          "metrics-apm*"
        ],
        "alias": "checkout-service-apm",
        "filter": {
          "term": { "data_stream.dataset": "checkout-service" }
        }
      }
    }
  ]
}

[DanOptimus]

I work with a rather large deployment of Elastic APM at a Fortune 100 company, presently running 7.15.1, where we have 130+ applications sending data to our elastic APM cluster. We uniquely send each app teams data to its own index using elastic, using logstash. At present page load times take 30 to 60 seconds on the APM dashboards. When I set this kibana space setting to filter by index (or index pattern), the UI goes to nearly sub-second response time. The problem is that when I make this setting it wipes out all other kibana spaces, we prefer to roll out each application to their own kibana space, with their own index filter or index pattern to alleviate this problem, but we really need this fixed to vastly improve our overall performance. Lastly, I just wanted to indicate, I really prefer this setting be externalized via an API, so that we can set this using automated kibana spaces creation. Thanks Dan

[romanlum]

Any progress here?
We also need this feature for managing microservices in different spaces using their own index.
#77882 would be a good start to merge :)

[sorenlouv]

Yes, we are actively looking into this. I can't give any timeline but safe to say that we haven't forgotten about it :)

[christianhuening]

We would also be very interested in this as a customer :)