[APM] Service overview: dependencies table

[dgieselaar]

Summary

Related design issue elastic/apm#300

The service overview introduces a new view for the selected service that encompasses a lot of the service data from existing views like metrics, transactions, and errors. The primary goal of the view is to aid in troubleshooting a service from a birds eye view rather than immediately diving into the Transactions, as that's the current landing page for a selected service. There's more information about the user stories and goals of the view in the design issue linked above.

Dependencies table

One of the new components of the view is the dependencies tables that list the downstream services or external connections that are relevant for the service at the selected time range. The information stems from the destination data captured for the service map. In this component, we're not only listing the connections but also showing the metrics per connection such as latency, traffic, and error rate.

Per #81120 (comment), we are not going to include the health status.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[dgieselaar]

@nehaduggal @sqren @formgeist I am running into an issue on dev-next, where opbeans is configured differently than usual. The problem is that services talk to each other via a load balancer (at opbeans:3000), and the load balancer decides which service a request is distributed to (via a round-robin algorithm).

The red line is how a request travels across services. The black lines is where span metrics are recorded. Span metrics contain the service name of the calling service, plus the address it is talking to reach another service (instrumented or not). Because the outgoing address is always opbeans:3000, the span metrics are for all requests to the load balancer, regardless of where it gets distributed.

The service map currently does show all connections, because we walk traces, and because we walk a set of traces for each connections to get a picture that is as complete as possible.

For the service dependencies table, while we could map one outgoing address to multiple services, we cannot break the metrics down per service. I don't think the current design takes a situation like this into account.

Some possible options:

• Show services grouped together. In this case, we would show all metrics for opbeans:3000, and then some kind of expand/collapse mechanism. Not sure if this is helpful, and it makes the query harder (because we have to apply a similar mechanism as the service map).
• Display the outgoing address if we find out the address maps to multiple services.
• Consider this an instrumentation error, and always show the first service that we can map to the outgoing request. This would be my preference. We might solve the resolution/query problem by adding paths, but I don't think we can for instance accurately show error metrics if one address results in multiple services (maybe we can). If a request to opbeans:3000 fails because of a network issue, we have no way of knowing what service this error should be attributed to.

@axw and @eyalkoren were discussing a similar scenario in elastic/apm#364.

[nehaduggal]

As per our discussion, we agreed that for 7.11 we will surface the first detected service in the dependency table as listed in #3 in Darios comment. This will mean that if customers have an opbeans like setup(multiple different services routed through the load balancer) they will possibly see a different dependencies table on refresh or when selected time window is changed. While inconsistency isn't great this will probably affect a small group of users. We will work on resolving this inconsistency in a future release. @alex-fedotyev

[alex-fedotyev]

@dgieselaar - I think this is a very legit issue, and we would likely see it in the real life often enough, thanks for bringing it up!
The most common example I saw with customers is using API Gateway between the services.
With API Gateway outbound HTTP(s) calls would appear having same host and port for all the calls, but then they could be routed to the different services.

Should we create a follow up issue to improve dependencies experience based on the outcomes of elastic/apm#364?