[Stack Monitoring] Support endpoint counters in APM

[cachedout]

(Copied out of a document originally created by @simitt )

With #2142 (planned for 7.4) we introduced dedicated monitoring counters per endpoint. The change was made to keep the counters meaningful, e.g. requests to the APM Server's root endpoint made by some agents for figuring out server details should not be shown as part of requests to the Intake API processing actual events.

With more agents using the server's root endpoint, and especially with introducing a new endpoint for agents to fetch the remote configuration, a distinction of requests is introduced, allowing for a better monitoring experience.

The names of the counters used so far have not been changed, to not break anything, but requests that already added up to these counters while belonging to a different endpoint have been removed.

Currently used counters:

apm-server.server.*: These counters will still be present to not introduce any breaking changes, but will only contain information about requests/responses from the Intake API endpoint, responsible for processing uploaded events.

Newly introduced counters are:

apm-server.acm.*: counting requests and responses to the newly introduced agent remote configuration endpoint (beta in 7.3), that agents are using for reloading config options. These counters are interesting, but should be clearly separated from requests sending events.

apm-server.sourcemap.*: counting sourcemap uploads; Uploading sourcemaps is a manual step that probably happens once per deploy for RUM agents. Showing these counters does not seem too important for now.

apm-server.root.*: requests to the server's root endpoint, mainly expected for healthcheck requests or figuring out details about the server's version etc. Not relevant for showing in the monitoring tab at the moment.

Suggested Changes in the UI:

Change visualization: title Response Count -> Response Count Intake API

• Add apm-server.server.response.errors.count

Change visualization title Response Errors -> Response Errors Intake API.

• Additionally change used colour scheme, as most of the error counters are shown in a black-ish colour

Change visualization title Requests -> Requests Intake API

Add visualization for Response Count Agent Configuration Management
Following counters should be included:

apm-server.acm.response.count
apm-server.acm.response.errors.count
apm-server.acm.response.valid.ok
apm-server.acm.response.valid.notmodified

Add visualization for Response Errors Agent Configuration Management

Following counters should be included:

apm-server.acm.response.errors.forbidden
apm-server.acm.response.errors.unauthorized
apm-server.acm.response.errors.unavailable
apm-server.acm.response.errors.method
apm-server.acm.response.errors.invalidquery

Add visualization for Requests Agent Configuration Management

Following counters should be included:

apm-server.acm.request.count

[elasticmachine]

Pinging @elastic/stack-monitoring