Review apollo_router_http_requests_total metric usage

[Geal]

The behaviour of this metric is confusing right now. Looking at the telemetry plugin, it is supposed to be recorded on every response:

router/apollo-router/src/plugins/telemetry/mod.rs

Lines 853 to 860 in 05a651d

	// http_requests_total - the total number of HTTP requests received
	u64_counter!(
	"apollo_router_http_requests_total",
	"Total number of HTTP requests made.",
	1,
	metric_attrs
	);

but is set in multiple places:

router/apollo-router/src/plugins/telemetry/mod.rs

Lines 1123 to 1128 in 05a651d

	u64_counter!(
	"apollo_router_http_requests_total",
	"Total number of HTTP requests made.",
	1,
	metric_attrs
	);

but it is also directly set in other places like

router/apollo-router/src/services/layers/query_analysis.rs

Lines 75 to 80 in 05a651d

	"apollo_router_http_requests_total",
	"Total number of HTTP requests made.",
	1,
	status = StatusCode::BAD_REQUEST.as_u16() as i64,
	error = "Must provide query string"
	);

I looked into that because the content-negotiation layer does not set this metric when returning a HTTP 415:

router/apollo-router/src/services/layers/content_negotiation.rs

Lines 55 to 73 in 05a651d

	let response: http::Response<hyper::Body> = http::Response::builder()
	.status(StatusCode::UNSUPPORTED_MEDIA_TYPE)
	.header(CONTENT_TYPE, APPLICATION_JSON.essence_str())
	.body(hyper::Body::from(
	serde_json::json!({
	"errors": [
	graphql::Error::builder()
	.message(format!(
	r#"'content-type' header must be one of: {:?} or {:?}"#,
	APPLICATION_JSON.essence_str(),
	GRAPHQL_JSON_RESPONSE_HEADER_VALUE,
	))
	.extension_code("INVALID_CONTENT_TYPE_HEADER")
	.build()
	]
	})
	.to_string(),
	))
	.expect("cannot fail");

The apollo_router_operations_total metric, though, correctly record a 415 status

[BrynCooke]

I think that this metric can largely be replaced once the work for custom instruments that @bnjjj is working on lands.

[abernix]

Will this be replaced by http.server.request.duration?

[bnjjj]

Yes this example of configuration should do the same job

telemetry:
  instrumentation:
    instruments:
      router:
        http.server.request.duration:
          attributes:
            error:
              error: reason
            operation_name:
              response_context: operation_name
            status:
              response_status: code

[bnjjj]

This PR will help to add operation_name at the router service without playing with the response_context

[BrynCooke]

Closing and we will look to review all metrics for the next major release.