API errors are incorrectly reported as peer errors

[BRMatt]

It seems refinery checks the API host option in outgoing events to see if the message is destined for a peer, or the honeycomb API:

refinery/transmit/transmit.go

Lines 164 to 171 in 003f55f

	if honeycombAPI == apiHost {
	// if the API host matches the configured honeycomb API,
	// count it as an API error
	d.Metrics.Increment(d.Name + counterResponseErrorsAPI)
	} else {
	// otherwise, it's probably a peer error
	d.Metrics.Increment(d.Name + counterResponseErrorsPeer)
	}

During a conversation in the pollinators slack we noticed that log lines emitted by refinery for some messages showed that the event's api_host option was empty, even though the event was being sent to honeycomb:

Apr 01 12:13:01 production-refinery-i-00c73eea958d0b46c refinery[11758]: time="2021-04-01T12:13:01Z" level=error msg="non-20x response when sending event" api_host= dataset= error="Post \"https://api.honeycomb.io/1/batch/production.traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" event_type= status_code=0 target=

This caused timeout errors in the upstream honeycomb API to be treated as peer errors:

This is a bit misleading as it causes the operator to believe there's a problem with their cluster, when the issue is really coming from the honeycomb API.

Given metrics are already prefixed with upstream_ or peer_, I'm wondering whether we really need the _api and _peer suffixes, as it's a bit confusing. e.g. How should upstream_response_errors_peer differ from peer_response_errors_peer?

Would it be possible to remove this suffix so that operators can determine whether a metric is related to the honeycomb API, or the peer, by its prefix?

[vreynolds]

Thanks for letting us know, Matt! 👋 We'll take a look as soon as we can.