Tempo returns 500s for some 400s

[joe-elliott]

Describe the bug
The Tempo query-frontend sharding code will hide certain errors (like 429) and instead incorrectly return 500.

[annanay25]

After some investigation, my conclusions are -

• the query frontend has always been reporting a 429
• the problem is that we're not checking resp.StatusCode in the plugin in tempo-query here -

  tempo/cmd/tempo-query/tempo/plugin.go

  Line 63 in 1cbfc21

  resp, err := http.DefaultClient.Do(req)

• on hitting resource limits, resp.StatusCode there is seen to be 429 and resp.Body is rpc error: code = Code(429) desc = too many outstanding requests but we're trying to parse it like a otlp trace batch.. and so its actually tempo-query that's throwing the 500

Next steps -

• Add check on response status code in tempo-query
• We expect Remove dependency on Jaeger-Query #382 to be fixed soon, and so error handling will switch over to Grafana

[annanay25]

This is fixed by #574