fix: properly return http status codes from ingester to querier for RPC function calls

[cstyan]

We noticed this earlier with incorrect query syntax on a recording rule that was then sent to an ingester. The error returned to the end user had an HTTP 500 status code without much information because the grpc server was not seeing a valid status code in the result from the ingester, and was wrapping the error as a generic 500 status code.

@slim-bean and I discussed on slack and we think all functions the Querier can call on the Ingester via RPC should use the same error wrapping as we have in our server utils code. Looking at the generated GRPC code, this should be:

• Query
• QuerySample
• Label
• Series
• Tail
• TailersCount
• GetChunkIDs
• GetStats
• GetVolume
• GetDetectedLabels
• GetDetectedFields (this one can never return an error afaict)

Note that I have just done the easy thing for now, and I'll simplify the duplicated code in the morning

[cyriltovena]

While this is a good idea to have. I think we should investigate how such queries end up on ingester, we should fail fast if this is not good syntax a querier or query-frontend should have caught it.

This was brought up during incident review meeting.

[cstyan]

we should fail fast if this is not good syntax a querier or query-frontend should have caught it.

Agreed, and I will look into this more. Do you have any objections to merging this as is while we continue to investigate why these queries are making it to the ingester in the first place?

[cyriltovena]

LGTM