Receiving tenantID single-tenant not found

[joe-elliott]

Reproducible locally in using tk single binary mode. Start it up and query for a trace. Something like this message will be logged

level=warn ts=2020-03-10T18:04:08.159046978Z caller=logging.go:49 traceID=32d5a7476bea519 msg="GET /api/traces/000000000000000000000000e8fb635e (500) 1.210753ms Response: \"tenantID single-tenant not found\\n\" ws: false; Accept-Encoding: gzip; User-Agent: Go-http-client/1.1; "

After the first block is cut and flushed this seems to go away.

[joe-elliott]

This is due to the queriers not find the trace in the ingesters here:

https://github.com/grafana/tempo/blob/master/modules/querier/querier.go#L138

and then tempodb returning an error b/c the requested tenant does not exist.

https://github.com/grafana/tempo/blob/master/tempodb/tempodb.go#L258

When the requested tenant does not exist tempodb should return a specific "not found" error which the querier then returns as a 404.