Read only prometheus with timescaledb remote storage

[jpigree]

Hi. I have trouble setting up a prometheus instance which only reads data from a timescaleDB remote storage.

What I tried to do

My idea is to have 2 prometheus instances. One which writes data in a timescale db and another which reads the data from the same timescale db. I use the timescale-observability chart to setup timescaledb + timescale-prometheus (read) + prometheus + grafana. The other prometheus (write) is installed separately in another location.

The result

The "write" prometheus works fine and I can see tables and data in timescaledb. I can even query it with the timescaleDB connector in grafana. However, my "read" prometheus instance can't see anything. It is empty (querying endpoint api/v1/label/__name__/values shows no data). I double checked the prometheus configuration but it is fine. It is actually automatically generated by the chart.

I found very few docs on the remote_read feature of prometheus so I struggle to know if I understand it correctly. From what I saw, prometheus will read data from the remote storage by querying it with matchers and time ranges.Does this means we have to know the metrics beforehand and query them at least one time to make them visible in prometheus?

I also found a few issues similar to mine on the previous adapter:

• How can read metrics data from timescaledb(pg_prometheus) and pass it to prometheus via prometheus-postgresql-adapter prometheus-postgresql-adapter#81
• remote read is not showing metrics prometheus-postgresql-adapter#68
• Problem with remote read prometheus-postgresql-adapter#66

Questions

In short:

• Is it possible to setup a "read-only" prometheus with a timescaledb remote storage?
• Is it encouraged or should we connect directly to the timescaledb database with grafana instead?
  I would prefer to use prometheus in between because it exists much more grafana dashboards for prometheus datasource.
• If we can do the "read-only" setup, how can I troubleshoot my read problems?
  I have no error logs either in prometheus or the adapter.

Thanks for your help!

[jpigree]

Damn, I tried to query the data directly in Grafana after creating this ticket. Yes, the data is available this way. It is just weird that prometheus won't show any of those data itself. We have to query it without any autocompletion. :/

[sammy-da]

Hello!

I have the exact same problem.
I need a read-only prometheus hooked up to TSDB, which I'll be attaching an Alert Manager to.

Other prometheus instances are writing to TSDB, and I can see the metrics in it.
I even see the metrics if I curl the /query endpoint of Promscale manually.

The read-only Prometheus, on the other hand, is failing to read.
The prometheus dashboard gets:

• {"status":"success","data":{"resultType":"vector","result":[]}} for /query
• {"status":"success","data":[],"warnings":["not implemented"]} for /label/__name__/values

Yes, the remote_read is configured correctly afaict. I checked the configuration page.

[jfojfo]

make sure you have the same external label configs for all your prometheus instance.
prometheus queries will add external labels to filter metrics automatically.