Exemplar support with prometheus v2.26.0

[bobrik]

Prometheus v2.26 came out with exemplar support:

• https://github.com/prometheus/prometheus/releases/tag/v2.26.0

I can query them when talking directly to Prometheus, but when not when going via promxy (v0.0.70):

404 page not found

It would be great to support exemplars.

cc @prymitive

[PaulFurtado]

It appears @bobrik is getting a 404 error from this endpoint, however, we're actually getting a 500 with a panic on our end:

Error running handler: goroutine 2157390 [running]:
runtime/debug.Stack(0xc00169acc8, 0x2a5b2e0, 0x4c2cca0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/jacksontj/promxy/pkg/logging.(*ApacheLoggingHandler).runHandler.func1(0xc00169bad0)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/pkg/logging/logging.go:139 +0x46
panic(0x2a5b2e0, 0x4c2cca0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/prometheus/prometheus/web/api/v1.(*API).queryExemplars(0xc0005a8900, 0xc001913300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:506 +0x8ab
github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x380b730, 0xc0016dc9c0, 0xc001913300)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:265 +0xa5
net/http.HandlerFunc.ServeHTTP(0xc00068bb90, 0x380b730, 0xc0016dc9c0, 0xc001913300)
	/usr/local/go/src/net/http/server.go:2049 +0x44
github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x37d54a0, 0xc00068bb90, 0x7f034ab839e0, 0xc00558d620, 0xc001913300)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7e
github.com/prometheus/prometheus/web.(*Handler).testReady.func1(0x7f034ab839e0, 0xc00558d620, 0xc001913300)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/web/web.go:529 +0x55
net/http.HandlerFunc.ServeHTTP(0xc00068bbc0, 0x7f034ab839e0, 0xc00558d620, 0xc001913300)
	/usr/local/go/src/net/http/server.go:2049 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0x7f034ab839e0, 0xc00558d5f0, 0xc001913300)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:198 +0xee
net/http.HandlerFunc.ServeHTTP(0xc000bc1aa0, 0x7f034ab839e0, 0xc00558d5f0, 0xc001913300)
	/usr/local/go/src/net/http/server.go:2049 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(0x7f034ab839e0, 0xc00558d5f0, 0xc001913300)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:76 +0xb2
net/http.HandlerFunc.ServeHTTP(0xc000bc1c20, 0x7f034ab839e0, 0xc00558d5f0, 0xc001913300)
	/usr/local/go/src/net/http/server.go:2049 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0x380b010, 0xc001680f30, 0xc001913300)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:101 +0xdf
github.com/prometheus/prometheus/web.setPathWithPrefix.func1.1(0x380b010, 0xc001680f30, 0xc001913200)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/web/web.go:1151 +0x1dc
github.com/prometheus/common/route.(*Router).handle.func1(0x380b010, 0xc001680f30, 0xc001913100, 0x0, 0x0, 0x0)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/common/route/route.go:83 +0x27f
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000099f20, 0x380b010, 0xc001680f30, 0xc001913100)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/julienschmidt/httprouter/router.go:387 +0xc7e
github.com/prometheus/common/route.(*Router).ServeHTTP(...)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/common/route/route.go:121
main.main.func5(0x380b010, 0xc001680f30, 0xc001913100)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/cmd/promxy/main.go:410 +0x1ce
net/http.HandlerFunc.ServeHTTP(0xc0004b29e0, 0x380b010, 0xc001680f30, 0xc001913100)
	/usr/local/go/src/net/http/server.go:2049 +0x44
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000c62de0, 0x380b010, 0xc001680f30, 0xc001913100)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/vendor/github.com/julienschmidt/httprouter/router.go:448 +0x1d2
github.com/jacksontj/promxy/pkg/logging.(*ApacheLoggingHandler).runHandler(0xc000e021b0, 0x380b010, 0xc001680f30, 0xc001913100, 0x0, 0x0)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/pkg/logging/logging.go:142 +0x85
github.com/jacksontj/promxy/pkg/logging.(*ApacheLoggingHandler).ServeHTTP(0xc000e021b0, 0x38155a0, 0xc001344000, 0xc001913100)
	/usr/share/hubspot/build/workspace/InfraOrch/Docker/promxy/gopath/src/github.com/jacksontj/promxy/pkg/logging/logging.go:165 +0x225
net/http.serverHandler.ServeHTTP(0xc00019a2a0, 0x38155a0, 0xc001344000, 0xc001913100)
	/usr/local/go/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc0012bdf40, 0x381e198, 0xc001259cc0)
	/usr/local/go/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2993 +0x39b

@jacksontj in our case, we don't actually currently care about exemplar support, but this gives us large error output when we load grafana dashboards due to that panic. So prior to adding support for exemplars, I'm wondering if there's a simple fix for the panic. I'm guessing this only happens when access logs are enabled?

[jacksontj]

@PaulFurtado #475 fixes the panic issue

[sushovan23]

@jacksontj any plans on supporting exemplars? More and more time series DBs are adding support for it. It would great if promxy would also query it

[jacksontj]

@sushovan23 definitely interested in adding support. I don't have a driving need for it in any of my production systems at my day job; so this is falling to my "free time". So it is on the list to get to -- but no promises on when.

The part I expect will take me the most time is understanding the exemplar API and setup the most (as I don't see a public endpoint with exemplar support; so I believe I have to make some local E2E environment to do testing).

So if anyone is interested in contributing a PR, contributing an example environment, or sponsoring the feature -- any/all would be welcome :)

[sushovan23]

@jacksontj I have found this local setup which can be used to have sample app which produced exemplar data and feeding it into prometheus
https://vbehar.medium.com/using-prometheus-exemplars-to-jump-from-metrics-to-traces-in-grafana-249e721d4192

example is available here -> https://github.com/prometheus/client_golang/blob/main/examples/random/main.go#L99