add prometheus service into docker compose and add a basic metric to frontend

[fatsheep9146]

add prometheus service into docker compose and add a basic metric to frontend

Signed-off-by: Ziqi Zhao zhaoziqi9146@gmail.com

Fixes #70

Changes

1. add prometheus service into docker compose
2. add a basic metric http.server.request_count to frontend

I attempted to add metric support to webstore demo in my own mind. The implementation may be too simple. I only add one metric for frontend service and add a prometheus service to scrape metric from otel-collector. I really hope the comminuty can give more advises, for examples,

• add more metrics
• add more metric visualization tool, such as grafana

[fatsheep9146]

add some screen shot about the metric scraped by prometheus.

you can view prometheus ui in http://127.0.0.1:9090

[fatsheep9146]

@cartersocha @puckpuck @julianocosta89 Could you help review this ?

[julianocosta89]

@fatsheep9146 could you take a look at the yaml lint and sanity check issues?
Regarding the PR itself it looks good, but I do not have much experience with metrics, maybe some of the other approvers have more experience, so I'd rather wait a bit and see if we get any insights from them.

[cartersocha]

FYI Austin & I are at OTel community day on Monday & won’t be able to take a look until later this week

thanks for making progress on this tho @fatsheep9146. It’s an area that needs a lot more attention

[reyang]

Regarding the overall direction of metrics: #70 (comment) mentioned that "services should use push metrics where possible" which sounds like OTLP exporter scenario?

Here is my understanding:

                       +-------------------------+
SDK + OTLP Exporter ---> OpenTelemetry Collector <----- Prometheus Scraper ----> TSDB
                       +------------+------------+
                                    |
                                    +---------> Vendor Ingestion

[fatsheep9146]

exporter

Regarding the overall direction of metrics: #70 (comment) mentioned that "services should use push metrics where possible" which sounds like OTLP exporter scenario?

Here is my understanding:

                       +-------------------------+
SDK + OTLP Exporter ---> OpenTelemetry Collector <----- Prometheus Scraper ----> TSDB
                       +------------+------------+
                                    |
                                    +---------> Vendor Ingestion

Yes, this is same as my understanding.

[reyang]

LGTM with some non-blocking suggestions.

[wph95]

LGTM

[cartersocha]

@austinlparker Are we okay using Prometheus as our metric consumption tool?

I know we want to produce OTel metrics from instrumentation. I’m not a go expert but it looks like this update uses OTLP metric submission which should meet that requirement?

Then it’s just if we prefer Prometheus over other tools

[austinlparker]

I think prometheus is probably ok -- realistically, what are our options here in terms of a sane default? I think the req behind the req is that we need to demo both pull and push metrics; the actual place they get sent isn't really that important from our perspective. An end-user should be able to swap out the prom exporter with any other OTLP metrics endpoint and it'd just work, right?

[reyang]

I think prometheus is probably ok -- realistically, what are our options here in terms of a sane default? I think the req behind the req is that we need to demo both pull and push metrics; the actual place they get sent isn't really that important from our perspective. An end-user should be able to swap out the prom exporter with any other OTLP metrics endpoint and it'd just work, right?

I would suggest Prometheus as the TSDB (Timeseries Database) and Grafana as the UI. Here is one example in OpenTelemetry C++ and I found Grafana UI to be much more usable than Prometheus UI https://github.com/open-telemetry/opentelemetry-cpp/blob/a847d0ce42209e458f8f98a5751626a33adfad95/examples/prometheus/README.md.

[puckpuck]

1 small nit pic but LGTM with the histogram metric.

[fatsheep9146]

I think all conversations are resolved, and ready to be merged. :) @cartersocha

[cartersocha]

Ok merging now. Let’s get a basic grafana dashboard running soon to consume this. I think this is using Prometheus’s front end still but no need to block.

[cartersocha]

@fatsheep9146 could you separately update the change log to reflect metrics being added. I think this qualifies as significant enough

[fatsheep9146]

@fatsheep9146 could you separately update the change log to reflect metrics being added. I think this qualifies as significant enough

Absolutely, except changelog , I think I should also update the ReadMe of the Webstore and the frontend service?

[fatsheep9146]

Ok merging now. Let’s get a basic grafana dashboard running soon to consume this. I think this is using Prometheus’s front end still but no need to block.

I will work on this right now

[cartersocha]

@fatsheep9146 could you separately update the change log to reflect metrics being added. I think this qualifies as significant enough

Absolutely, except changelog , I think I should also update the ReadMe of the Webstore and the frontend service?

Yeah let’s flesh out the documentation. Maybe we set up dedicated signal docs that holds all the links to various implementations & the consumption / storage details too?

In each service’s readme we can keep the generic feature table then some code highlights

[cartersocha]

No need to start this structure now but something for us to consider in the next couple weeks as we clean stuff up

[aengusrooneygrafana]

@fatsheep9146 the dashboard provider is missing. Will I raise an issue or a PR to fix this? You need to add a block for your dashboard provider, in addition to providing your dashboard.json.