[WIP] Add metrics endpoint to watchdog

[alexellis]

Signed-off-by: Alex Ellis (VMware) alexellis2@gmail.com

Description

Uses Prometheus client library. This commit is WIP and will need
to be altered to provide RED metrics in the format expected by
K8s HPAv2 / Weave Cloud / Prometheus operator.

By collecting metrics at the watchdog level we can enable Kubernetes HPAv2 to work on requests/per/second.

Motivation and Context

• I have raised an issue to propose this change (required)

#583 #532

How Has This Been Tested?

WIP - shows that the metrics are showing up.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I've read the CONTRIBUTION guide
• I have signed-off my commits with git commit -s
• I have added tests to cover my changes.
• All new and existing tests passed.

[alexellis]

@stefanprodan can you link back to the RED metrics format with the histogram/counter expected by the tooling?

[stefanprodan]

These are the Prometheus metrics names and format that the tooling expects:

HPA v2 http_requests_total

	counter := prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Subsystem: "http",
			Name:      "requests_total",
			Help:      "The total number of HTTP requests.",
		},
		[]string{"status"},
	)

RED http_request_duration_seconds

	histogram := prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Subsystem: "http",
		Name:      "request_duration_seconds",
		Help:      "Seconds spent serving HTTP requests.",
		Buckets:   prometheus.DefBuckets,
	}, []string{"method", "path", "status"})

A full example of a HTTP server instrumentation can be found here https://github.com/stefanprodan/k8s-podinfo/blob/master/pkg/server/instrument.go

[stefanprodan]

The recommend way of exposing the metrics endpoint is with github.com/prometheus/client_golang/prometheus/promhttp. The handler should be http.Handle("/_/metrics", promhttp.Handler())

[alexellis]

This was put out for discussion and you didn't raise an issue with it.

#547

We agreed to use a private namespace for the routes of /_/ so that it could not clash with the function in the container. Same for the health endpoint.

[alexellis]

I think there was a reason I moved back to this version of the handler. How to the metrics get registered and associated with the first method/approach?

[LucasRoesler]

@alexellis and @stefanprodan is this something we still want to do?

[alexellis]

Yes

[alexellis]

Derek close: replaced by #1151