Watchdog: introduce endpoints for health and metrics

[alexellis]

Update the OpenFaaS Watchdog to introduce two new endpoints:

• /_/health

This should check for the lock file at /tmp/.lock - use the existing behavior for this.

• /_/metrics

Vendor Prometheus go library and expose:

Add a Prometheus counter:

function_invocation{}

This counter should represent the requests/second.

Need to review Prometheus naming guidelines before approving the name function_invocation for the bucket. https://prometheus.io/docs/practices/naming/

Once complete - make equivalent change to the of-watchdog here:

https://github.com/openfaas-incubator/of-watchdog

Worries / other concerns

• This may be a large chunk of work - so I would prefer two separate Pull Requests for this
• There will be a size increase of watchdog binary when adding Prometheus go library, we need to evaluate this before merging.
• This is the first time the watchdog will override any URL - normally these are marshaled directly to the underlying function

[viveksyngh]

I can start looking this if it is not urgent.

[stefanprodan]

Adding the Prometheus client will bring the watchdog binary size to 6MB

[alexellis]

Thanks for commenting. The current size is around 3.92MB - so adding instrumentation almost doubles the size of the binary.

[viveksyngh]

@alexellis for _/health endpoint, what if .lock suppressed with suppressLock configuration? We need to return 404 in that case.

[alexellis]

Great question.. Can you check what Kubernetes expects to fail a health-check? (liveness probe)https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-a-liveness-http-request

[viveksyngh]

Anything greater than 200 to less than 400 is a success. Any other code indicates failure. For failed, the example uses 500 response.

[alexellis]

Sounds good to me

[kdubuc]

Dumb question, but what is the difference betweek HEALTHCHECK Dockerfile and the /_/health endpoint ? They returns the same information. OpenFaaS cannot read the docker service HEALTHCHECK status ?

[alexellis]

I have done this work now