Let the proxy pod expose Prometheus metrics

[consideRatio]

With configurable-http-proxy version 4.4.0 just released, there is support for exposing prometheus metrics on a given port.

But, it is not obvious to me where to add annotations to help prometheus scrape this metrics.

• The CHP/proxy pod should run in a single replica.
• The CHP/proxy pod is exposed behind the k8s service named proxy-http if autohttps is enabled, or behind proxy-public if it isn't.
• The hub pod's k8s service is annotated so the hub pod is scraped.

Option 1: We could apply the prometheus scraping annotations to either the proxy-http service or the proxy-public service depending on where its located.
Option 2: We could apply the prometheus scraping annotations to the proxy pod itself

Those are the viable options I think, and I'm not sure what makes sense or how the details work out.

Action points

• Configure --metrics-port=<some-port> when starting configurable-http-proxy
• Decide where to configure scraping annotations
  • Configure annotations

Related

jupyterhub/configurable-http-proxy#314

zero-to-jupyterhub-k8s/jupyterhub/templates/hub/service.yaml

Lines 7 to 19 in 714a3d2

	annotations:
	{{- if not (index .Values.hub.service.annotations "prometheus.io/scrape") }}
	prometheus.io/scrape: "true"
	{{- end }}
	{{- if not (index .Values.hub.service.annotations "prometheus.io/path") }}
	prometheus.io/path: {{ .Values.hub.baseUrl | trimSuffix "/" }}/hub/metrics
	{{- end }}
	{{- if not (index .Values.hub.service.annotations "prometheus.io/port") }}
	prometheus.io/port: "8081"
	{{- end }}
	{{- with .Values.hub.service.annotations }}
	{{- . | toYaml | nindent 4 }}
	{{- end }}

[a3626a]

is it possible just to allow configuring extraPorts for the chp container?
I can expose prometheus metrics by

• adding extraCommandLineFlags to proxy.chp
• adding extraPorts to proxy.service
• AND adding ports to chp container by kubectl. (not helm)

  proxy:
    chp:
      extraCommandLineFlags:
      - "--metrics-port=8080"
    service:
      extraPorts:
      - name: metric
        port: 8080
        targetPort: metric

apiVersion: apps/v1
kind: Deployment
metadata: ...
spec:
  ...
  template:
    metadata: ...
    spec:
      ...
      containers:
      - command:
        - configurable-http-proxy
        ...
        - --metrics-port=8080
        ...
        ports:
        - containerPort: 8000
          name: http
          protocol: TCP
        - containerPort: 8001
          name: api
          protocol: TCP
        // this should be able to added by helm value.
        - containerPort: 8080
          name: metric
          protocol: TCP
        ...

additionalScrapeConfigs for kube-prometheus-stack

      additionalScrapeConfigs:
        - job_name: jupyterhub
          metrics_path: "/hub/metrics"
          scheme: https
          static_configs:
            - targets:
                - ...
          authorization:
            credentials_file: "/etc/prometheus/secrets/jupyterhub/token"
        - job_name: proxy
          metrics_path: "/metrics"
          scheme: http
          static_configs:
            - targets:
                - jupyterhub-proxy-public.jupyter-hub.svc.cluster.local:8080