support prometheus ingresses probing with the same annotation than services

[eburghar]

Description

Monitoring https ingresses with blackbox gives some useful insights like ssl expiry you don't have if
you just monitor the associated service with http. Also in case your service only serve https with a public certificate, monitoring the associated service gives you a certificate error because the target of blackbox is service.namespace.svc which doesn't match the name associated with the certificate.

Steps to Reproduce

Expected behavior

Environment

Possible Solution

add a new job to prometheus config

- job_name: "kubernetes-ingresses"

  metrics_path: /probe
  params:
    module: [http_2xx]

  kubernetes_sd_configs:
    - role: ingress

  relabel_configs:
    - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_ingress_scheme, __address__, __meta_kubernetes_ingress_path]
      regex: (.+);(.+);(.+)
      replacement: ${1}://${2}${3}
      target_label: __param_target
    - target_label: __address__
      replacement: blackbox
    - source_labels: [__param_target]
      target_label: instance
    - action: labelmap
      regex: __meta_kubernetes_ingress_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_ingress_name]
      target_label: kubernetes_name

add change prometheus clusterrole to allow listing of ingress

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: ['']
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  - ingress
  verbs: [get, list, watch]
- nonResourceURLs: [/metrics]
  verbs: [get]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch

Now you just have to add the annotation prometheus.io/probe: "true" to you ingresses

[eburghar]

also add this prometheus alerting rule as a good measure: see Get alerted before your SSL certificates expire

ssl_expiry.rules

groups: 
  - name: ssl_expiry.rules 
    rules: 
      - alert: SSLCertExpiringSoon 
        expr: probe_ssl_earliest_cert_expiry{job="blackbox"} - time() < 86400 * 30 
        for: 10m

and add this grafana dashboard Prometheus Blackbox Exporter

[dghubble]

The standard Prometheus Ingress scrape config (for those with a Blackbox exporter) is ok. But I don't want to expand to providing Blackbox Exporter manifests or graphs (also I don't copy-paste others' graphs) or tracking those releases. Users can manage that.