Metrics router is a router for Custom and External Metrics. It allows the use of multiples metric servers as sources for the K8S autoscaling controllers.
⚠️ This project should be considered as a proof of concept used to experiment around the ideas proposed in this KEP.
% kubectl apply -f https://raw.githubusercontent.com/barkbay/metrics-router/main/config/deploy/all-in-one.yaml
Check that the metrics server is up and running:
% kubectl get pods
NAME READY STATUS RESTARTS AGE
metrics-router-controller-manager-5474f885c5-jtwmw 2/2 Running 0 53
% kubectl apply -f https://raw.githubusercontent.com/barkbay/metrics-router/main/config/deploy/api-service.yaml
cat <<EOF | kubectl apply -f -
apiVersion: metricsrouter.io/v1alpha1
kind: MetricsSource
metadata:
name: prometheus
spec:
insecureSkipTLSVerify: true
priority: 100
metricTypes:
- CustomMetrics
service:
namespace: custom-metrics
name: prometheus-metrics-apiserver
port:
number: 443
EOF
You can check if the resource has been loaded using kubectl get ms
:
% kubectl get ms
NAME SERVICE PORT SYNCED METRICS
prometheus custom-metrics/prometheus-metrics-apiserver 443 true 476
SYNCED
reports if the metrics list has been successfully retrieved from the metrics source backend.- The number of metrics loaded is displayed in the
METRICS
columns.
If a metric is served by more than one backend, the metrics source with the higher priority
is used. The higher the value, the higher the priority. Having two metrics sources with the same priority should be avoided, in such a case the metrics sources are sorted by name.
% kubectl logs -l control-plane=controller-manager -c server -f
I0624 07:11:55.995653 1 controller.go:70] syncing metrics from /prometheus
I0624 07:11:55.995728 1 registry.go:67] Update metrics source prometheus
I0624 07:11:56.027753 1 controller.go:94] 61 metrics loaded from /prometheus
I0624 07:12:19.155079 1 registry.go:201] custom metric pods/foo(namespaced) served by https://custom-metrics-apiserver.custom-metrics.svc:443
I0624 07:12:19.155114 1 metricsclient.go:190] custom metric info: provider.CustomMetricInfo{GroupResource:schema.GroupResource{Group:"", Resource:"pods"}, Namespaced:true, Metric:"foo"}
HPA events can provide useful information to understand why metrics are not retrieved, for example using the describe
subcommand:
% kubectl describe hpa/my-hpa -n my-ns
Name: my-hpa
Namespace: my-ns
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 11s horizontal-pod-autoscaler failed to compute desired number of replicas based on listed metrics for Resource/ns/resource: invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric foo: unable to fetch metrics from custom metrics API: custom metric foo is not provided by any metrics backend
The core algorithm is a fork from https://github.com/arjunrn/custom-metrics-router