Skipper Ingress Controller support

[universam1]

fix: #452

✨ zalan.do/Skipper Router Implementation

An HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress
https://github.com/zalando/skipper/

• The concept is to define routes with specific weights via the skipper specific annotation predicate of zalando.org/backend-weights.
• apex Ingress is immutable
• A new "canary ingress" is created with to paths for primary and canary service that has higher "weight" hence receiving all traffic, which distributes progressively
• After the canary process is finished, this ingress is disabled via the False() annotation predicate to route traffic again back to the apex Ingress.

Skipper Principles:

There are certain Skipper principles which are taken into account:

• if only one backend has a weight, only one backend will get 100% traffic
• if two of three or more backends have a weight, only those two should get traffic.
• if two backends don't have any weight, it's undefined and right now they get equal amount of traffic.
• weights can be int or float, but always treated as a ratio.

📈 Prometheus Metrics Observer

• "request-success-rate" and "request-duration" queries are implemented and tested that provide those oberservations from Skipper metrics
• Takes into account how Skipper renders the paths accordingly and reformats the quieries.

Tests

• adding CircleCI tests
• 👷 Add high-level E2E test steps for Skipper
  • install Skipper ingress with Kustomize
  • load Flagger image onto the local cluster
  • install Flagger and Prometheus in the flagger-system namespace
• local testing - could be also saved as file

#!/usr/bin/env bash
REPO_ROOT=$(git rev-parse --show-toplevel)
cd $REPO_ROOT

make test
make build
docker tag weaveworks/flagger:latest test/flagger:latest
make loadtester-build
(kind get clusters && kubectl delete ns/test --force) || kind create cluster --wait 5m --image kindest/node:v1.16.9
./test/e2e-skipper.sh
# port forward prometheus UI to localhost:9090
kubectl port-forward $(kubectl get pods -l=app=flagger-prometheus -o name -n flagger-system | head -n 1) 9090:9090 -n flagger-system &

./test/e2e-skipper-tests.sh

Proof

Successful progress

Failed checks

{"level":"info","ts":"2020-08-14T10:07:36.357Z","caller":"controller/events.go:28","msg":"Halt podinfo.test advancement request duration 915ms > 500ms","canary":"podinfo.test"}                                                                                                                                    
{"level":"debug","ts":"2020-08-14T10:07:36.358Z","logger":"event-broadcaster","caller":"record/event.go:278","msg":"Event(v1.ObjectReference{Kind:\"Canary\", Namespace:\"test\", Name:\"podinfo\", UID:\"c1162c0e-409a-4825-9dac-a0a6c3418e72\", APIVersion:\"flagger.app/v1beta1\", ResourceVersion:\"3704\", FieldPath:\"\"}): type: 'Warning' reason: 'Synced' Halt podinfo.test advancement request duration 915ms > 500ms"}                                             
{"level":"debug","ts":"2020-08-14T10:07:51.341Z","caller":"router/skipper.go:155","msg":"GetRoutes primaryWeight: 60, canaryWeight: 40","GetRoutes":"podinfo.test"}                                                                                                                                                 
{"level":"info","ts":"2020-08-14T10:07:51.348Z","caller":"controller/events.go:28","msg":"Rolling back podinfo.test failed checks threshold reached 5","canary":"podinfo.test"}

Prometheus graphs

http://localhost:9090/graph?g0.range_input=5m&g0.moment_input=2020-08-13%2004%3A25%3A27&g0.expr=sum(rate(skipper_response_duration_seconds_bucket%7Broute%3D~%22kube(ew)%3F_test__podinfo_ingress_canary__.*__podinfo_service_canary(_%5B0-9%5D%2B)%3F%22%2Ccode!~%225..%22%2Cle%3D%22%2BInf%22%7D%5B15s%5D))%20%2F%20%0Asum(rate(skipper_response_duration_seconds_bucket%7Broute%3D~%22kube(ew)%3F_test__podinfo_ingress_canary__.*__podinfo_service_canary(_%5B0-9%5D%2B)%3F%22%2Cle%3D%22%2BInf%22%7D%5B15s%5D))%20*%20100%0A&g0.tab=0&g1.range_input=5m&g1.expr=sum(rate(skipper_serve_route_duration_seconds_sum%7Broute%3D~%22kube(ew)%3F_test__podinfo_ingress_canary__.*__podinfo_service_canary(_%5B0-9%5D%2B)%3F%22%7D%5B15s%5D))%20%2F%20%0Asum(rate(skipper_serve_route_duration_seconds_count%7Broute%3D~%22kube(ew)%3F_test__podinfo_ingress_canary__.*__podinfo_service_canary(_%5B0-9%5D%2B)%3F%22%7D%5B15s%5D))%20*%201000&g1.tab=0

[universam1]

FYI @webframp @szuecs

[szuecs]

From my side it looks good, but of course I have not much idea about flagger.
I am not 100% sure if the Marshal/Unmarshal with string() will work in Go1.15. Please check that.
I Just added some comments, which are no blockers.

[universam1]

I am not 100% sure if the Marshal/Unmarshal with string() will work in Go1.15.

@szuecs thanks for reviewing!
Looked up the go 1.15 changes like the ascii-character-number-to-string conversion deprecation which does not apply here - other than that I could not find anything in this regard!? Cannot find any other, more cannonical way to convert a []byte to string though. Anything else you have in mind?

[szuecs]

@universam1 it’s fine if you checked. I just remembered “something changed in that area”, so please ignore my comment. 😊

[universam1]

@stefanprodan I think this PR is now ready for merge. Please note the added tutorial for Skipper

[stefanprodan]

LGTM

Awesome work Samuel and David 🏅

Looking forward to the A/B testing implementation if you want to tackle that.