Support in-cluster ingress proxies

[linki]

Mate completely relies on data from the Kubernetes API to create DNS records. E.g. it gets all Ingress objects from your cluster, inspects the rules and external load balancer IP and creates the corresponding A records.

This works fine on GKE and GCE with Google's GLBC controller (https://github.com/kubernetes/contrib/tree/master/ingress/controllers/gce) managing their load-balancer-as-a-service as it updates the assigned IP of the load balancer to the status section of the Ingress object (https://github.com/kubernetes/contrib/blob/c7d250db7f17b8bd9f5d5eca66ddc1c6522d4e4e/ingress/controllers/gce/controller/controller.go#L364-L407).

However, in-cluster proxies that are used to implement the ingress functionality seem to not update the external IP of the corresponding Ingress objects, e.g. traefik and nginx (see #64 (comment)). It needs to be clarified if this is desired or if Mate should handle those cases better.

Nevertheless, on AWS it's possible to use https://github.com/zalando-incubator/kube-ingress-aws-controller as an in-cluster ingress proxy which does update the status section and works well in conjunction with Mate.

[ideahitme]

IMHO we should not try to support custom solutions, which do not write back the ingress status.
Having multiple sources of truth will not only be difficult to manage, but may lead to potential ambiguous decisions, e.g. having both traefik and ingress status storing different values, which should get the priority. I think having a single source of information is certainly the most reliable way to go. Furthermore, I think it is a sane requirement for all custom solutions to be in sync with K8S API server, at the end of the day ingress object has the IP/Hostname for a reason.

By the way it seems nginx controller does currently support updating ingress status (even though could not find it in the documentation):
https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/controller.go#L504

On the other hand, it might make sense to support custom producers, which serve as a source of service/ingress objects, but then K8S API should not be used at all IMO.

[linki]

@tolbrino seems like the nginx ingress controller should report back the correct load balancer IP. you want to give it another shot? which one did you use?

[tolbrino]

Strange. I'll try again and troubleshoot a little bit.

[linki]

Looks like there are two nginx based ingress controllers in https://github.com/kubernetes/contrib/tree/master/ingress/controllers. alpha definitely doesn't update the status.

[tolbrino]

So version 0.8.3 of the nginx-ingress-controller doesn't report the necessary information. However, I built and tested the version from the current master which works just fine with Mate and results in DNS entries for the services being proxied.

[linki]

That's lovely to hear. We should probably create an issue for traefik to support this as well.

[tolbrino]

Agreed. This seems like the way it should work.

[tolbrino]

I'm sorry for the my confusion, but upon further testing I noticed the following issue. nginx-ingress-controller writes the IP of the node which runs the respective Ingress into the status object. However, this alone doesn't work, since Mate will register the respective DNS entry with this IP, so all requests using the DNS entry will try to connect to a port on that IP which is not there. Instead the request should go through nginx which knows how to forward the request to the endpoint. Thus, in order to make this work the status object would need to be set to the IP running nginx instead.

[linki]

@tolbrino we just gave nginx-ingress-controller (v0.8.3) another look and found out that it works fine with mates design. Actually, mate makes it much more usable. Still, it's unsuited for production use cases.

So, you are right, the controller puts the external IP of the node where the controller pod is currently running on into the address field of the ingress. This is because the controller pod also runs the nginx that's serving the traffic. Consequently it sets up hostPort 80 and 443 in the pod. So, hitting the external IP of the node where the pod is currently running on (on port 80 or 443) and providing the correct host header will forward your traffic correctly to the service (of course, those ports need to be open on that node).

If you use it in conjunction with mate, then mate will create DNS records for the corresponding names to point to that node. At this point you don't need to care where the controller runs anymore. ❤️

However, your controller pod could die and be rescheduled on another node. Once it comes up it will change the IP address of all your ingresses to the new node IP. Here, mate takes over and modifies the DNS records accordingly.

You will have a downtime for all your ingresses whenever your controller pod dies 💔 but if you don't care about that then you can just wait for DNS to propagate and find all of your services at the same DNS name after a while.

If nginx-controller could run multiple instances concurrently then this could be mitigated, but we didn't check.

[tolbrino]

@linki Thanks for the thorough explanation. I guess I didn't test my setup properly, since I wasn't able to hit my proxied services via Mate's DNS records given the proper host headers. This might have something to do with the firewall settings for the various nodes, which would be an unrelated issue.

[linki]

@tolbrino mate currently doesn't map all target IP addresses of an Ingress or Service. This is addressed here but needs thorough testing on AWS before getting merged.

Meanwhile you can use this feature branch if you like.

If you're advantageous you can try it out in combination with https://github.com/linki/armor-ingress-controller which also gives you Ingress with automatic certificate retrieval. It runs as a DaemonSet on all your nodes instead of just one. Though, it's far less sophisticated than nginx + kube-lego.