-
Notifications
You must be signed in to change notification settings - Fork 670
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ingress addresses not set unless using a LoadBalancer #2539
Comments
Hi @al45tair, thanks for this issue. You're right, we only supported watching the service and basic addressing in the first cut of this feature. Do you know how nginx-ingress builds the list of IPs that it puts in there? We used the Envoy service-watching method because it's what we use in our example deployment, and I hoped that the |
@youngnick As it happens, I do know how ingress-nginx (sorry, got the name the wrong way around; very confusing that there are two) does that, because I had to file a bug against Rancher because the latter breaks the mechanism ingress-nginx uses, which is ironic because ingress-nginx is what Rancher installs by default. If you've got the ingress-nginx source code handy, if you look in internal/ingress/status, there is a function If you were going to do something similar, you'd want to avoid the label problem (or it'll break on Rancher), and I suppose you'd want to look for the envoy Pods. |
@youngnick A couple more thoughts on this: while ingress-nginx just does this at a point in time (so if the set of nodes changes, you'd have to kick it to get it to update the ingresses), I think the right way to do this is to explicitly watch the Envoy Pods for NodeName changes and then watch the resulting set of Nodes for IP address changes. That could feed into the existing mechanism in Contour that updates ingresses when the LoadBalancer address changes. |
Yes, I think if we're going to take addresses from the node, we should make sure it's updated whenever the Envoy pods are. It's quite a bit of additional complexity, but at least the Service status watcher gives us a pattern for it. I'll have a think about how we would build that and put something in here. |
I've added the
|
They simply have more complex logic for pod discovery and service parsing: By default ingress-nginx without any options just describe itself: https://github.com/kubernetes/ingress-nginx/blob/38c73233f3db84866892c68845fa188802550d0e/internal/ingress/status/status.go#L188 So for Contour that is not so simple, because we need a way to specify how to find envoy pods (by service?)
|
Thanks for those notes @nefelim4ag, that's excellent. |
Something like that:
So just for your information For publishing, dirty hack used, external-dns NodePort annotation with CNAME.
So, external-dns create A record with ExternalIP addresses of envoy nodes and contour just set name of A record as CNAME for ingress/HTTPProxy resources. Also, we disable the shutdown manager, use TerminationGraceTimeout like 600s, because we expect the migration to other nodes by DNS drain, and we try to avoid traffic spikes, because nodes, while slow shutdown, still serve traffic requests from clients. So, generally speaking, anyone can already run contour in the same setup as Nginx Ingress.
|
What steps did you take and what happened:
Installed Contour and Envoy in a cluster that doesn't have LoadBalancer support. Per the instructions, I configured Envoy to run in a DaemonSet on all the nodes and to use host networking, which is basically the same thing that the community ingress-nginx controller does in the same situation.
Then I created an Ingress to test it.
What did you expect to happen:
I expected the Ingress to list the cluster IPs. Instead I saw
Anything else you would like to add:
The problem appears to be that Contour watches the "envoy" service to get the addresses to apply to ingresses. If it can't find the "envoy" service, then unless you either add annotations to your Ingress or use the
--ingress-status-address
option, Contour sets an empty LoadBalancerStatus.Adding annotations is annoying (you don't need to do that with ingress-nginx, after all), and the
--ingress-status-address
option only supports a single address (so, for a multi-node cluster where more than one node can serve for the ingress, you're stuck using a DNS name, which would make External DNS provision a CNAME record, which is undesirable because CNAMEs clash with everything). I'll file a separate issue about--ingress-status-address
only supporting a single address.Environment:
kubectl version
): Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5", GitCommit:"e0fccafd69541e3750d460ba0f9743b90336f24f", GitTreeState:"clean", BuildDate:"2020-04-16T11:35:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/arm64"}
/etc/os-release
): Ubuntu 20.04 LTSThe text was updated successfully, but these errors were encountered: