Can VirtualServers use Services as upstream instead of endpoints?

[zaroman]

Ran into an issue where newly added VirtualServers were being considered invalid after an older VirtualServer was created but it's upstream endpoint got deleted shortly after. I found that there is an option for Ingress objects to use an annotation to use services as upstream (nginx.ingress.kubernetes.io/service-upstream: 'true'), but I cannot find documentation on how to do the same for VirtualServers. Is this possible to do?

[pleshakov]

Hi @zaroman

Using services as upstream, unfortunately, is not supported.

I wonder what problem it created? In case of 0 endpoints in upstream, NGINX will start returning 502 responses for client requests. However, you will get similar behavior if NGINX tries to connect to a service IP with zero endpoints behind.

[zaroman]

@pleshakov what we are trying to do is dynamically add/remove new VirtualHosts for new deployments that can happen at different points in time. If one of these deployments fail, but the VirtualServer is still around, all future attempts to create a new VirtualServer fail with errors like this:

stderr: "nginx: [emerg] host not found in upstream \"vs_default_id1-dynamic1_vsr_id1_dynamic1_dynamic1\" in /etc/nginx/conf.d/vs_default_id1-dynamic1.conf:85\n"
finished with error: exit status 1
2020/12/10 14:37:14 [emerg] 47552#47552: host not found in upstream "vs_default_id1-dynamic1_vsr_id1_dynamic1_dynamic1" in /etc/nginx/conf.d/vs_default_id1-dynamic1.conf:85
E1210 14:37:14.306135       1 controller.go:560] Error updating endpoints for [default/id2-dynamic2]: Error reloading NGINX when updating endpoints: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "nginx: [emerg] host not found in upstream \"vs_default_id1-dynamic1_vsr_id1_dynamic1_dynamic1\" in /etc/nginx/conf.d/vs_default_id1-dynamic1.conf:85\n"
finished with error: exit status 1
2020/12/10 14:37:15 [emerg] 47554#47554: host not found in upstream "vs_default_id1-dynamic1_vsr_id1_dynamic1_dynamic1" in /etc/nginx/conf.d/vs_default_id1-dynamic1.conf:85
E1210 14:37:15.833529       1 controller.go:560] Error updating endpoints for [default/id2-dynamic2]: Error reloading NGINX when updating endpoints: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "nginx: [emerg] host not found in upstream \"vs_default_id1-dynamic1_vsr_id1_dynamic1_dynamic1\" in /etc/nginx/conf.d/vs_default_id1-dynamic1.conf:85\n"
finished with error: exit status 1

In this case, the deployment for id1-dynamic1 failed, but the VirtualServer was not deleted, so when we tried to create id2-dynamic2 we got the error above and the VirtualServer for id2-dynamic2 was labeled as invalid even though it was correct and it's endpoint existed.

One thing to note is that even deleting the VirtualServer for id1-dynamic1 did not fix the issue for id2-dynamic2 after it was deemed invalid, but it did fix it for new VirtualServers created after deleting id1-dynamic1.

[pleshakov]

Hi @zaroman

This looks like a bug in our config generation. The error you see -- host not found in upstream -- means that the corresponding upstream wasn't generated for some reason.

If it is possible, could you share with us VirtualServer,VirtualServerRoute YAML files to reproduce this problem? and provide the generated config for the problematic virtual server (https://docs.nginx.com/nginx-ingress-controller/troubleshooting/#checking-the-generated-config)

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.