Add graceful termination

[pleshakov]

• When the controller receives SIGTERM, it shutdowns itself as well as NGINX, using nginx -s quit.
• If NGINX master process exists for some reason (for example, the master crashes), the controller exits as well.
• NGINX is no longer running as a daemon

[akuzni2]

I still see shutdowns where the IC scales and my LB returns 500 errors from Nginx.

How does the Ingress Controller prevent the Nginx pod being shutdown to stop accepting new connections? My suspicion is that the IC service still sees the pod as a potential endpoint and will route requests to it during it's termination process.

[pleshakov]

Hi @akuzni2

How does the Ingress Controller prevent the Nginx pod being shutdown to stop accepting new connections? My

When Kubernetes terminates the Ingress Controller pod, k8s sends a SIGTERM. The IC process sees that and in turn gracefully shutdowns NGINX (nginx -s quit). Once NGINX quits, the Ingress Controller pod finishes. Gracefully means NGINX will finish serving any pending requests.

Note that after Kubernetes sends SIGTERM, it waits for 30s (configurable) and if the pod doesn’t finish by that time, k8s will kill it. If NGINX is load balancing websocket traffic or any other traffic with long lived connections, it will not quit until those connections are terminated - either by the client or the backend. As a result, the Ingress Controller pod might not finish in those 30s, so Kubernetes will kill it. That could lead to errors seeing by clients/backends, as connections are unexpectedly terminated.

My suspicion is that the IC service still sees the pod as a potential endpoint and will route requests to it during it's termination process

We can also use a prestop hook to create a delay between the pod recieving SIGTERM and NGINX starting a graceful shutdown, to make sure the corresponding endpoint removal is seen by the fronting load balancer. For that, we can add the following to the pod spec:

    lifecycle:
      preStop:
        exec:
          command:
          - sh
          - -c
          - sleep 15

this blog post explains this topic in detail -- https://blog.gruntwork.io/delaying-shutdown-to-wait-for-pod-deletion-propagation-445f779a8304

[akuzni2]

@pleshakov wow thank you very much for the detailed response.

We can also use a prestop hook to create a delay between the pod recieving SIGTERM and NGINX starting a graceful shutdown, to make sure the corresponding endpoint removal is seen by the fronting load balancer. For that, we can add the following to the pod spec:

Just curious - if this is known is there any reason why it isn't included in this ingress controller?

[pleshakov]

@akuzni2

Just curious - if this is known is there any reason why it isn't included in this ingress controller?

there weren't a lot of requests for it nor a PR, so different things got implemented