Nginx ingress controller listening port changed to 442 after adding enable-ssl-passthrough: "" in helm for nginx ingress.

[kmarimuthu90]

What happened:

We have to support IPv6 clients, but APIM is IPv4 only. So, we plan to use a dual-stack Nginx ingress controller to support both IPv4 and IPv6 and forward the traffic to the IPv4 backend ( APIM ).

How it works usually :
The devices connect with APIM by sending requests and client certificates for the handshake. We have configured the policy like below in API to verify the client's certification in the request.

<choose>
            <when condition="@(context.Request.Certificate == null || !context.Request.Certificate.VerifyNoRevocation())">
                <return-response>
                    <set-status code="403" reason="Invalid client certificates" />
                </return-response>
            </when>
        </choose>

TLS termination is handled at the APIM custom domain. Flow is Client device -> APIM DNS ( Cloud Flare ) -> APIM

What you expected to happen:

New dual stack setup:
We have deployed a dual-stack ingress controller on the AKS cluster and it is exposed to the 443. And ingress route to forward the traffic to the backend external service ( APIM ). But I see 442 alone available to access after we enabled the SSL passthrough.

We want to forward client requests and cert to backend APIM where we handle the TLS termination client cert validation

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
Helm chart:ingress-nginx-4.11.3
App version 1.11.3

Kubernetes version
1.29.X

Environment:

• Cloud provider or hardware configuration: Azure

• OS (e.g. from /etc/os-release): Ubuntu

• Kernel (e.g. uname -a): Linux nginx-ingress-ingress-nginx-controller-78b7467cf7-9lspz 5.15.0-1073-azure Fix incorrect X-Forwarded-Port for TLS #82-Ubuntu SMP Mon Sep 2 11:36:34 UTC 2024 x86_64 Linux

• Install tools:

  • Helm chart used to install ingress on AKS cluster.

• Basic cluster related info: AKS cluster

  • kubectl version - 1.29.x
  • `kubectl get nodes -o wide
  • kubectl get nodes -o wide
    NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
    aks-emqx00-16705065-vmss000000 Ready 16h v1.29.8 10.1.0.7 Ubuntu 22.04.5 LTS 5.15.0-1073-azure containerd://1.7.22-1
    aks-nodepool1-52744880-vmss000000 Ready 17h v1.29.8 10.1.0.5 Ubuntu 22.04.5 LTS 5.15.0-1073-azure containerd://1.7.22-1
    aks-nodepool1-52744880-vmss000001 Ready 17h v1.29.8 10.1.0.6 Ubuntu 22.04.5 LTS 5.15.0-1073-azure containerd://1.7.22-1
    aks-nodepool1-52744880-vmss000002 Ready 17h v1.29.8 10.1.0.4 Ubuntu 22.04.5 LTS 5.15.0-1073-azure containerd://1.7.22-1`

• How was the ingress-nginx-controller installed:
  helm list -ningress NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION nginx-ingress ingress 33 2024-10-29 08:20:13.0357821 -0700 PDT deployed ingress-nginx-4.11.3 1.11.3

Pls find the attachment for below,

• ingress controller svc
• pod
• helm values.yaml for ingress controller
• Ingresss
• Backend service

How to reproduce issue:

AKS cluster 1.29 ( dual stack )
Deploy dual stack nginx ingress and enable ssl passthrough
create ingress to forward traffic to backend
backend service to connect APIM
Check the listening ports in ingresss controller pod, it is opened for 442 and 80

We wanted to connect 443

https://api-dev.insg-pegasus.com -> CF -> ingress ipv4 IP:443 -> ingress -> backend 443 ( APIM )

backend-svc.txt
ingress.txt
pod-ingress-controller.txt
values.txt
svc.-ingress-controller.txt

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind bug
/kind support

Duplicate of #12262

@kmarimuthu90 once again unfortunate that its not working for you. Please consider these facts

• The example I provided in the other issue https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#kubernetesingress-nginx is basically a piece of data that implies that the ssl-passthrough feature is not broken and working for every-one is the documented & prescribed config

• Hence the error message you report <set-status code="403" reason="Invalid client certificates" />" is not caused by the controller. Its a error caused by the environment and the use case

• Some suggestions based on history and the data you have posted here are below

  • If you copy/paste the commands and outputs as asked in the bug report template, right there in the issue-description, in markdown format, it provides a single window pane to co-relate and associate the controller, the ingress, the request, the logs and the response to the request. The attached files are not a problem but a different experience from that single window pane that is so helpful
  • The ingress resource shows a annotation for backend-protocol: HTTPS but the termination of TLS is not in the controller pod, so what is the reason for configuring that annotation. Its not used in the ArgoCD example as well
  • The error message is stating clearly that the client-cert is invalid. Here the most important aspect is that the controller only is accountable for processing routing rules and the controller is not the one presenting the server-certificate. So this error has nothing to do with the ingress-controller
  • You are reporting port 442 as a problem. Why ? Which documentation are you referring to ? The details of that listening port number 442 are a dive into the implementation of the ssl-passthrough feature and I already proved to you that the ssl-passthrough is used so widely that OTHER projects like ArgoCD have a documented example to use the ssl-passthrough feature. So the port number 442 is not a matter of discussion

• Beyond this, I suspect the problem is a misconfiguration. But after repeated suggestions, you have not helped in the way that really really helps the readers here in providing data. For example if you answered all the questions asked in the issue template, in a markdown format, and copy/pasted the answers in the issue description, one of the information that would emerge is the exact curl command or browser URL, that was the request sent to the ingress. Then it would be possible to relate that request to the ingress rule. And then a look at the logs would help co-relate some events

So please wait for other comments as well. And in the meanwhile, in case you want to help out the readers more, then follow the instructions in the new bug report template, and copy/paste the commands and their outputs, in the issue description of this issue, in markdown format.

But there is shortage of resources, so if its urgent, then you can take this discussion to kubernetes.slack.com in the ingres-nginx-users channel, because there are mnay many more experts and users there, round the clock.

thanks.

[longwuyuan]

hi @kmarimuthu90 ,

Keeping this issue open with its data means the details are unknown. But the fact that the project knows the port 442 and that you have not even copy/pasted the command that shows port 442 in this issue, does not help add to the list of action items pending on the project.

So I will close this issue for now. After you have posted the data as suggested and changed the title to not reflect port 442, you can reopen the issue to continue discussion about what error you are getting in response to your request sent to the ingress.

To be clear one more time. Port 442 is the implementation of the ssl-passthrough and its not a focus of discussion as a problem to be solved by the project.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

hi @kmarimuthu90 ,

Keeping this issue open with its data means the details are unknown. But the fact that the project knows the port 442 and that you have not even copy/pasted the command that shows port 442 in this issue, does not help add to the list of action items pending on the project.

So I will close this issue for now. After you have posted the data as suggested and changed the title to not reflect port 442, you can reopen the issue to continue discussion about what error you are getting in response to your request sent to the ingress.

To be clear one more time. Port 442 is the implementation of the ssl-passthrough and its not a focus of discussion as a problem to be solved by the project.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

duplicate of #12262