AKS Load Balancer's IP instead of real IP used in whitelist check

[wimi]

NGINX Ingress controller version: 0.28, 0.29, 0.30, 0.31.1

Kubernetes version (use kubectl version): v1.15.7

Environment:

• Cloud provider or hardware configuration: Azure AKS
• OS (e.g. from /etc/os-release): 16.04.6 LTS (Xenial Xerus)
• Kernel (e.g. uname -a): 4.15.0-1077-azure Fix incorrect X-Forwarded-Port for TLS #82-Ubuntu SMP Thu Mar 26 03:25:09 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: helm
• Others:

What happened:

We are currently using NGinx ingress controller 0.25 and ip whitelisting on our ingresses - using standard nginx.ingress.kubernetes.io/whitelist-source-range labels. It works perfectly fine.

We've updated the NGinx to the newest releases and they do work mostly fine, but sometimes instead of real client IP address, Load Balancer's IP address is used for whitelist check.

The following messages are then in the NGinx logs:

{"ts": "2020-04-28T08:10:53+00:00", "requestID": "33426cd08c73b94b097072b14fc3573a", "upstreamAddr": "", "proxyUpstreamName": "xxx-xxxxxxx-8080", "proxyAlternativeUpstreamName": "", "requestMethod": "GET", "requestUrl": "xxx.xxxx.com/?q=", "status": 403, "upstreamStatus": "", "requestSize": "27", "responseSize": "", "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36", "remoteIp": "1.2.3.4", "referer": "", "latency": ""}

2020/04/28 08:10:53 [error] 4975#4975: *5303 access forbidden by rule, client: 1.2.3.4, server: xxx.xxxx.com, request: "GET /favicon.ico HTTP/2.0", host: "xxx.xxxx.com", referrer: "https://xxx.xxxx.com/?q="

Where 1.2.3.4 is an external IP of our Load Balancer.

The situation gets back to normal after a while (real IP is used as expected), but breaks after random time period again.

What you expected to happen:

Real client IP is used for whitelist validation all the time.

How to reproduce it:

Ingress controller service definition

apiVersion: v1 
kind: Service
metadata: 
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: XXXX
  labels: 
    app: nginx-ingress
    chart: nginx-ingress-1.36.0
    component: controller
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-ingress-controller
  namespace: ingress
spec:
  clusterIP: 10.0.100.74
  externalTrafficPolicy: Local
  healthCheckNodePort: 32737
  loadBalancerIP: 1.2.3.4
  ports:
  - name: http
    nodePort: 31799
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 32430
    port: 443 
    protocol: TCP
    targetPort: https
  selector:
    app: nginx-ingress
    release: nginx-ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: 
    ingress:
    - ip: 1.2.3.4

Anything else we need to know:

The installation uses regular Basic AKS Load balancer.
Just reverting version back to 0.25 brings back fully stable operation, without 403 errors.
We are using VPN to connect to our ingresses - not sure if it makes any difference.

/kind bug

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.