Services with externalTrafficPolicy: Local seem not to work

[dkalintsev]

If I create a NodePort Service with externalTrafficPolicy: Local, I can't connect to it on NODE_IP:NodePort. If the Service is patched to set externalTrafficPolicy to Cluster, connectivity works.

Using a QuickStart deployed with all defaults (CoreDNS / Calico).

Example Deployment + Service that doesn't work unless I remove externalTrafficPolicy:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
          name: http
        #
        # There isn't anything listeing on 443; included just as an example
        #
        - containerPort: 443
          name: https
        #
        # This will change the page nginx serves to show container's hostname
        #
        command: ["/bin/bash"]
        args: ["-c", "printenv HOSTNAME > /usr/share/nginx/html/index.html && nginx -g \"daemon off;\""]
      #
      # This Anti-Affinity rule will prevent K8s from scheduing
      # more than pod on the same node, so that we can provide
      # proper session persistence and pod health checking.
      #
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: run
                operator: In
                values:
                - my-nginx
            topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx-service
  labels:
    run: my-nginx-svc
spec:
  # Prevent this Service from sending traffic to other nodes
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: http
    name: http
  selector:
    run: my-nginx
  type: NodePort

Example session:

kubectl get pods -o wide
NAME                       READY     STATUS    RESTARTS   AGE       IP               NODE
my-nginx-5b44bffd9-mx654   1/1       Running   0          14m       192.168.89.195   ip-10-0-0-168.ap-southeast-2.compute.internal

kubectl get svc -o wide
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE       SELECTOR
kubernetes         ClusterIP   10.96.0.1       <none>        443/TCP        1h        <none>
my-nginx-service   NodePort    10.111.158.65   <none>        80:32459/TCP   14m       run=my-nginx

From Bastion host, after adjusting the SG applied to nodes to allow appropriate connectivity:

ubuntu@ip-10-0-128-5:~$ curl http://10.0.0.168:32459
^C

kubectl patch svc my-nginx-service -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
service "my-nginx-service" patched

ubuntu@ip-10-0-128-5:~$ curl http://10.0.0.168:32459
my-nginx-5b44bffd9-mx654

kubectl patch svc my-nginx-service -p '{"spec":{"externalTrafficPolicy":"Local"}}'
service "my-nginx-service" patched

ubuntu@ip-10-0-128-5:~$ curl http://10.0.0.168:32459
^C

[dkalintsev]

Issue looks similar to what I see here: https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/489

I get very much the same symptoms. In iptables:

[0:0] -A KUBE-XLB-Y5R64WIC3CQGH3LS -m comment --comment "default/my-nginx-service:http has no local endpoints" -j KUBE-MARK-DROP

In kube-proxy's logs:

W0507 22:33:10.372369       1 server.go:586] Failed to retrieve node info: nodes "ip-10-0-23-78" not found
W0507 22:33:10.372474       1 proxier.go:463] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP

[timothysc]

/assign @detiber

[detiber]

I'm currently testing a fix for this in #190, which uses an init container to update overrideHostname in the kube-proxy config.

[detiber]

I just finished testing #179 with the reproducer provided and I am now able to access the NodePort with externalTrafficPolicy: local as expected.

[bamb00]

@detiber I have the same issue running kubernetes v1.9.5. Is there a workaround? Thanks.

[detiber]

@bamb00 the workaround is documented here: https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#services-with-externaltrafficpolicy-local-are-not-reachable

It may need to be modified to work with v1.9.5, though.

[Zanis7]

@detiber I'm facing the same issue in kubernetes v1.18.1. Workaround you mentioned is not available in the above link . Can you help me to solve the issue ?

[detiber]

@Zanis7 I'm not sure this project is being maintained anymore, but if you are hitting this issue, then you likely need to make sure that the Node resource, the Kubelet configuration, and the kube-proxy configuration all agree on what the local hostname of the local machine is, otherwise traffic is likely not being handled correctly.

[Zanis7]

Thanks for your suggestion. I'll verify the same.