ServiceLB does not get traffic to pod on external ip node

[a-camacho]

Environmental Info:
K3s Version:
k3s version v1.25.7+k3s1 (f7c20e2)
go version go1.19.6

Node(s) CPU architecture, OS, and Version:
Linux k3s-master 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Cluster Configuration:
Proxmox Hosts
1 server, 2 agents (one of those agents is at a remote location, external-ip set up)

Describe the bug:
I have a timeout when I have a service of type LoadBalancer that exposes a hello world pod, deploys it on a remote node (not same local network, but external ip declared).

Steps To Reproduce:

• Installed K3s server with external-ip declared
• Installed K3s agent on node1
• Installed K3s agent on remote-node1 with external-ip declared (and external ip of server)

Expected and actual behavior:

When declaring external ip on a node, it seemed to me that ServiceLB would use external-ip for getting traffic to the desired pod. I have the pod running on the node on the remote location, but get a timeout when ping the NodePort on master ip.

If the pod is running on one of the two other nodes (same local network) it works, but not on the remote one.

Additional context / logs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-k8s-app-001
  namespace: hello-kubernetes
  labels:
    app.kubernetes.io/instance: hello-k8s-app-001
    app.kubernetes.io/name: hello-k8s-app-001
spec:
  selector:
    matchLabels:
      app: hello-k8s-app-001
  replicas: 3
  template:
    metadata:
      labels:
        app: hello-k8s-app-001
    spec:
      containers:
      - name: hello-kubernetes
        image: paulbouwer/hello-kubernetes:1.10
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        env:
        - name: KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: KUBERNETES_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

---

apiVersion: v1
kind: Service
metadata:
  name: hello-k8s-app-001-svc
  namespace: hello-kubernetes
spec:
  selector:
    app: hello-k8s-app-001
  type: LoadBalancer
  ports:
  - name: http
    #nodePort: 32710
    port: 8080
    protocol: TCP
    targetPort: http

Thanks a lot for any help.
I think I am misunderstanding something network related.

Could not find any logs or information in journal of agent or server. Or traefik / coredns pods.

[brandond]

I have the pod running on the node on the remote location, but get a timeout when ping the NodePort on master ip. If the pod is running on one of the two other nodes (same local network) it works

It sounds like pod to pod or service to pod traffic is not working between the local and remote nodes. What flannel backend are you using? Note the default vxlan is only meant for use across secure local networks; it is unencrypted, unauthenticated, and will probably not work across the internet.

[acamacho-unige]

Thanks for your reply, the sentence I was missing and did not get from documentation was : Note the default vxlan is only meant for use across secure local networks; it is unencrypted, unauthenticated, and will probably not work across the internet.

What would you suggest if I wanted to join to the cluster a node that is only accessible by its external_ip ? What I mean is that I have no network (VPN) setup making this remote node available on local network.

Is there any option that does not include creating a VPN network in which all my nodes can talk to each other ?
Should I use the WireGuard Flannel backend () or does this only take care about encryption of communication between nodes and does not allow a remote node to join a local network ?

Thanks again (a lot) for your reply and for any input, kinda lost.

[brandond]

You might check out the suggestions at https://docs.k3s.io/installation/network-options#distributed-hybrid-or-multicloud-cluster

[caroline-suse-rancher]

Please feel free to continue this as a discussion :) At this time there's not a clear bug.