Cannot open TCP and UDP ports on same port number

[cskwrd]

Environmental Info:
K3s Version:

k3s version v1.25.4+k3s1 (0dc63334)
go version go1.19.3

Node(s) CPU architecture, OS, and Version:
Linux server1 6.1.4-1-default #1 SMP PREEMPT_DYNAMIC Mon Jan 9 11:00:31 UTC 2023 (4b9b43c) x86_64 x86_64 x86_64 GNU/Linux
Linux agent1 6.1.4-1-default #1 SMP PREEMPT_DYNAMIC Mon Jan 9 11:00:31 UTC 2023 (4b9b43c) x86_64 x86_64 x86_64 GNU/Linux
Linux agent2 6.1.0-1-default #1 SMP PREEMPT_DYNAMIC Wed Dec 21 16:29:21 UTC 2022 (ab10a11) x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server, 2 agents

Describe the bug:
Trying to open a UDP port and TCP on the same port number doesn't work. It appears that only the TCP port is actually open.

Steps To Reproduce:

• Installed K3s: Followed the quickstart guide, https://docs.k3s.io/quick-start
• After install SFTP following as /var/lib/rancher/k3s/server/manifests/traefik-customizations.yaml

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    logs:
      general:
        level: INFO
      access:
        enabled: true
    ports:
      random-tcp:
        port: 8999
        expose: true
        exposedPort: 8999
        protocol: TCP
      random-udp:
        port: 8999
        expose: true
        exposedPort: 8999
        protocol: UDP

• Finally kubectl apply -f ./port-test.yaml:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    owner-email: tcp@example.com
  labels:
    istio-injection: enabled
  name: whoami-tcp
---
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    owner-email: udp@example.com
  labels:
    istio-injection: enabled
  name: whoami-udp
---
apiVersion: v1
kind: Service
metadata:
  name: svc-whoami-tcp
  namespace: whoami-tcp
spec:
  ports:
  - port: 8080
  selector:
    app: app-whoami-tcp
    version: v1
---
apiVersion: v1
kind: Service
metadata:
  name: svc-whoami-udp
  namespace: whoami-udp
spec:
  ports:
  - port: 8080
  selector:
    app: app-whoami-udp
    version: v1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-whoami-tcp
    version: v1
  name: whoami-tcp-v1
  namespace: whoami-tcp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-whoami-tcp
      version: v1
  template:
    metadata:
      labels:
        app: app-whoami-tcp
        version: v1
    spec:
      containers:
      - image: traefik/whoamitcp:latest
        imagePullPolicy: IfNotPresent
        name: whoamitcp
        ports:
        - containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-whoami-udp
    version: v1
  name: whoami-udp-v1
  namespace: whoami-udp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-whoami-udp
      version: v1
  template:
    metadata:
      labels:
        app: app-whoami-udp
        version: v1
    spec:
      containers:
      - image: traefik/whoamiudp:latest
        imagePullPolicy: IfNotPresent
        name: whoamiudp
        ports:
        - containerPort: 8080
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: ingressroute-tcp-whoami
  namespace: whoami-tcp
spec:
  entryPoints:
  - random-tcp
  routes:
  - match: HostSNI(`*`)
    services:
    - name: svc-whoami-tcp
      port: 8080
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteUDP
metadata:
  name: ingressroute-udp-whoami
  namespace: whoami-udp
spec:
  entryPoints:
  - random-udp
  routes:
  - services:
    - name: svc-whoami-udp
      port: 8080

Expected behavior:
Both the TCP and UDP pods should be reachable at server1:8999

Actual behavior:
Only the TCP port works.

Additional context / logs:
kubectl commands used remotely from Windows 10.

[brandond]

There is a bug in Kubernetes that makes it difficult to patch a Service to add a port on a new protocol, if that same port is already present in the Service. This is alluded to in the Traefik helm chart itself: https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml#L592-L594

   ## There are known limitations when trying to listen on same ports for
   ## TCP & UDP (Http3). There is a workaround in this chart using dual Service.
   ## https://github.com/kubernetes/kubernetes/issues/47249#issuecomment-587960741

The fundamental problem is patchMergeKey for service is "port" whereas the uniqueness of a port comes from "port+protocol" (Ex. you cannot have duplicate port with same port number and protocol). Similar issue also happens for "ContainerPort".

As a result of this bug, you'll need to delete and re-create the service if you want to add a UDP port that is already exposed as TCP.

[cskwrd]

When performing the following:

1. kubectl delete svc traefik -n kube-system
2. Upload config changes via SFTP.
3. Wait a moment for changes to go into effect.
4. kubectl get svc -o wide -n kube-system

Both ports now appear in the output for the traefik service.

However, I still can't communicate over UDP. It seems like the packet gets lost "inside" k3s somewhere.

[brandond]

How are you troubleshooting? Have you confirmed that the packet is not making it to traefik, or are packets not successfully going from traefik to the backend service?

[cskwrd]

How are you troubleshooting?

The pods outlined in the yaml file above are UDP and TCP echo servers. I have a client locally that can send and receive successfully when I separate the TCP and UDP port definitions. (8998/TCP and 8999/UDP).
From there I am using wireshark locally, and tcpdump inside of ephemeral containers injected into pods and tcpdump on the server host, and trying to trace packets through k3s.

If there is a better way to investigate or provide info let me know. I am new to kubernetes, at the moment.

Have you confirmed that the packet is not making it to traefik, or are packets not successfully going from traefik to the backend service?

As I mentioned above, When I configure the proxy to use different ports for TCP and UDP, my test clients can connect, send and receive successfully. When I try to use the same port number for both protocols, my TCP client still works but my UDP client times out.

[brandond]

RIght so where are you seeing the packets disappear in your tcpdump when things aren't working. Do they make it in to the Traefik pod? Do they make it from Traefik to the echo server pod? Or are you trying to connect directly to the service, and not through Traefik? I'm a bit confused as to what specifically you're testing and where things are getting lost.

[cskwrd]

tcpdump from server host:

# tcpdump -i any port 8999
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
16:45:43.889970 enp1s0 In  IP MY-PC.lan.55555 > server.lan.bctp: UDP, length 3
16:45:43.890087 cni0  Out IP MY-PC.lan.55555 > 10.42.0.10.bctp: UDP, length 3
16:45:43.890103 veth102b69ca Out IP MY-PC.lan.55555 > 10.42.0.10.bctp: UDP, length 3
16:45:43.890166 veth102b69ca P   IP 10.42.0.10.55555 > 10.43.31.86.bctp: UDP, length 3
16:45:43.890190 cni0  In  IP 10.42.0.10.55555 > 10.43.31.86.bctp: UDP, length 3
16:45:43.890224 enp1s0 Out IP server.lan.14124 > 10.43.31.86.bctp: UDP, length 3

from pod at 10.42.0.10:

$ tcpdump -i any not port 9000
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
21:45:43.890120 eth0  In  IP MY-PC.lan.55555 > svclb-traefik-d939bb39-55rv6.8999: UDP, length 3
21:45:43.890161 eth0  Out IP svclb-traefik-d939bb39-55rv6.55555 > traefik.kube-system.svc.cluster.local.8999: UDP, length 3

I don't see any matching packets in the traefik pod. There is one exception on a prior test, I got this output but haven't been able to recreate it:

$ tcpdump -i any not port 9000
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
20:34:28.707107 eth0  P   IP MY-PC.lan.55555 > 10.42.0.7.8999: UDP, length 3
20:34:28.746970 eth0  Out IP traefik-5c5f5d996-sjkg5.56800 > kube-dns.kube-system.svc.cluster.local.53: 15130+ PTR? 7.0.42.10.in-addr.arpa. (40)
20:34:28.749924 eth0  In  IP kube-dns.kube-system.svc.cluster.local.53 > traefik-5c5f5d996-sjkg5.56800: 15130 NXDomain* 0/0/0 (40)
20:34:28.750482 eth0  Out IP traefik-5c5f5d996-sjkg5.46057 > kube-dns.kube-system.svc.cluster.local.53: 17689+ PTR? 70.21.168.192.in-addr.arpa. (44)
20:34:28.753134 eth0  In  IP kube-dns.kube-system.svc.cluster.local.53 > traefik-5c5f5d996-sjkg5.46057: 17689* 1/0/0 PTR MY-PC.lan. (96)

TCP packets make it all the way to the TCP echo server. UDP packets don't appear to make it to the traefik pod, generally. I am trying to connect via the server node's IP address for TCP and UDP (so through traefik).

[brandond]

I have successfully done this for HTTP3 support in #5184 (comment) so I know it does work.

Have you checked both the traefik service and the traefik pod to confirm that the ports are defined there? I can't remember if the traefik chart automatically adds an entrypoint flag to the args, or if you need to do that yourself still?

[cskwrd]

What commands should I use to verify the ports are defined?

[brandond]

just use kubectl get -o yaml to get the traefik service and deployment, and confirm that the ports are listed correctly in both?

[cskwrd]

Ah, gotcha.

kubectl get -o yaml -n kube-system svc traefik indicated the correct ports were open and mapped.

kubectl get -o yaml -n kube-system deployment traefik indicated that the UDP port was missing. I deleted the deployment (kubectl delete deployment traefik -n kube-system) and triggered a reapplication of the traefik manifest. I now see the ports on the deployment as well. UDP connections are now successful!

I guess the deployment is affected by the helm chart bug mentioned in your comment as well.

Thanks for your help getting me sorted out!

[tataue]

Ah, gotcha.

kubectl get -o yaml -n kube-system svc traefik indicated the correct ports were open and mapped.

kubectl get -o yaml -n kube-system deployment traefik indicated that the UDP port was missing. I deleted the deployment (kubectl delete deployment traefik -n kube-system) and triggered a reapplication of the traefik manifest. I now see the ports on the deployment as well. UDP connections are now successful!

I guess the deployment is affected by the helm chart bug mentioned in your comment as well.

Thanks for your help getting me sorted out!

same here after reinstall by the helm, websecure-http3 on container comed out.