RKE2 server breaks nginx

[koxu1996]

Environmental Info

• RKE2 Version: v1.27.12+rke2r1 (25b27b4e4709a2ac4c550609ad730a9e172d110a)

Node - medium size VPS running in Linode:

• OS: Arch Linux
• Architecture: x64
• Kernel version: 6.8.7

No firewalld, no network managers.
Nginx server installed and listening on ports 80/443.

Cluster Configuration:

1 server with WireGuard backend:

# /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    flannel:
      backend: "wireguard"

Problem Description

After installing RKE2 server, my website served with nginx suddenly stopped working:

I see that network traffic is hijacked by Kuberenetes ingress:

Interestingly, there is no other process listening on port 443:

$ sudo ss -tulpn | grep 443
tcp   LISTEN 0      511            0.0.0.0:443        0.0.0.0:*    users:(("nginx",pid=1153231,fd=7),("nginx",pid=1130375,fd=7))
tcp   LISTEN 0      4096                 *:6443             *:*    users:(("kube-apiserver",pid=949,fd=3))

I guess RKE2 or CNI uses iptables to hijack the whole traffic. This is unexpected behavior and I am looking for fix/workaround.

[brandond]

The ingress-nginx that comes with rke2 binds to host ports 80 and 443. You can start rke2 with --disable=rke2-ingress-nginx, or customize the HelmChartConfig to use different ports.

Why are you trying to run nginx on the same host as Kubernetes? Normally if you want to host something on your cluster nodes, you'd deploy it to the cluster, instead of running it unmanaged along side it.

[koxu1996]

Whoa, that was quick! I missed the fact that ingress is provided by default, so I deployed Traefik 😅. I also used nodePorts 32080 and 30443 because I thought there was no easy way to run a LoadBalancer, but now I see there's the --serviceLB option. This is dope!

Why are you trying to run nginx on the same host as Kubernetes?

I was already running nginx on a VPS to serve my website and just wanted to gradually migrate the entire stack to Kubernetes with RKE2.