CoreDNS I/O error which suggests that pods / services from within a local cluster cannot reach the cluster’s service network in the range of 10.43.0.0/16.

[jordankrp]

Environmental Info:
K3s Version: v1.31.3+k3s1

Node(s) CPU architecture, OS, and Version: aarch64, Ubuntu 20.04.5 LTS
Linux nvidia-desktop 5.10.104-tegra #1 SMP PREEMPT Sun Mar 19 07:55:28 PDT 2023 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration: single node cluster

Describe the bug:
Installing v1.31.3 of k3s suddenly started making our core-dns pod in kube-system unable to resolve external domains and stuck waiting for Kubernetes API server (10.43.0.1:443). There is an I/O error in the core-dns logs which suggests that pods cannot reach the cluster’s service network in the range of 10.43.0.0/16. This also causes the local-path-provisioner in kube-system to fail to start because it cannot reach the K3S API server, as well as the following error when the helm-install pod tries to reach a public domain: 10.43.0.10:53: read udp 10.42.0.7:49774->10.43.0.10:53: i/o timeout.

Steps To Reproduce:

• Installed K3s: curl -sfL https://get.k3s.io | sh -s - server --resolv-conf /run/systemd/resolve/resolv.conf
• Create /etc/rancher/k3s/registries.yaml with the following content:

mirrors:
  europe-west4-docker.pkg.dev:
    endpoint:
      - "https://europe-west4-docker.pkg.dev"
configs:
  europe-west4-docker.pkg.dev:
    auth:
      username: _json_key
      password: '{  "type": "service_account",  "project_id": "...",  "private_key_id": "...",  "private_key": "-----BEGIN PRIVATE KEY-----...\n-----END PRIVATE KEY-----\n",  "client_email": "...",  "client_id": "...",  "auth_uri": "...",  "token_uri": "...",  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",  "client_x509_cert_url": "...",  "universe_domain": "googleapis.com"}'

Restart k3s, create app namespace and apply chart:
sudo service k3s restart
kubectl create namespace app
kubectl apply -f app.yaml

Expected behavior:

All pods / services are able to reach public domains in the range 10.43.0.0/16.

Actual behavior:

There is an I/O error in the core-dns logs which suggests that pods / services cannot reach the cluster’s service network in the range of 10.43.0.0/16.

Additional context / logs:

[brandond]

I have no idea what you're getting at with your app or namespace. You didn't include the app manifest so I don't know what you're doing there.

That said, it sounds like something is blocking traffic within the cluster. If there are no errors in the k3s service logs, ensure that you've disabled ufw, firewalld, or any other local endpoint security agents.