K8s CoreDns not starting on 32.20200629.3.0 (Pods on master can't talk to the apiserver)

[teintuc]

Hi all,

I have a production kubernetes (v1.17.4) cluster running on fedora coreos 32.20200615.3.0 and all is fine.

When I try the last release 32.20200629.3.0, on my test cluster, some core service (like coredns) can't contact the apiserver because the service address (10.3.0.1) is timing out.

I rollbacked the nodes to 32.20200615.3.0 and it is all good again.
Which makes me think it comes from something new in the last version.

Is there something that changed between those version that could cause the problem?
Where can I go check in order to fix my problem?

Here the result of my rpm-ostree status:

State: idle
Deployments:
  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200629.3.0 (2020-07-10T17:58:03Z)
                    Commit: 6df95bdb2fe2d36e091d4d18e3844fa84ce4b80ea3bd0947db5d7a286ff41890
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200629.3.0 (2020-07-10T17:58:03Z)
                    Commit: 6df95bdb2fe2d36e091d4d18e3844fa84ce4b80ea3bd0947db5d7a286ff41890
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

Thanks

[lucab]

@teintuc thanks for the report. You can see the whole list of package differences on the website, 46 packages were upgraded in this release.
To the best of my knowledge, we didn't receive any related breakage report while this set of packages was soaking in testing/next.

[cgwalters]

Of those changes it looks to me like by far the most likely to be involved are iptables and kernel, and I'd put more money on the latter. Can you try using e.g. rpm-ostree override replace to downgrade the kernel on a node starting from the newer version, to bisect this?

[teintuc]

I was thinking about iptables but it seems ok.
Haven't thought about the kernel. I'll try to downgrade it (once I found how to do it :) )

Thanks

[dustymabe]

@teintuc - can you easily recreate these nodes? If so let's do our experiment on one that is disposable. You'll want to grabe the rpms from the kernel from 32.20200615.3.0 (looks like those are here) and then download them to the node that is showing the problem (on 32.20200629.3.0) and then:

rpm-ostree override replace ./kernel-5.6.18-300.fc32.x86_64.rpm ./kernel-core-5.6.18-300.fc32.x86_64.rpm ./kernel-modules-5.6.18-300.fc32.x86_64.rpm --reboot

I'd recommend doing this on a disposable node because you may end up losing your 32.20200615.3.0 deployment when performing these operations.

[teintuc]

thanks for the tips.

Yes, I can do this on disposable node :)
I'm already running nodes on 32.20200629.3.0.

[teintuc]

Ok I followed your instructions:
I got the following error:

 Problem: conflicting requests
  - nothing provides kernel-modules-uname-r = 5.6.18-300.fc32.x86_64 needed by kernel-5.6.18-300.fc32.x86_64

So after looking in the kernel repo you gave me, I added this package kernel-modules-5.6.18-300.fc32.x86_64.rpm.
I get this error:
error: Loading pkgcache branch rpmostree/pkg/kernel/5.6.18-300.fc32.x86__64: Failed to find metadata key rpmostree.sepolicy (signature s)

I'm looking to make it work :)

[cgwalters]

Dusty meant ./kernel-modules-5.6.18-300.fc32.x86_64.rpm and not kernel-debug-modules.

error: Loading pkgcache branch rpmostree/pkg/kernel/5.6.18-300.fc32.x86__64: Failed to find metadata key rpmostree.sepolicy (signature s)

Hmm...did you disable SELinux? That...isn't very well tested in rpm-ostree today.

[teintuc]

I tried without the debug module. I also, as you said, had to enable SELinux to make it worked.
Once all my nodes rebooted:

$ uname -a
Linux ip-10-102-145-247 5.6.18-300.fc32.x86_64 #1 SMP Wed Jun 10 21:38:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/os-release
NAME=Fedora
VERSION="32.20200629.3.0 (CoreOS)"
ID=fedora
VERSION_ID=32
VERSION_CODENAME=""
PLATFORM_ID="platform:f32"
PRETTY_NAME="Fedora CoreOS 32.20200629.3.0"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:32"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=32
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=32
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='32.20200629.3.0'

I still get the same error on my kubernetes cluster: timeout on 10.3.0.0.1

I start to think my problems comes from iptables. Since it has been updated in the last release, the coincidence is too big.
I could do the same test by downgrading iptables packages.

[dustymabe]

Dusty meant ./kernel-modules-5.6.18-300.fc32.x86_64.rpm and not kernel-debug-modules.

oops.. copy/pasta.. fixed in my original comment now.

[dustymabe]

hey @teintuc - did you take a closer look at iptables or find out another reason for the regression?

[teintuc]

Hi, thanks for asking :)

I didn't had much time to have an other look at it.
I found some kubernetes related logs in my iptables in the KUBE-SERVICES chain like :

    0     0 REJECT     udp  --  any    any     anywhere             ip-10-3-0-10.ec2.internal  /* kube-system/kube-dns:dns has no endpoints */ udp dpt:domain reject-with icmp-port-unreachable

But don't know what is causing it. I'll keep digging.

[dustymabe]

Are you using docker/containerd or cri-o for the container runtime for kube? We have noticed issues in the past where running multiple container runtimes at the same time can cause issues with the firewall.

[teintuc]

We are using docker.

But there is something interesting: I pop some kubernetes workers, not only masters. Seems that pods hosted on master get the problem. In my case Coredns

[DirkTheDaring]

@teintuc are you using by any chance flannel? The same issue showed up in my environment
#583

[teintuc]

@DirkTheDaring We are not using flannel. We use the AWS-VPC-CNI.
But your problem looks a lot like mine. I can't reach coredns to resolve kubernetes.default (it goes to timeout).

[DirkTheDaring]

Then this goes back to the original questions of my engineers. What did CoreOS Change on the bottom level of the network? It seems like calico, which uses tunnels is not affected. Flannel uses VXLAN, and it seems that the packages only sometimes reach the DNS server (old CoreOS version begin of June) or it doesn't work at all (latest end of June version of CoreOS). We can note that on the baremetal image i am using the network device naming has changed from eth0 to ens18. But even a careful comparison of all firewall rules did not show any obvious difference. Whatever it is it happens somewhere the packages are encapsulated for the network transfer (flannel) and then either the transfer is rejected or let go, by some unknown mechanism.

So fare my understanding...

[teintuc]

Thanks for you analysis. I don't have much more to say.
Still have some more digging to do (And I will be on vacation soon).

[DirkTheDaring]

I have one request before you leave, would you please so kind to change the title your report so that it does reflect more the gist of what we are discussing here? This might help to attract other people with the same issue.

[teintuc]

@DirkTheDaring Done

[dghubble]

Let's be precise. The problem appears in 32.20200629.3.0 and is observed as a lack of pod-to-pod connectivity across node boundaries, when using the flannel CNI provider.

• Pod to pod traffic for pods on the same node is ok
• Calico (IPIP or vxlan) and Cilium (vxlan) are not affected
• Pod outgoing masquerade traffic is ok (remember internal DNS is a cluster service connection)
• Connectivity problems consistent, not random (double check your pods aren't on the same node, and pin host OS)

We can monitor traffic to identify where something may go wrong. With Pod A and Pod B on different nodes, Pod A pinging Pod B can be observed on Pod B's host flannel.1 interface, but does not reach the cni0, despite the route table reporting that's the interface to route to next. This shows traffic traverses the vxlan tunnel fine, matching other vxlan-based providers not being affected. But does not make it to Pod B will never see the ping request. From Pod B's host, Pod B can be reached.

On the prior FCOS version, you can monitor pod-to-pod connections in the same way and see ICMP packets reach flannel.1 and cni and Pod B replies of course.

OS Experiments

The problem can be observed in both new 32.20200629.3.0 nodes (new NIC names) and in nodes that upgrade to 32.20200629.3.0 or beyond (legacy NIC names), suggesting the NIC name migration is not a factor.

1. Starting with a good 32.20200615.3.0 node, incrementally overriding RPM packages to those used in 32.20200629.3.0 and rebooting, changing iptables, moby-engine, or kernel packages did not cause the node to exhibit the problem behavior. Further experiments start to run into conflicts.
2. Starting with a bad 32.20200629.3.0 node, rolling back the kernel RPMs did not convert it into a working node.
3. Its possible the change lies in OS configurations rather than specific RPM packages, but I don't see a good way to diff those. Perhaps we start looking into any intermediate published images (from testing or next btw those times).

I found few device sysctl changes of note and no iptables rules alterations either.

We probably want to stop describing this in terms of troubleshooting connecting the cluster services (kube-apiserver, CoreDNS). That's likely to draw random Google hits. But when pod-to-pod connectivity is broken at a specific point, those problems follow as symptoms that just happen to be the first thing anyone sees.

[dghubble]

FCOS next 32.20200625.1.0 is also affected so that may help reduce the change set.

[dustymabe]

I'd love to dig in and help find the problem. I can even do some custom builds and share them if people have a theory they want to test and don't have a build environment set up.

3. Its possible the change lies in OS configurations rather than specific RPM packages, but I don't see a good way to diff those. Perhaps we start looking into any intermediate published images (from testing or next btw those times).

Yep. That's what I typically do. You'll get the most granularity if you traverse the testing-devel stream. I'd suggest that you identify a node to test against and disable zincati on it. Then bisect over the commits in the stream (use the builds browser to find commit hashes) to find the last good and first bad.

Example:

commit=45ebfc671e478822bac3b1b99b47a5be6683e3fe859649c241f0fc4f1ea326a1 # 32.20200614.20.0
sudo rpm-ostree rebase fedora-compose:$commit --reboot

# check if it's good or bad (bisect)

commit=b6bf1636c8d04d6ced39ca9cd4ee3b7a7759aabb6b416cc877c25f15b2ad2a28 # 32.20200625.20.1
sudo rpm-ostree rebase fedora-compose:$commit --reboot

I used to have this whole process automated for Atomic Host. I should probably try to revive that.

[DirkTheDaring]

@dghubble cilium is also not working in my test environment
Currently I have the following hypothesis:
Calico does kernel space package encapsulation (IPIP), while both other CNIs like flannel (VXLAN) and cilium are doing Userspace package encapsulation.
So I put out the following "testable" claim: All CNIs which are doing user space encapsulation are broken in the latests builds.

And by test just test on every node if nslookup kubernet.default returns success. If it is succesful on all nodes it works (like in any CNI before the broken builds). In the latest builds, the test passes only for calico.
Note on test setup. At least 1 master and 3 nodes. Usually, it passes on nodes where the coredns is running, but it fails then on nodes where "only" nodelocaldns resides. Which begs the questions if it might be mtu related, as this only applies afaik when a real network transfer happens.

[dghubble]

Thanks @dustymabe! In the testing-devel stream, the problem is absent in 32.20200624.20.0 and arises in 32.20200624.20.1 (9650ef3690346e22b1045223699bd7a428323a31642fcc9165dc15410d52c4c1). Rolling back fixes it again. Rolling forward to that commit causes it again.

@DirkTheDaring I have v1.18.6 clusters with Cilium v1.8.1/v1.8.2 and FCOS 32.20200715.3.0 nodes which don't show the traffic issue or application lookup problems. For reference. Perhaps the underlying cause of the issue you see may be different. Also, Cilium status has full-mesh connectivity checks you can perform, so it can do much better than querying CoreDNS or other indirect measures of connectivity.

[DirkTheDaring]

@dghubble sorry, if i am persistent but i had two people here claiming that everything is fine with their cilium and i just could prove it happens also (but it is not triggered very often).
Would you please so kind to ease my mind and run one query on a cluster which is at least up for 3 days? (this was one a 5 days running cluster... and it had 5 entries... so it happens once a day - if you switch to calico, same workload -> no entries)

kubectl -n kube-system logs -lk8s-app=nodelocaldns --max-log-requests 20 |grep plugin/errors
[ERROR] plugin/errors: 2 221.70.168.192.in-addr.arpa. PTR: read tcp 192.168.0.3:51226->192.168.0.3:53: i/o timeout

and by the way looks cool this typhoon 👍

[dghubble]

The OS commits correspond to these changes. FCOS had been preventing systemd-udev's default.link being added.

FCOS testing-devel 32.20200624.20.1 is the first build to add /usr/lib/systemd/network/99-default.link.

[Match]
OriginalName=*

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

A nice flannel report describes flannel's wish to pick its own mac address for flannel.1, but this will conflict with systemd's pick. "This results in all cross node traffic being dropped at layer 2 on the destination node due to incorrect destination vtep mac". This matches the traffic problem observed here and neighbor nodes arp tables do indeed show a differing address, this all checks out.

Solution

So flannel users can add a flannel specific link configuration (kindly mentioned in that thread). Adapted here for Ignition:

variant: fcos
version: 1.0.0
storage:
  files:
    - path: /etc/systemd/network/10-flannel.link
      mode: 0644
      contents:
        inline: |
          [Match]
          OriginalName=flannel*
          [Link]
          MACAddressPolicy=none

PS: If you use Typhoon with flannel (not the default):

module "mycluster" {
  ...
  networking = "flannel"
  controller_snippets = [
    file("./snippets/flannel.yaml"),
  ]
  worker_snippets = [
    file("./snippets/flannel.yaml"),
  ]
}

Followup

Flatcar shipped a flannel.link for the same reason. Its not clear to me the OS should be involved in such CNI-specific needs since users can do this. But I mention it for completeness.

@teintuc could you retitle this?

[dghubble]

@DirkTheDaring I don't run nodelocaldns, but CoreDNS logs (12 days) on Cilium clusters show no error logs and graphs look fine too (else I'd be paged). If that is helpful to you.

[lucab]

@dghubble thanks for the followup! It looks like in Container Linux we had a network unit to mark flannel interfaces as unmanaged: https://github.com/coreos/coreos-overlay/blob/6cbf3556370722f11612a872a7d223e686c672c0/app-admin/flannel-wrapper/files/50-flannel.network. I guess that may have had the same effect of preventing systemd to modify the interface.

[dghubble]

Seems so. I could see it being nice to have builtin (odds of a random person calling their link flannel* seem low), but also maybe setting a bad precendent if future projects come along wanting the same. Either way seems ok to me!

[DirkTheDaring]

@dghubble Thanks for providing a fact that cilium works
@dghubble Thanks also for tracing down the problem. I have setup a fresh cluster with flannel and the fix above. I am happy to report that it works, I cannot reproduce any of the DNS problems.

On a personal note:
It seems that there are now a lot of "cooks" which fiddle around with the network:

• systemd
• NetworkManager
• flannel (or any other CNI)

Is it really necessary that systemd is involved?
The change is also not obvious for anybody, without finding this issue on github.

[dustymabe]

@dghubble +1000 for the great detective work! This is unfortunate fallout from #484.

Will try to discuss this during the meeting today. Please come if you're free.

[dustymabe]

We discussed this in the meeting yesterday.

The two options we discussed to solve this problem are:

• including a config snippet in the OS to handle this
• reaching out to upstreams to include it when they deliver the software

The first is easy, but can be problematic for a few reasons:

• we set a precedent of baking in configs for different projects (i.e. what about other projects similar to flannel that need something equivalent?, do we need to bake in a config for each?
• if the config ever needs to be changed we need to update it. The upstream project itself should be making those decisions.

In the meeting we decided to wait a week and see if more information surfaces. If there are a few upstreams that we can target that will capture most flannel users then it would be nice to make the changes there. Like:

• flannel rpm in Fedora (if package layering)
• flannel config via typhoon
• flannel config via the upstream docs

[DirkTheDaring]

@dustymabe we configure flannel (download and install) with kubespray.

Maybe it makes sense to open a ticket in the kubespray project so that they deploy a fix similar to typhoon.

[dustymabe]

Maybe it makes sense to open a ticket in the kubespray project so that they deploy a fix similar to typhoon.

That would be great at least to have the conversation. In the meeting (as summarized above) we weren't super opposed to the first option, but obviously including it upstream means that the fix it for other platforms as well (not just FCOS). Do you mind opening a ticket?

[DirkTheDaring]

@dustymabe already had opened one.
kubernetes-sigs/kubespray#6451

[dghubble]

Flannel effectively has a host requirement on how the flannel link is brought up, so I think this is best declared via Ignition. For myself (and Typhoon users), going with Ignition inclusion poseidon/typhoon#795. No changes needed in FCOS.

While Flannel is often run as a DaemonSet on container-optimized OSes and its possible to (ab)use the init container for this, Ignition seems the better option in my comparison. Via the daemonset mutates systemd network files after boot, requires privilege, may require networking restart, and its worrying to be mounting systemd files without strict cause.

[cgwalters]

One thing I've floated in the past is that we really want a clean mechanism to say to things NetworkManager-and-below (including systemd) like "don't touch this interface". For a long time NetworkManager even tried to do DHCP on veth pairs created by container runtimes for example.

So rather than matching on name, something like an extended attribute in the kernel network object or so. Then when flannel creates the interface it would atomically add that flag and everything in the OS would know to ignore it.

[spaced]

Disclaimer: i did not read each comment, but:
I had a similiar issue using latest fcos, because package podman comes with a cni plugin, using kubespray installed a calico cni and result was a strange behavior with dns and k8s network policies.
Make sure you have only ONE cni plugin installed (check /etc/cni/net.d).
I removed the podman network with podman network rm podman command.

TLDR: base package comes with a configured cni plugin (for podman) and may cause issues if using other cni plugins without uninstall it.

[dghubble]

I think this issue is should be closed. The problem centered around the link behavior changes following 32.20200629.3.0 and the behavior matches other distros pulling in the systemd defaults, with the solution of specifying flannels link needs.

An issue with a broad title (like "pods can't talk") will start getting unrelated reports.

[mleoca]

I've tried the flannel solution with the unmanaged link flag to eni's created by the aws-k8s-cni plugin but does not seem to be working there or I'm not doing it right.

[teintuc]

Hi all,

Sorry for the late late reply!
My colleague got to the same conclusions and found a solution which works for us.
You need to drop a file into: /etc/systemd/network/
With:

[Match]
OriginalName=eni* ens*
[Link]
MACAddressPolicy=none

We tested it on Fedora CoreOS 32.20200809.3.0

Thanks