Upgrade complains of not migratable CoreDns plugin grpc

[Jeansen]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version): 1.25.3

Environment:

• Kubernetes version (use kubectl version): 1.25.3
• Cloud provider or hardware configuration: KVM
• OS (e.g. from /etc/os-release): Debian 11
• Kernel (e.g. uname -a): Linux master0.k8s 5.10.0-20-amd64 kubeadm join on slave node fails preflight checks #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
• Container runtime (CRI) (e.g. containerd, cri-o): cri-o
• Container networking plugin (CNI) (e.g. Calico, Cilium): Calico
• Others:

What happened?

Creating an upgrade plan with kubeadm says:
[WARNING CoreDNSUnsupportedPlugins]: CoreDNS cannot migrate the following plugins:
[Plugin "grpc" is unsupported by this migration tool in 1.9.3.]

What you expected to happen?

Flawless migration plan to be executable.

How to reproduce it (as minimally and precisely as possible)?

Create a cluster running kubernetes 1.25.3 (or earlier).
Add grpc to CoreDns config.
My current config looks like this:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . 192.168.178.45:53 {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
    grpc . 192.168.178.45:53 
}

Connect to the master node, update packages from e.g. APT and create an upgrade plan with kubeadm.

apt-get update
apt upgrade
kubeadem upgrade plan

Anything else we need to know?

Here's more what I get during the upgrade.

1. Try to get an upgrade plan:

sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[preflight] The corefile contains plugins that kubeadm/CoreDNS does not know how to migrate. Each plugin listed should be manually verified for compatibility with the newer version of CoreDNS. Once ready, the upgrade can be initiated by skipping the preflight check. During the upgrade, kubeadm will migrate the configuration while leaving the listed plugin configs untouched, but cannot guarantee that they will work with the newer version of CoreDNS.
[preflight] Some fatal errors occurred:
        [ERROR CoreDNSUnsupportedPlugins]: CoreDNS cannot migrate the following plugins:
[Plugin "grpc" is unsupported by this migration tool in 1.9.3.]
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

2. Get an upgrade plan again, ignoring the error above:

sudo kubeadm upgrade plan --ignore-preflight-errors=CoreDNSUnsupportedPlugins 
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[preflight] The corefile contains plugins that kubeadm/CoreDNS does not know how to migrate. Each plugin listed should be manually verified for compatibility with the newer version of CoreDNS. Once ready, the upgrade can be initiated by skipping the preflight check. During the upgrade, kubeadm will migrate the configuration while leaving the listed plugin configs untouched, but cannot guarantee that they will work with the newer version of CoreDNS.
        [WARNING CoreDNSUnsupportedPlugins]: CoreDNS cannot migrate the following plugins:
[Plugin "grpc" is unsupported by this migration tool in 1.9.3.]
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.25.3
[upgrade/versions] kubeadm version: v1.25.3
I0223 16:19:15.837958   18904 version.go:256] remote version is much newer: v1.26.1; falling back to: stable-1.25
[upgrade/versions] Target version: v1.25.6
[upgrade/versions] Latest version in the v1.25 series: v1.25.6

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     5 x v1.25.3   v1.25.6

Upgrade to the latest version in the v1.25 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.25.3   v1.25.6
kube-controller-manager   v1.25.3   v1.25.6
kube-scheduler            v1.25.3   v1.25.6
kube-proxy                v1.25.3   v1.25.6
CoreDNS                   v1.9.3    v1.9.3
etcd                      3.5.4-0   3.5.4-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.25.6

Note: Before you can perform this upgrade, you have to update kubeadm to v1.25.6.

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

[neolit123]

[Plugin "grpc" is unsupported by this migration tool in 1.9.3.]

this is coredns specific and we cannot do much on the kubeadm side.
you could temporary remove the grpc config and then re-adding it back if this is a migration tool only issue - i.e. coredns next actually still supports the plugin.

cc @chrisohaver @rajansandeep

[Jeansen]

Oh, I see. Should I close this one then and reopen it somewhere else?

[neolit123]

the migration tool is here:
https://github.com/coredns/corefile-migration

but let's wait for feedback first.

[chrisohaver]

Thanks. The migration tool only supports plugins used in the default k8s deployment and a few common plugins. Grpc is not one of them.

[chrisohaver]

You should not need to remove the plugin and add it back. Unsupported plugins are left in the corefile unmodified. The warning is there just to let you know that the migration tool is not actively migrating grpc in case there was a backward incompatible change to the unsupported plugin’s configuration (eg deprecated options). You can probably ignore the warning.

[neolit123]

thanks @chrisohaver

closing given the warning can be just ignored.

[chrisohaver]

COMPONENT CURRENT TARGET
...
CoreDNS v1.9.3 v1.9.3

It is odd though that kubeadm raises migration warnings when the target version is the same as the current version.
I'd think in this case, no migration would be performed since the version is not changing, and thus no plugin migration supportability checks are necessary.

[neolit123]

i think kubeadm would ask the migration lib if migration is possible from ver x to y (or in this case from x to x). perhaps migration warnings should be ignored if coredns versions are the same, or the migration lib should not return the warnings.

[chrisohaver]

Yes, the migration lib should check to see if it has been called with a nonsense migration.

But it's still odd that kubeadm would bother asking the migration lib anything, if it knows it is not upgrading CoreDNS.

[pacoxu]

It sounds like a small bug that we can fix. I will try to fix it. Or migration from same versions can skip. I need take a deep look.

[Jeansen]

Yes, the migration lib should check to see if it has been called with a nonsense migration.

But it's still odd that kubeadm would bother asking the migration lib anything, if it knows it is not upgrading CoreDNS.

Yes, that was my thinking, too. I add --ignore-preflight-errors=CoreDNSUnsupportedPlugins and ignored it. But still, it's on the one hand misleading and on the other hand a "false positive", anyway.

[Jeansen]

It sounds like a small bug that we can fix. I will try to fix it. Or migration from same versions can skip. I need take a deep look.

Let me know, if there is anything I can test. It's easy for me to revert back and forth since my cluster is lokal and fully virtualized.

[chrisohaver]

FYI, fix here in migration lib: coredns/corefile-migration#75

[Jeansen]

@chrisohaver Awesome! I like things done that swiftly! ;-)

[chrisohaver]

Ive made the fix in the lib, and adjusted tests. There's a very slim possibility that someone may be using the Unsupported/Deprecated lib funcs to check single versions for supportability. If so, this change will break that usage. However, I think in the context of migrations, a single version check is dubious anyways. So the change is probably safe.

[chrisohaver]

Released: https://github.com/coredns/corefile-migration/releases/tag/v1.0.20

[pacoxu]

kubernetes/kubernetes#116060

• commit1: upgrade https://github.com/coredns/corefile-migration/releases/tag/v1.0.20
  Minor change: Don't return Unsupported/Deprecated plugins for non-migration cases, i.e. when the start version is the same as the end version.
• commit2: try to fix the problem in kubeadm side as well