Kubeadm coredns image path wrong when use local docker registry

[mabombo]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version): 1.21.2-0

Environment:

• Kubernetes version (use kubectl version): 1.21.2-0
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): centos 7
• Kernel (e.g. uname -a):Linux centos01 3.10.0-1160.31.1.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Thu Jun 10 13:32:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

When you get the list of images to pull using default repo end custom repo for coredns image the result is different.
For the custom repo the image path it's wrong

Default repo
k8s.gcr.io/coredns/coredns:v1.8.0
Custom repo
x.x.x.x:8082/coredns:v1.8.0

What you expected to happen?

The path of coredns it should be the same but at the moment it's different. In the second case (using custom repo) it's wrong.
It's not possible pull the image.

How to reproduce it (as minimally and precisely as possible)?

$> kubeadm -v 10 config images list

k8s.gcr.io/kube-apiserver:v1.21.2
k8s.gcr.io/kube-controller-manager:v1.21.2
k8s.gcr.io/kube-scheduler:v1.21.2
k8s.gcr.io/kube-proxy:v1.21.2
k8s.gcr.io/pause:3.4.1
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns/coredns:v1.8.0

When you get the list of images to pull using custom docker registry

kubeadm -v 10 config images list --image-repository=x.x.x.x:8082

x.x.x.x:8082/kube-apiserver:v1.21.2
x.x.x.x:8082/kube-controller-manager:v1.21.2
x.x.x.x:8082/kube-scheduler:v1.21.2
x.x.x.x:8082/kube-proxy:v1.21.2
x.x.x.x:8082/pause:3.4.1
x.x.x.x:8082/etcd:3.4.13-0
x.x.x.x:8082/coredns:v1.8.0

Anything else we need to know?

This problem it is not presente in the preview version of kubeadm (1.20.8).
In this case the results of the commands are the same.

kubeadm -v 10 config images list

k8s.gcr.io/kube-apiserver:v1.20.8
k8s.gcr.io/kube-controller-manager:v1.20.8
k8s.gcr.io/kube-scheduler:v1.20.8
k8s.gcr.io/kube-proxy:v1.20.8
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns:1.7.0

kubeadm -v 10 config images list --image-repository=x.x.x.x:8082

x.x.x.x:8082/kube-apiserver:v1.20.8
x.x.x.x:8082/kube-controller-manager:v1.20.8
x.x.x.x:8082/kube-scheduler:v1.20.8
x.x.x.x:8082/kube-proxy:v1.20.8
x.x.x.x:8082/pause:3.2
x.x.x.x:8082/etcd:3.4.13-0
x.x.x.x:8082/coredns:1.7.0

[neolit123]

/kind support

It used to be customrepo/coredns/coredns but we changed that back to customrepo/coredns after too many complaints that we do not have to preserve the gcr.k8s.io paths for custom registries.

To customise it you can use the kubeadm config file - clusterconfiguration.dns.imageregistry.

Also please always read the release notes before upgrading kubeadm.

/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

/kind support

It used to be customrepo/coredns/coredns but we changed that back to customrepo/coredns after too many complaints that we do not have to preserve the gcr.k8s.io paths for custom registries.

To customise it you can use the kubeadm config file - clusterconfiguration.dns.imageregistry.

Also please always read the release notes before upgrading kubeadm.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[smcquay]

@neolit123 I'm still seeing this issue. Do you have a workaround?

[neolit123]

we added docs for that here:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init#without-internet-connection
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init#custom-images

summary:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#preparing-the-required-container-images

[Raptus1]

/kind support

It used to be customrepo/coredns/coredns but we changed that back to customrepo/coredns after too many complaints that we do not have to preserve the gcr.k8s.io paths for custom registries.

To customise it you can use the kubeadm config file - clusterconfiguration.dns.imageregistry.

Also please always read the release notes before upgrading kubeadm.

/close

But now that coredns have mixed around and moved on k8s.gcr.io even more, the path for custom registries is not longer preserved (again).

Can this be restored so that the path is preserved correctly for the custom registry?
IE: customrepo/coredns/coredns

[neolit123]

Once the coredns team made the change, the custom registry in kubeadm had a sub path, but users still complained that the subpath is not desired. So yes the discrepancy is anoying but we based the change on complains. Please read https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#preparing-the-required-container-images And the linked pages in there. Once you obtain the images from gcr.io you should not just string replace s/ gcr.io/myreg/ for the push locations.