CoreDNS image cannot be pulled when use --image-repository #2714

codeachange · 2022-06-26T15:21:57Z

What keywords did you search in kubeadm issues before filing this one?

coredns image

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:20:54Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Kubernetes version (use kubectl version): kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:20:54Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration: exsi virtual machine
OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)
Kernel (e.g. uname -a):Linux master-01 3.10.0-1160.66.1.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Wed May 18 16:02:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Container runtime (CRI) (e.g. containerd, cri-o):containerd
Container networking plugin (CNI) (e.g. Calico, Cilium):
Others:

What happened?

kubeadm config images list output:
...
...
k8s.gcr.io/coredns/coredns:v1.8.6

BUT when kubeadm config images pull --image-repository=harbor.xxx.tech/k8s-gcr-proxy
ERROR:
failed to pull image "harbor.xxx.tech/k8s-gcr-proxy/coredns:v1.8.6"

the image path should be harbor.xxx.tech/k8s-gcr-proxy/coredns/coredns:v1.8.6 instead of harbor.xxx.tech/k8s-gcr-proxy/coredns:v1.8.6

What you expected to happen?

this command kubeadm config images pull --image-repository=harbor.xxx.tech/k8s-gcr-proxy should be able to pull coredns image

How to reproduce it (as minimally and precisely as possible)?

clean machine
install kubeadm, kubelet and kubectl
execute kubeadm config images pull --image-repository=harbor.xxx.tech/k8s-gcr-proxy

Anything else we need to know?

harbor.xxx.tech/k8s-gcr-proxy is a registry proxy using https://github.com/goharbor/harbor

The text was updated successfully, but these errors were encountered:

neolit123 · 2022-06-26T19:04:48Z

Please see https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images The difference in image path is by design. /close /kind support

k8s-ci-robot · 2022-06-26T19:05:04Z

@neolit123: Closing this issue.

In response to this:

Please see
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images

The difference in image path is by design.

/close
/kind support

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wnow20 · 2022-07-06T10:27:36Z

I think coredns not be resolved correctly with --image-repository argument, example:

root@master:/etc/containerd/certs.d# kubeadm config images list --image-repository=next-repo-host
next-repo-host/kube-apiserver:v1.24.2
next-repo-host/kube-controller-manager:v1.24.2
next-repo-host/kube-scheduler:v1.24.2
next-repo-host/kube-proxy:v1.24.2
next-repo-host/pause:3.7
next-repo-host/etcd:3.5.3-0
next-repo-host/coredns:v1.8.6

next-repo-host/coredns:v1.8.6 is wrong, should conbine with project name next-repo-host/coredns/coredns:v1.8.6

wnow20 · 2022-07-06T10:27:47Z

/reopen

k8s-ci-robot · 2022-07-06T10:28:02Z

@wnow20: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

neolit123 · 2022-07-06T11:34:33Z

See https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images

If the repo is not the official k8s gcr it doesn't the coredns subpath

mhbahmani · 2022-10-09T14:52:33Z

Same here. it isn't possible to pull coredns image with custom image repository.
In this doc, there is a imageTag config that i couldn't find anywhere to set this.
It seems to be a kubeadm problem.

mhbahmani · 2022-10-09T14:57:01Z

more description on stackoverflow:
https://stackoverflow.com/questions/74005758/coredns-image-cannot-be-pulled-when-use-image-repository

neolit123 · 2022-10-09T16:25:05Z

this section explains how to handle it:

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images

For example, one image might have a subpath at registry.k8s.io/subpath/image, but be defaulted to my.customrepository.io/image when using a custom repository.

and steps below.

pacoxu · 2022-10-14T08:31:15Z

Here is a demo.
We use synced image repo in China for GFW.

we sync images from registry.k8s.io to k8s.m.daocloud.io
registry.k8s.io/coredns/coredns is synced to k8s.m.daocloud.io/coredns/coredns.

[root@paco ~]# cat init.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  imageRepository: k8s.m.daocloud.io/coredns
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.m.daocloud.io
kind: ClusterConfiguration
kubernetesVersion: 1.25.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}



[root@paco ~]# kubeadm config images pull --config init.yaml
[config/images] Pulled k8s.m.daocloud.io/kube-apiserver:v1.25.0
[config/images] Pulled k8s.m.daocloud.io/kube-controller-manager:v1.25.0
[config/images] Pulled k8s.m.daocloud.io/kube-scheduler:v1.25.0
[config/images] Pulled k8s.m.daocloud.io/kube-proxy:v1.25.0
[config/images] Pulled k8s.m.daocloud.io/pause:3.8
[config/images] Pulled k8s.m.daocloud.io/etcd:3.5.4-0
[config/images] Pulled k8s.m.daocloud.io/coredns/coredns:v1.9.3

BTW, you can generate the init.yaml by kubeadm config print init-defaults > init.yaml.

et304383 · 2023-04-11T17:20:45Z

This is absurd. This issue must be resolved. We're encountering this trying to create 1.21 K8s cluster with --image-repository pointing at registry.k8s.io

Lirt · 2023-05-26T10:34:49Z

This is example for anyone wondering how to write a kubeadm config file (docs):

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
imageRepository: my-private-registry/registry.k8s.io
dns:
  imageRepository: my-private-registry/registry.k8s.io/coredns

You will also need to edit containerd config to use sandbox image from your private repo:

version = 2
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "my-private-repository/registry.k8s.io/pause:3.8"

Then run kubeadm init --config ...

The old k8s.gcr.io Kubernetes image registry has been frozen in Apr, 2023, in favor of registry.k8s.io. Although old images are still present in the old registry, we have recently started witnessing failures (e.g., [1]) in the k8s-1.16-kernel-4.19 Jenkins test during the provisioning phase, with errors along the lines of: [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.15-0: output: Error response from daemon: Head "https://k8s.gcr.io/v2/etcd/manifests/3.3.15-0": unable to decode token response: invalid character '<' looking for beginning of value Let's attempt to address this error by explicitly configuring the usage of the newer registry, until v1.13 goes EOL and we can finally get rid of these tests. Additionally, we explicitly specify the coredns image repository, as it seems that in certain versions (v1.21 in particular) it otherwise defaults to using an incorrect path (i.e., without the coredns subpath) if a custom registry is specified [2]: ERROR ImagePull]: failed to pull image registry.k8s.io/coredns:v1.8.0: output: Error response from daemon: manifest for registry.k8s.io/coredns:v1.8.0 not found: manifest unknown: Failed to fetch "v1.8.0" [1]: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1323/console [2]: kubernetes/kubeadm#2714 Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

The old k8s.gcr.io Kubernetes image registry has been frozen in Apr, 2023, in favor of registry.k8s.io. Although old images are still present in the old registry, we have recently started witnessing failures (e.g., [1]) in the k8s-1.16-kernel-4.19 Jenkins test during the provisioning phase, with errors along the lines of: [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.15-0: output: Error response from daemon: Head "https://k8s.gcr.io/v2/etcd/manifests/3.3.15-0": unable to decode token response: invalid character '<' looking for beginning of value Let's attempt to address this error by explicitly configuring the usage of the newer registry, until v1.13 goes EOL and we can finally get rid of these tests. Additionally, we explicitly specify the coredns image repository, as it seems that in certain versions (v1.21 in particular) it otherwise defaults to using an incorrect path (i.e., without the coredns subpath) if a custom registry is specified [2]: ERROR ImagePull]: failed to pull image registry.k8s.io/coredns:v1.8.0: output: Error response from daemon: manifest for registry.k8s.io/coredns:v1.8.0 not found: manifest unknown: Failed to fetch "v1.8.0" Finally, let's hard-code the coredns version for older k8s versions, as coredns older than v1.8.0 seems to follow yet another versioning scheme in the registry. [1]: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1323/console [2]: kubernetes/kubeadm#2714 Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

codeachange mentioned this issue Jun 26, 2022

Push the CoreDNS 1.8.0 image in gcr.io kubernetes/kubernetes#99321

Closed

k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Jun 26, 2022

k8s-ci-robot closed this as completed Jun 26, 2022

SGStino mentioned this issue Aug 18, 2022

CoreDNS image incorrectly resolved when using --image-repository and can't use InitConfiguration on existing clusters? #2745

Closed

pacoxu mentioned this issue Mar 13, 2023

Re-add handling of rename from official image "k8s.gcr.io/coredns" to "k8s.gcr.io/coredns/coredns" to support deprecated installations kubernetes/kubernetes#114978

Closed

chess-knight mentioned this issue Jun 21, 2023

Replace k8s registry locations (#412) SovereignCloudStack/k8s-cluster-api-provider#422

Merged

cuttingedge1109 mentioned this issue Jun 27, 2023

fix: CoreDNS image cannot be pulled when use custom registry as kubernetes_image_repository vexxhost/ansible-collection-kubernetes#30

Closed

giorio94 mentioned this issue Jul 18, 2024

test: explicitly configure kubeadm image repository cilium/cilium#33864

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CoreDNS image cannot be pulled when use --image-repository #2714

CoreDNS image cannot be pulled when use --image-repository #2714

codeachange commented Jun 26, 2022

neolit123 commented Jun 26, 2022 via email

k8s-ci-robot commented Jun 26, 2022

wnow20 commented Jul 6, 2022 •

edited

Loading

wnow20 commented Jul 6, 2022

k8s-ci-robot commented Jul 6, 2022

neolit123 commented Jul 6, 2022 •

edited

Loading

mhbahmani commented Oct 9, 2022

mhbahmani commented Oct 9, 2022

neolit123 commented Oct 9, 2022 •

edited

Loading

pacoxu commented Oct 14, 2022 •

edited

Loading

et304383 commented Apr 11, 2023

Lirt commented May 26, 2023 •

edited

Loading

CoreDNS image cannot be pulled when use --image-repository #2714

CoreDNS image cannot be pulled when use --image-repository #2714

Comments

codeachange commented Jun 26, 2022

What keywords did you search in kubeadm issues before filing this one?

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

neolit123 commented Jun 26, 2022 via email

k8s-ci-robot commented Jun 26, 2022

wnow20 commented Jul 6, 2022 • edited Loading

wnow20 commented Jul 6, 2022

k8s-ci-robot commented Jul 6, 2022

neolit123 commented Jul 6, 2022 • edited Loading

mhbahmani commented Oct 9, 2022

mhbahmani commented Oct 9, 2022

neolit123 commented Oct 9, 2022 • edited Loading

pacoxu commented Oct 14, 2022 • edited Loading

et304383 commented Apr 11, 2023

Lirt commented May 26, 2023 • edited Loading

wnow20 commented Jul 6, 2022 •

edited

Loading

neolit123 commented Jul 6, 2022 •

edited

Loading

neolit123 commented Oct 9, 2022 •

edited

Loading

pacoxu commented Oct 14, 2022 •

edited

Loading

Lirt commented May 26, 2023 •

edited

Loading