Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm init phase certs apiserver should not need to check CRI endpoints #2937

Closed
FerminCastro opened this issue Sep 22, 2023 · 13 comments
Closed
Assignees
Labels
area/phases kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@FerminCastro
Copy link

What happened?

While adding additional hostname alias to kube-api, we are using kubeadm to generate new certificates including the new hostnames:

[opc@k8iramaster-1 ~]$ cat  /tmp/kubeadm-newhn.yaml | grep certSANs -A2 -B2
apiServer:
  certSANs:
  - "test.com"
  extraArgs:

When we try to apply this new kubeadm config, the operation is failing in all nodes because https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/runtime/runtime.go#L148 is checking for CRI endpoints and it is finding multiple endpoints (docker and containerd):

[opc@k8iramaster-1 ~]$ sudo kubeadm init phase certs apiserver --config /tmp/kubeadm-newhn.yaml
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock
To see the stack trace of this error execute with --v=5 or higher

we don't understand why generating new certs needs to check for CRI endpoints. We do know we can fix this including the appropriate nodeRegistration.criSocket info in the kubeadm config file, but we think this CRI endpoint is an overkill for this op

What did you expect to happen?

Update of kube-api certSANs should NOT require CRI endpoints verifications

How can we reproduce it (as minimally and precisely as possible)?

In a system running docker (uses containerd as container runtime), execute kubeadm init phase certs apiserver to update certs

Anything else we need to know?

No response

Kubernetes version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.11", GitCommit:"8cfcba0b15c343a8dc48567a74c29ec4844e0b9e", GitTreeState:"clean", BuildDate:"2023-06-14T09:57:26Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.11", GitCommit:"8cfcba0b15c343a8dc48567a74c29ec4844e0b9e", GitTreeState:"clean", BuildDate:"2023-06-14T09:49:38Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

NA

OS version

$ cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="7.9"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Oracle Linux Server 7.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:9:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.9
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.9
$ uname -a
Linux k8iramaster-1 5.4.17-2136.319.1.2.el7uek.x86_64 kubernetes/kubernetes#2 SMP Wed May 10 16:56:50 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux
</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>
Client: Docker Engine - Community
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        42e35e61f3
 Built:             Mon Jun  1 09:13:48 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       42e35e61f3
  Built:            Mon Jun  1 09:12:26 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>
@FerminCastro FerminCastro added the kind/bug Categorizes issue or PR as related to a bug. label Sep 22, 2023
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 22, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@FerminCastro
Copy link
Author

/sig cluster-lifecycle

@k8s-ci-robot k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 22, 2023
@neolit123
Copy link
Member

/transfer kubeadm

@k8s-ci-robot k8s-ci-robot transferred this issue from kubernetes/kubernetes Sep 22, 2023
@SataQiu
Copy link
Member

SataQiu commented Sep 22, 2023

/assign

@neolit123
Copy link
Member

it's a valid argument and maybe we can resolve the problem. we've done similar in the past.
however this node is not useable for init/join etc due to the double CRs on the host.

@neolit123 neolit123 added priority/backlog Higher priority than priority/awaiting-more-evidence. area/phases and removed sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 22, 2023
@SataQiu
Copy link
Member

SataQiu commented Sep 22, 2023

@FerminCastro Thanks for your feedback.
This problem has been fixed since v1.27 kubernetes/kubernetes#114455
What version of kubeadm are you using?
Perhaps you need to upgrade your kubeadm version.

@SataQiu
Copy link
Member

SataQiu commented Sep 22, 2023

A workaround is to set the criSocket field to something, but not leave it as blank(empty), for example:

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: unix:///var/run/something.sock
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  certSANs:
  - "test.com"

@FerminCastro
Copy link
Author

Yes, we have actually updated our scripts to get the criSocket from the local kubelet proc. You can check it at
https://github.com/oracle-samples/maa/blob/main/kubernetes-maa/maak8s-kube-api-alias.sh

@neolit123
Copy link
Member

Yes, we have actually updated our scripts to get the criSocket from the local kubelet proc. You can check it at https://github.com/oracle-samples/maa/blob/main/kubernetes-maa/maak8s-kube-api-alias.sh

normally kubeadm as the higher level tool tells the kubelet what socket to use. the kubelet doesn't have auto detection.

@FerminCastro
Copy link
Author

FerminCastro commented Sep 25, 2023

Well if you have more than one crisocket kubelet would be running with the specific one you want to use, Right?:
endpoint=$(tr \0 ' ' < /proc/"$(pgrep kubelet)"/cmdline | awk -F'--container-runtime-endpoint=' '{print $2}' |awk '{print $1}')

@neolit123
Copy link
Member

neolit123 commented Sep 25, 2023

Well if you have more than one crisocket kubelet would be running with the specific one you want to use, Right?: endpoint=$(tr \0 ' ' < /proc/"$(pgrep kubelet)"/cmdline | awk -F'--container-runtime-endpoint=' '{print $2}' |awk '{print $1}')

if you explicitly pass criSocket in kubeadm it will end up as the flag --container-runtime-endpoint, which means you don't need a script to take the socket from the kubelet as you already know what socket you want, supposedly.

@FerminCastro
Copy link
Author

I need to detect the socket from the node. Plus kubeadm init phase certs apiserver does not allow using --crisocket as flag

@FerminCastro
Copy link
Author

Since this is addressed in v1.27 kubernetes/kubernetes#114455, I am closing this one #since v1.27 kubernetes/kubernetes#114455

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/phases kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Development

No branches or pull requests

4 participants