kcm rootCA missing apiserver ca leads to kube-root-ca problem

[lance5890]

1 Bug phenomenon：

1. The kcm(kube-controller-manager) rootCA is generated by manageServiceAccountCABundle in targerconfigcontroller, and this func will get kube-apiserver-server-ca cm first , and then use it and other two cm to generate kcm rootCA
2. when sometimes（very unlikely to happen , but i have met this just one time） the kube-apiserver-server-ca cm is missing, and the manageServiceAccountCABundle generate rootCA without kube-apiserver-server-ca , and finally the kcm leader holds the wrong rootCA， it will lead to the kube-root-ca problem in every pod, and the ocp release wil not work until i stop the wrong kcm leader

2 Bug fix:
2.1. this bug can be resolved by adding kube-apiserver-server-ca check in manageServiceAccountCABundle of targerconfigcontroller ,as follows

cluster-kube-controller-manager-operator/pkg/operator/targetconfigcontroller/targetconfigcontroller.go

Lines 706 to 720 in 4ca346e

	func manageServiceAccountCABundle(ctx context.Context, lister corev1listers.ConfigMapLister, client corev1client.ConfigMapsGetter, recorder events.Recorder) (*corev1.ConfigMap, bool, error) {
	requiredConfigMap, err := resourcesynccontroller.CombineCABundleConfigMaps(
	resourcesynccontroller.ResourceLocation{Namespace: operatorclient.TargetNamespace, Name: "serviceaccount-ca"},
	lister,
	// include the ca bundle needed to recognize the server
	resourcesynccontroller.ResourceLocation{Namespace: operatorclient.GlobalMachineSpecifiedConfigNamespace, Name: "kube-apiserver-server-ca"},
	// include the ca bundle needed to recognize default
	// certificates generated by cluster-ingress-operator
	resourcesynccontroller.ResourceLocation{Namespace: operatorclient.GlobalMachineSpecifiedConfigNamespace, Name: "default-ingress-cert"},
	)
	if err != nil {
	return nil, false, err
	}
	return resourceapply.ApplyConfigMap(ctx, client, recorder, requiredConfigMap)
	}

2.1 maybe like this , but this is not the best way to resolve this problem

2.3 this is also can be resolved by modifying the openshift library func CombineCABundleConfigMaps in resourcesynccontroller as

cluster-kube-controller-manager-operator/vendor/github.com/openshift/library-go/pkg/operator/resourcesynccontroller/core.go

Lines 17 to 67 in 4ca346e

	func CombineCABundleConfigMaps(destinationConfigMap ResourceLocation, lister corev1listers.ConfigMapLister, inputConfigMaps ...ResourceLocation) (*corev1.ConfigMap, error) {
	certificates := []*x509.Certificate{}
	for _, input := range inputConfigMaps {
	inputConfigMap, err := lister.ConfigMaps(input.Namespace).Get(input.Name)
	if apierrors.IsNotFound(err) {
	continue
	}
	if err != nil {
	return nil, err
	}
	// configmaps must conform to this
	inputContent := inputConfigMap.Data["ca-bundle.crt"]
	if len(inputContent) == 0 {
	continue
	}
	inputCerts, err := cert.ParseCertsPEM([]byte(inputContent))
	if err != nil {
	return nil, fmt.Errorf("configmap/%s in %q is malformed: %v", input.Name, input.Namespace, err)
	}
	certificates = append(certificates, inputCerts...)
	}
	certificates = crypto.FilterExpiredCerts(certificates...)
	finalCertificates := []*x509.Certificate{}
	// now check for duplicates. n^2, but super simple
	for i := range certificates {
	found := false
	for j := range finalCertificates {
	if reflect.DeepEqual(certificates[i].Raw, finalCertificates[j].Raw) {
	found = true
	break
	}
	}
	if !found {
	finalCertificates = append(finalCertificates, certificates[i])
	}
	}
	caBytes, err := crypto.EncodeCertificates(finalCertificates...)
	if err != nil {
	return nil, err
	}
	return &corev1.ConfigMap{
	ObjectMeta: metav1.ObjectMeta{Namespace: destinationConfigMap.Namespace, Name: destinationConfigMap.Name},
	Data: map[string]string{
	"ca-bundle.crt": string(caBytes),
	},
	}, nil
	}

3 This is related to the openshift library-go issue # issue 1472 github.com/openshift/library-go missing key configmap

[lance5890]

@ingvagabund @atiratree @deads2k @soltysh

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[lance5890]

/remove-lifecycle stale

[lance5890]

/lifecycle frozen