[OCPCLOUD-1171] Implement CCCMO render to run CCM on bootstrap node #4947

Danil-Grigorev · 2021-05-20T12:21:20Z

This allows master nodes to be initialized with CCM replica from
bootstrap, speeding up process of cluster installation.

Without CCM running on bootstrap node, most of the cluster operator
and operand pods would have to tolerate new taint and a different
configuration of node.kubernetes.io/not-ready (NoSchedule) taint:

effect: NoSchedule
key: node.cloudprovider.kubernetes.io/uninitialized
value: "true"

Without this bootstrap addition or taint toleration the cluster with CCM
would never become ready.

To test this feature on AWS build installer and deploy on AWS.

To create a cluster run:

./hack/build.sh
./bin/openshift-install create manifests
cat <<EOF > manifests/manifest_feature_gate.yaml
---
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  annotations:
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
  name: cluster
spec:
  customNoUpgrade:
    enabled:
    - ExternalCloudProvider
    - CSIMigrationAWS
    - CSIMigrationOpenStack
  featureSet: CustomNoUpgrade
EOF

OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/dgrigore/release:cccmo-render ./bin/openshift-install create cluster

Gist with files CCCMO render outputs for AWS: https://gist.github.com/Danil-Grigorev/5732fd375dba7affff54eda267c4cd47

wking · 2021-05-20T12:28:15Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

+	# Source config map is located in https://github.com/openshift/cluster-cloud-controller-manager-operator/blob/master/manifests/0000_26_cloud-controller-manager-operator_01_images.configmap.yaml
+	bootkube_podman_run \
+	--name copy-cloud-controller-manager-images \
+	--rm -it \


You probably don't need an interactive terminal on the bootstrap machine, right? And since in the happy case we will shortly remove the whole machine, I think we want to keep the metadata for debug logging. Can we drop this line?

wking · 2021-05-20T12:30:30Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

+	# Copy the CCCMO images configMap to resolve pod images from internal registry in CCCMO render run
+	# Source config map is located in https://github.com/openshift/cluster-cloud-controller-manager-operator/blob/master/manifests/0000_26_cloud-controller-manager-operator_01_images.configmap.yaml
+	bootkube_podman_run \
+	--name copy-cloud-controller-manager-images \


nit: existing style seems to be an extra level of indentation for these wrapped lines.

Error: error creating container storage: the container name "copy-cloud-controller-manager-images" is already in use by "5ca8294f8b2b2089541e267b4d46aeed08295f1ca16e14a45dd80807bf8e03ca". You have to remove that container to be able to reuse that name.: that name is already in use

see my comment below, that is the root cause, https://github.com/openshift/installer/pull/4947/files#r662375716

wking · 2021-05-20T12:33:14Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

+		--dest-dir=/assets/cloud-controller-manager-bootstrap \
+		--cluster-infrastructure-file=/assets/manifests/cluster-infrastructure-02-config.yml
+
+	cp cloud-controller-manager-bootstrap/bootstrap/* bootstrap-manifests//


nit: // -> /

wking · 2021-05-20T12:34:35Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

+	cp cloud-controller-manager-bootstrap/bootstrap/* bootstrap-manifests//
+
+	touch cloud-controller-manager-bootstrap.done
+    record_service_stage_success


nit: inconsistent indent, although that appears to be the existing style for record_service_stage_success in this file.

wking · 2021-05-20T12:40:27Z

...most of the cluster operator and operand pods...

Since a fair amount of thought and testing has gone into this, I think it's worth explaining the deadlock that this is avoiding in more detail. Which operators and pods? Why would their being unable to come up on the tainted control plane node's block the cloud-controller-manager from coming up?

Danil-Grigorev · 2021-05-20T15:13:38Z

@wking In my testing I saw CCCMO stuck in containerCreating state, as it was not able to pull the image. It might be that we should start with image-registry, the rest is under still under investigation. Would definitely document the findings

wking · 2021-05-20T22:17:29Z

It might be that we should start with image-registry...

Image-registry is for local images, and the release image and its references all come from external registries during bootstrapping. The local registry is one of the last components to come up, because it runs on computer node's, which are in turn provisioned by machine-API components hosted on control-plane nodes.

JoelSpeed · 2021-05-26T11:02:34Z

@Danil-Grigorev Just want to make sure that you are aware that, this PR, in its current state, shouldn't merge until the CCCMO image has been added to the full payload, and are accounting for that in your planning around this.

If you want to merge this before CCCMO is added to the full payload, you will need to ensure that the bootstrap script accounts for when there is no CCCMO image

Danil-Grigorev · 2021-05-27T11:15:42Z

@JoelSpeed We would definitely look into this once again once render implementation would get merged in CCCMO and release addition request would be satisfied. The whole feature won't work until OpenStack, AWS and Azure cloud-provider repos won't be included in the release too.

This allows master nodes to be initialized with CCM replica from bootstrap, speeding up process of cluster installation. Without CCM running on bootstrap node, most of the cluster operator and operand pods would have to tolerate new taint and a different configuration of node.kubernetes.io/not-ready (NoSchedule) taint: - effect: NoSchedule key: node.cloudprovider.kubernetes.io/uninitialized value: "true" - effect: NoSchedule key: node.kubernetes.io/not-ready Without this bootstrap addition or taint toleration the cluster with CCM would never become ready. To test this feature on AWS build installer and deploy on AWS. To create a cluster run: ```bash ./hack/build.sh ./bin/openshift-install create manifests cat <<EOF > manifests/manifest_feature_gate.yaml --- apiVersion: config.openshift.io/v1 kind: FeatureGate metadata: annotations: include.release.openshift.io/self-managed-high-availability: "true" include.release.openshift.io/single-node-developer: "true" release.openshift.io/create-only: "true" name: cluster spec: customNoUpgrade: enabled: - ExternalCloudProvider - CSIMigrationAWS - CSIMigrationOpenStack featureSet: CustomNoUpgrade EOF OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/dgrigore/release:cccmo-render ./bin/openshift-install create cluster ```

elmiko

ran into a bug while testing

elmiko · 2021-07-01T14:22:07Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

+	# Copy rendered resources to manifests folder
+	cp cloud-controller-manager-bootstrap/bootstrap/* bootstrap-manifests/
+	# Copy cloud config to /etc/kubernetes/bootstrap-configs
+	cp cloud-controller-manager-bootstrap/config/* /etc/kubernetes/bootstrap-configs


this will fail if no cloud config is generated, when i run on aws and there is no cloud config, i see this in the bootkube.service output:

Jul 01 13:29:18 ip-10-0-12-89 bootkube.sh[2207]: cp: cannot stat 'cloud-controller-manager-bootstrap/config/*': No such file or directory

this patch is allowing me to proceed

diff --git a/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template b/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template index a36dbee28..3c3c61d6c 100755 --- a/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template +++ b/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template @@ -245,8 +245,10 @@ then -c "cat /release-manifests/0000_26_cloud-controller-manager-operator_01_images.configmap.yaml" > manifests/ccm-images.yaml ADDITIONAL_FLAGS="" + CLOUD_CONFIG_EXISTS="" if [ -f "$PWD/manifests/cloud-provider-config.yaml" ]; then ADDITIONAL_FLAGS="--cloud-config-file=/assets/manifests/cloud-provider-config.yaml" + CLOUD_CONFIG_EXISTS="yes" fi bootkube_podman_run \ @@ -262,7 +264,9 @@ then # Copy rendered resources to manifests folder cp cloud-controller-manager-bootstrap/bootstrap/* bootstrap-manifests/ # Copy cloud config to /etc/kubernetes/bootstrap-configs - cp cloud-controller-manager-bootstrap/config/* /etc/kubernetes/bootstrap-configs + if [ ! -z "$CLOUD_CONFIG_EXISTS" ]; then + cp cloud-controller-manager-bootstrap/config/* /etc/kubernetes/bootstrap-configs + fi touch cloud-controller-manager-bootstrap.done

Ahh... Bash weirdness. Apparently this throws an error you saw if the directory is empty, even if there is wildcard at the source for the copy.

cp cloud-controller-manager-bootstrap/config/* /etc/kubernetes/bootstrap-configs

This works instead, even if the directory is empty:

cp -r cloud-controller-manager-bootstrap/config/. /etc/kubernetes/bootstrap-configs

openshift-ci · 2021-07-01T14:23:46Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign patrickdillon after the PR has been reviewed.
You can assign the PR to them by writing /assign @patrickdillon in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Fedosin · 2021-07-01T16:16:00Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

+	bootkube_podman_run \
+	--name copy-cloud-controller-manager-images \
+	--entrypoint /bin/bash "${RELEASE_IMAGE_DIGEST}" \
+	-c "cat /release-manifests/0000_26_cloud-controller-manager-operator_01_images.configmap.yaml" > manifests/ccm-images.yaml


/release-manifests/0000_26_cloud-controller-manager-operator_01_images.configmap.yaml: No such file or directory

Was it a CI build? Seems like CCCMO is not present in the payload you were using

Fedosin · 2021-07-01T16:16:52Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

+	# Copy the CCCMO images configMap to resolve pod images from internal registry in CCCMO render run
+	# Source config map is located in https://github.com/openshift/cluster-cloud-controller-manager-operator/blob/master/manifests/0000_26_cloud-controller-manager-operator_01_images.configmap.yaml
+	bootkube_podman_run \
+	--name copy-cloud-controller-manager-images \


Error: error creating container storage: the container name "copy-cloud-controller-manager-images" is already in use by "5ca8294f8b2b2089541e267b4d46aeed08295f1ca16e14a45dd80807bf8e03ca". You have to remove that container to be able to reuse that name.: that name is already in use

…g in CCCMO

openshift-ci · 2021-07-02T11:12:48Z

@Danil-Grigorev: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-libvirt	`a45cffc`	link	`/test e2e-libvirt`
ci/prow/e2e-aws-workers-rhel7	`a45cffc`	link	`/test e2e-aws-workers-rhel7`
ci/prow/e2e-aws-upgrade	`a45cffc`	link	`/test e2e-aws-upgrade`
ci/prow/e2e-crc	`a45cffc`	link	`/test e2e-crc`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

JoelSpeed · 2021-07-15T12:01:14Z

/close

We have decided to take an alternate approach to bootstrapping which means this is no longer required.

See openshift/enhancements#463 for more details on the updated approach

openshift-ci · 2021-07-15T12:01:23Z

@JoelSpeed: Closed this PR.

In response to this:

/close

We have decided to take an alternate approach to bootstrapping which means this is no longer required.

See openshift/enhancements#463 for more details on the updated approach

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested review from patrickdillon and rna-afk May 20, 2021 12:21

wking reviewed May 20, 2021

View reviewed changes

Danil-Grigorev mentioned this pull request May 20, 2021

[OCPCLOUD-1162] Implement render binary for use in installer openshift/cluster-cloud-controller-manager-operator#45

Merged

Danil-Grigorev force-pushed the cccmo-render branch from dd6b99b to 233896e Compare June 8, 2021 10:57

Danil-Grigorev mentioned this pull request Jun 8, 2021

OCPCLOUD-1194: Implement cloud-config-file flag in CCCMO render openshift/cluster-cloud-controller-manager-operator#55

Merged

Danil-Grigorev changed the title ~~Implement CCCMO render to run CCM on bootstrap node~~ [OCPCLOUD-1171] Implement CCCMO render to run CCM on bootstrap node Jun 14, 2021

This was referenced Jun 16, 2021

[OCPCLOUD-1162] Implement image substitution for Pod and generalise the approach for other objects openshift/cluster-cloud-controller-manager-operator#58

Merged

Step addition: Pre-create CCM feature gate during cluster install openshift/release#19366

Merged

elmiko suggested changes Jul 1, 2021

View reviewed changes

Fedosin reviewed Jul 1, 2021

View reviewed changes

elmiko mentioned this pull request Jul 1, 2021

[OCPCLOUD-1209] Add feature gate to bootstrap mode manifests openshift/machine-config-operator#2647

Closed

Add support for cloud-config-file flag to unpack and copy cloud-confi…

a45cffc

…g in CCCMO

Danil-Grigorev force-pushed the cccmo-render branch from 233896e to a45cffc Compare July 2, 2021 08:25

openshift-ci bot closed this Jul 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OCPCLOUD-1171] Implement CCCMO render to run CCM on bootstrap node #4947

[OCPCLOUD-1171] Implement CCCMO render to run CCM on bootstrap node #4947

Danil-Grigorev commented May 20, 2021 •

edited

Loading

wking May 20, 2021

wking May 20, 2021

Fedosin Jul 1, 2021

elmiko Jul 1, 2021

wking May 20, 2021

wking May 20, 2021 •

edited

Loading

wking commented May 20, 2021

Danil-Grigorev commented May 20, 2021

wking commented May 20, 2021 •

edited

Loading

JoelSpeed commented May 26, 2021

Danil-Grigorev commented May 27, 2021

elmiko left a comment

elmiko Jul 1, 2021

elmiko Jul 1, 2021 •

edited

Loading

Danil-Grigorev Jul 2, 2021

openshift-ci bot commented Jul 1, 2021

Fedosin Jul 1, 2021

Danil-Grigorev Jul 1, 2021

Fedosin Jul 1, 2021

openshift-ci bot commented Jul 2, 2021

JoelSpeed commented Jul 15, 2021

openshift-ci bot commented Jul 15, 2021

[OCPCLOUD-1171] Implement CCCMO render to run CCM on bootstrap node #4947

[OCPCLOUD-1171] Implement CCCMO render to run CCM on bootstrap node #4947

Conversation

Danil-Grigorev commented May 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wking May 20, 2021 • edited Loading

Choose a reason for hiding this comment

wking commented May 20, 2021

Danil-Grigorev commented May 20, 2021

wking commented May 20, 2021 • edited Loading

JoelSpeed commented May 26, 2021

Danil-Grigorev commented May 27, 2021

elmiko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elmiko Jul 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 2, 2021

JoelSpeed commented Jul 15, 2021

openshift-ci bot commented Jul 15, 2021

Danil-Grigorev commented May 20, 2021 •

edited

Loading

wking May 20, 2021 •

edited

Loading

wking commented May 20, 2021 •

edited

Loading

elmiko Jul 1, 2021 •

edited

Loading