Implement OpenStack Cloud Controller Manager support #423

Fedosin · 2020-08-06T12:41:03Z

No description provided.

Fedosin · 2020-08-06T13:23:15Z

/assign @enxebre

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

JoelSpeed · 2020-08-24T09:24:13Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+
+**Note:** Example of manual testing: https://asciinema.org/a/303399?speed=2
+
+#### Write Cluster Cloud Controller Manager Operator


Which team should be owning this long term? Would it make sense for the cloud-team to own this?

Yep, the cloud team. Because I expect there will be more cloud controller manager for other platforms.
But OpenStack team can maintain some related submodules by placing OWNERS file in appropriate places.

Do we have a clear understanding of how the code structure will be laid out to support generic CCCMO vs platform specific so that we can properly partition the work and have the correct OWNERS file?

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

derekwaynecarr · 2020-08-24T15:07:24Z

Can you share any thoughts on the intersection of this component with the Infrastructure type?
https://github.com/openshift/api/blob/master/config/v1/types_infrastructure.go#L25

I assume there is no additional configurability needed for OpenStack?

jichenjc · 2020-08-25T07:35:07Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+
+Actions:
+
+- Make sure Cinder CSI driver is supported in OpenShift.


why this CSI is related to CCM? seems those are different operator/repo?

Now we use in-tree cloud provider, that manages Cinder volumes as well. Cloud controller manager can't do it, since this functionality was moved to Cinder CSI driver. So, to maintain feature parity with the in-tree cloud provider we need to enable Cloud Controller Manager and deploy Cinder CSI driver.

jichenjc · 2020-08-25T07:38:21Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+
+- Manually install CCM's [daemonset](https://github.com/kubernetes/cloud-provider-openstack/blob/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml) on a working OCP cluster deployed on OpenStack.
+
+- Update configuration of `kube-apiserver`, `kube-controller-manager` and `kubelet` by replacing `--cloud-provider openstack` with `--cloud-provider external` and removing `--cloud-config` parameters.


this operator installed optionally or mandatory?
I think you need have CCM running in order to make node ready, which meaning this is not installed from OperateHub?

The operator will be installed mandatory. But depending on the platform, it will either deploy Cloud Controller Manager or not.

openshift-ci-robot · 2020-09-02T13:18:25Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Fedosin
To complete the pull request process, please assign enxebre
You can assign the PR to them by writing /assign @enxebre in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

EricArrakis · 2020-09-03T20:50:40Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+
+- Implement the operator, using [library-go](https://github.com/openshift/library-go) primitives.
+
+#### Build the operator image by OCP automation


Do we know which group in OCP will be responsible for the image building automation? Have we engaged them yet?

ART team, because it will be a cluster operator. They are not involved yet.

EricArrakis · 2020-09-03T20:51:15Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+
+#### Write Cluster Cloud Controller Manager Operator
+
+The operator should be able to create, configure and manage different CCMs (including one for OpenStack) in OpenShift.


Are these operations the scope of the unit/CI tests? Are there more/others?

I think the main goal is to reach the feature parity with the in-tree cloud provider. So, if e2e-openstack tests pass, then we are good.
For sure, unit tests for the operator are required.

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

EricArrakis · 2020-09-03T20:53:26Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+- Add CCM operator support in Cluster Version Operator. CCM operator will be deployed on early stages of installation and then it deploys and configures OpenStack CCM itself.
+
+- Change Kubelet configuration for OpenStack in [Machine Config Operator templates](https://github.com/openshift/machine-config-operator/blob/d044c74ea4b9900c269ee8de8131ed49ba6fedc8/templates/master/01-master-kubelet/openstack/units/kubelet.service.yaml#L30) to adopt external cloud provider.


Should we have someone from CVO and MCO review?

Yes, 3CMO (cluster cloud controller manager operator) will be managed by CVO so their opinion is really appreciated.

There won't be so much interaction with MCO, I believe. We will need to change just one line of code there:
https://github.com/openshift/machine-config-operator/blob/master/templates/master/01-master-kubelet/openstack/units/kubelet.service.yaml#L30
But the more people look, the better :)

enxebre · 2020-09-17T10:44:37Z

A lot of the transition for all out of tree providers is being worked with detail here #463. I'd suggest we conflate any Openstack specific info into 463 and close this PR to avoid confusion and scatter knowledge for the same topic. We can then later in time follow up on 463 as we add more providers. cc @Danil-Grigorev @Fedosin

sttts · 2020-09-28T16:01:47Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+A: If CCM is not available, new nodes in the cluster will be left unschedulable.
+
+Q: Who talks to CCM?
+A: From architectural point of view, only Kube API server is able to communicate with CCM. [Source](https://kubernetes.io/docs/concepts/architecture/cloud-controller/#design). All requests to CCM are sent by `kubelet`. KCM doesn't interract with CCM.


kube-apiserver or kubelet (have not yet read the link) ?

kube-apiserver as written below.

sttts · 2020-09-28T16:03:04Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+A: From architectural point of view, only Kube API server is able to communicate with CCM. [Source](https://kubernetes.io/docs/concepts/architecture/cloud-controller/#design). All requests to CCM are sent by `kubelet`. KCM doesn't interract with CCM.
+
+Q: Does CCM provide an API? How is it hosted? HTTPS endpoint? Through load balancer?
+A: CCM provides a gRPC interface on port [10258](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/cloud-provider/ports.go#L22). Like KCM, CCM supports high available setup using leader election (which is on by default). [Source](https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/#requirements).


this means that clients of ccm (~~kubelet?~~ kube-apiserver ?) have to find the leader? I.e. you need a load balancer? Or is it a kubernetes service?

In the former case you have to integrate with the cloud API LB, and also integrate with MCO and it gcp/azure-routes.sh mechanism. Those might even be duplicated because API server and ccm life-cycles don't match.

In the latter case, you depend on SDN. What would that mean? No scheduling, no SDN. No SDN, no ccm, no scheduling.

Or you tell kube-apiserver about all the IPs similarly to etcd-operator today. Grpc can handle multiple endpoints and find one that works.

sttts · 2020-09-28T16:05:03Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+A: CCM provides a gRPC interface on port [10258](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/cloud-provider/ports.go#L22). Like KCM, CCM supports high available setup using leader election (which is on by default). [Source](https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/#requirements).
+
+Q: What are the thoughts about certificate management?
+A: Certificate management should be performed by the CCM operator, like it's done in the KCM operator.


The consumers of ccm have to know the CA. How should that work? Note, that kcm has no API beyond the metrics and health endpoints, i.e. a very different and simpler setup. What you describe sounds more like kube-apiserver.

sttts · 2020-09-28T16:05:55Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+A: No. Control plane nodes only. [Source](https://kubernetes.io/docs/concepts/overview/components/#cloud-controller-manager)
+
+Q: How does SDN depend on CCM?
+A: Most likely they are not related.


See the LB requirement above.

sttts · 2020-09-28T16:14:15Z

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md

+
+2. How to migrate PVs created by the in-tree cloud provider?
+[CSIMigration](https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/) looks like the best option, especially if it is GA in 1.19.
+


we need a section about bootstrapping. Probably we need a bootstrap instance of ccm.

Fedosin · 2020-09-29T10:40:02Z

I close this PR now in favor of #463

openshift-ci-robot requested review from kbsingh and mrunalp August 6, 2020 12:41

openshift-ci-robot assigned enxebre Aug 6, 2020

russellb mentioned this pull request Aug 17, 2020

enhancement proposal for Equinix Metal IPI #436

Merged

mandre reviewed Aug 21, 2020

View reviewed changes

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md Show resolved Hide resolved

JoelSpeed reviewed Aug 24, 2020

View reviewed changes

derekwaynecarr reviewed Aug 24, 2020

View reviewed changes

enhancements/cloud-controller-manager/openstack-cloud-controller-manager.md Show resolved Hide resolved

jichenjc reviewed Aug 25, 2020

View reviewed changes

Fedosin force-pushed the openstack-cloud-controller-manager branch from b048501 to 8f8c198 Compare September 2, 2020 13:18

EricArrakis reviewed Sep 3, 2020

View reviewed changes

russellb mentioned this pull request Sep 10, 2020

Describing steps to support out-of-tree providers #463

Merged

Fedosin force-pushed the openstack-cloud-controller-manager branch from 8f8c198 to 86fecb2 Compare September 16, 2020 13:20

jichenjc mentioned this pull request Sep 22, 2020

[cinder-csi-plugin] find device by path from lun address kubernetes/cloud-provider-openstack#1197

Closed

Fedosin force-pushed the openstack-cloud-controller-manager branch from 86fecb2 to 994cf2b Compare September 28, 2020 15:54

sttts reviewed Sep 28, 2020

View reviewed changes

Implement OpenStack Cloud Controller Manager support

9c7789e

Fedosin force-pushed the openstack-cloud-controller-manager branch from 994cf2b to 9c7789e Compare September 29, 2020 10:28

Fedosin closed this Sep 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement OpenStack Cloud Controller Manager support #423

Implement OpenStack Cloud Controller Manager support #423

Fedosin commented Aug 6, 2020

Fedosin commented Aug 6, 2020

JoelSpeed Aug 24, 2020

Fedosin Sep 2, 2020

EricArrakis Sep 3, 2020

derekwaynecarr commented Aug 24, 2020

jichenjc Aug 25, 2020

Fedosin Sep 2, 2020

jichenjc Aug 25, 2020

Fedosin Sep 2, 2020

openshift-ci-robot commented Sep 2, 2020

EricArrakis Sep 3, 2020

Fedosin Sep 9, 2020

EricArrakis Sep 3, 2020

Fedosin Sep 9, 2020

EricArrakis Sep 3, 2020

Fedosin Sep 9, 2020

enxebre commented Sep 17, 2020 •

edited

Loading

sttts Sep 28, 2020

sttts Sep 28, 2020

sttts Sep 28, 2020 •

edited

Loading

sttts Sep 28, 2020

sttts Sep 28, 2020

sttts Sep 28, 2020

sttts Sep 28, 2020

sttts Sep 28, 2020

Fedosin commented Sep 29, 2020


		Note: Example of manual testing: https://asciinema.org/a/303399?speed=2

		#### Write Cluster Cloud Controller Manager Operator


		Actions:

		- Make sure Cinder CSI driver is supported in OpenShift.


		- Manually install CCM's [daemonset](https://github.com/kubernetes/cloud-provider-openstack/blob/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml) on a working OCP cluster deployed on OpenStack.

		- Update configuration of `kube-apiserver`, `kube-controller-manager` and `kubelet` by replacing `--cloud-provider openstack` with `--cloud-provider external` and removing `--cloud-config` parameters.


		- Implement the operator, using [library-go](https://github.com/openshift/library-go) primitives.

		#### Build the operator image by OCP automation


		#### Write Cluster Cloud Controller Manager Operator

		The operator should be able to create, configure and manage different CCMs (including one for OpenStack) in OpenShift.

		- Add CCM operator support in Cluster Version Operator. CCM operator will be deployed on early stages of installation and then it deploys and configures OpenStack CCM itself.

		- Change Kubelet configuration for OpenStack in [Machine Config Operator templates](https://github.com/openshift/machine-config-operator/blob/d044c74ea4b9900c269ee8de8131ed49ba6fedc8/templates/master/01-master-kubelet/openstack/units/kubelet.service.yaml#L30) to adopt external cloud provider.


		2. How to migrate PVs created by the in-tree cloud provider?
		[CSIMigration](https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/) looks like the best option, especially if it is GA in 1.19.

Implement OpenStack Cloud Controller Manager support #423

Implement OpenStack Cloud Controller Manager support #423

Conversation

Fedosin commented Aug 6, 2020

Fedosin commented Aug 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derekwaynecarr commented Aug 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci-robot commented Sep 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enxebre commented Sep 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts Sep 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fedosin commented Sep 29, 2020

enxebre commented Sep 17, 2020 •

edited

Loading

sttts Sep 28, 2020 •

edited

Loading