Describing steps to support out-of-tree providers

enhancements/machine-api/out-of-tree-provider-support.md

@@ -332,37 +332,18 @@ Once the provider is moved to out-of-tree, the migration mechanism will be disab
 
 #### Bootstrap changes
 
-Once an out-of-tree provider is released (GA), the `CCM` will be created as a static pod on the bootstrap node, to ensure swift removal of the `node.cloudprovider.kubernetes.io/uninitialized` taint from any newly created `Nodes`. Later stages, including cluster upgrades will be managed by an operator, which will ensure stability of the configuration, and will run the `CCM` in a `Deployment`.
-Initial usage of the static pod is justified by the need for `CCM` to initialise `Nodes` before the `kube-scheduler` is able to schedule regular workloads (eg the operator for `CCM` managed by Deployment).
-
-A static pod deployed on the bootstrap node will only run the `cloud-node` controller. This controller in particular, manages `Node` taint removal and all other `Node` related operations in `CCM`. Other controllers are not needed during bootstrap, and so can be excluded.
-
-The bootstrap static pod for the cloud provider will be provisioned unconditionally on bootstrap nodes once the platform is supported by the `CCCMO` render implementation.
-The constraint for enabling the `cloud-node` controller only on bootstrap will make sure this pod won't do anything for cloud providers which have not yet fully transitioned to `CCM`s, or don't have the `TechPreview` `FeatureGate` enabled from startup.
-
-*Note: Render implementation used in bootstrap is currently a standard step for operators required to be a part of day 1 installation procedure. Our case is justified by reqirement to provision `control-plane` Nodes, and allow API server to be fully rolled-out there.*
-
-The procedure would include the following:
-
-1) `cluster-config-operator` render would fully populate `cloud-config` `ConfigMap` and store in a manifest on the bootstrap machine.
-2) `CCCMO` would generate `CCM` manifest in form of a static pod, and provide all nessesary dependencies, such as `cloud-config` or `cloud-credentials` to use. `CCM` would have only `cloud-node` controller enabled.
-3) All static pods created by render steps, including `CCM`, would be copied to static manifests folder for bootstrap `kubelet` to pick up.
+One of the responsibilities of the initialisation process for Kubelet is to set the `Node`’s IP addresses within the status of the `Node` object. The remaining responsibilities are not important for this document bar the removal of a taint which prevents workloads running on the `Node` until the initialisation has completed.
 
-`CCCMO` will provide a `render` implementation, similar to other operators, to generate initial manifests that allow deployment of `CCM` as a static pod on the bootstrap node.
-This will be used with the `bootkube.sh` [script](https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template).
+A second part of the bootstrap process for a new `Node`, is to initialise the `CNI` (networking). Typically in an OpenShift cluster, this is handled once the Networking Operator starts.
+The Networking operator will create the `CNI` pods (typically OpenShift SDN), which schedule on the `Node`, use the `Node` IP addresses to create a `HostSubnet` resource within Kubernetes and then mark then complete the initialisation process for the `CNI`, in doing so, marking the `Node` as ready and allowing the remaining workloads to start.
 
-At the initial stage of `CCM` installation, [installer](https://github.com/openshift/installer) creates the initial cloud-config `ConfigMap` and [cluster-config-operator](https://github.com/openshift/cluster-config-operator) could additionally populate (depending on platform) some values to the config.
-Here are static contents of `cloud.conf` for OpenStack, which are generated by the installer:
+Before the `CNI` is initialized on a `Node`, in-cluster networking such as Service IPs, in particular the API server Service, will not work for any `Pod` on the `Node`. Additionally, any `Pod` that requires the Pod Networking implemented by `CNI`, cannot start. For this reason, `Pods` such as the Networking Operator must use host networking and the “API Int” load balancer to contact the Kube API Server.
 
-Example for OpenStack:
+Because the `CCM` is taking over the responsibility of setting the `Node` IP addresses, `CCM` will become a prerequisite for networking to become functional within any Cluster. Because the `CNI` is not initialised, we must ensure that the `CCCMO` and `CCM` Pods tolerate the scenario where `CNI` is non-functional.
 
-```txt
-[Global]
-secret-name = openstack-credentials
-secret-namespace = kube-system
-```
+To do so, we must tolerate the not-ready taint for these pods and they must all run with host networking and use the API load balancer, rather than using the internal Service. This will ensure that the cluster can bootstrap successfully and recover from any disaster recovery scenario.
 
-Resources deployed on the bootstrap machine will be destroyed by the bootstrap cleanup process, after which operator provisioned CCM will take care of worker `Nodes` initialization.
+Our operator will take precedence for CNI operator. It will tolerate `NotReady` `NoSchedule` taint and `CCM` specific `Uninitialized` taint. Operator would start as the first operator in the cluster when first `control-plane` is created, and be responsible for initializing `Nodes` which will allow latter operators to start.
 
 #### Metrics
 
@@ -661,6 +642,42 @@ Requirement to separate CCM from Kubelet and KCM complicates bootstrap process.
 1. Continue development and support for in-tree cloud providers after exclusion from upstream as a carry patch.
 2. Integrate external CCM with KCMO and proceed with support for new cloud providers this way, yet following described requirements for bootstrap and post install phases.
 
+### Alternative for bootstrap changes
+
+*Note: This is currently an alternative approach tested and confirmed functional on AWS and Azure cloud providers. Yet due to a more complicated design, it is here as an alternative to the current approach.*
+
+Once an out-of-tree provider is released (GA), the `CCM` will be created as a static pod on the bootstrap node, to ensure swift removal of the `node.cloudprovider.kubernetes.io/uninitialized` taint from any newly created `Nodes`. Later stages, including cluster upgrades will be managed by an operator, which will ensure stability of the configuration, and will run the `CCM` in a `Deployment`.
+Initial usage of the static pod is justified by the need for `CCM` to initialise `Nodes` before the `kube-scheduler` is able to schedule regular workloads (eg the operator for `CCM` managed by Deployment).
+
+A static pod deployed on the bootstrap node will only run the `cloud-node` controller. This controller in particular, manages `Node` taint removal and all other `Node` related operations in `CCM`. Other controllers are not needed during bootstrap, and so can be excluded.
+
+The bootstrap static pod for the cloud provider will be provisioned unconditionally on bootstrap nodes once the platform is supported by the `CCCMO` render implementation.
+The constraint for enabling the `cloud-node` controller only on bootstrap will make sure this pod won't do anything for cloud providers which have not yet fully transitioned to `CCM`s, or don't have the `TechPreview` `FeatureGate` enabled from startup.
+
+*Note: Render implementation used in bootstrap is currently a standard step for operators required to be a part of day 1 installation procedure. Our case is justified by reqirement to provision `control-plane` Nodes, and allow API server to be fully rolled-out there.*
+
+The procedure would include the following:
+
+1) `cluster-config-operator` render would fully populate `cloud-config` `ConfigMap` and store in a manifest on the bootstrap machine.
+2) `CCCMO` would generate `CCM` manifest in form of a static pod, and provide all nessesary dependencies, such as `cloud-config` or `cloud-credentials` to use. `CCM` would have only `cloud-node` controller enabled.
+3) All static pods created by render steps, including `CCM`, would be copied to static manifests folder for bootstrap `kubelet` to pick up.
+
+`CCCMO` will provide a `render` implementation, similar to other operators, to generate initial manifests that allow deployment of `CCM` as a static pod on the bootstrap node.
+This will be used with the `bootkube.sh` [script](https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template).
+
+At the initial stage of `CCM` installation, [installer](https://github.com/openshift/installer) creates the initial cloud-config `ConfigMap` and [cluster-config-operator](https://github.com/openshift/cluster-config-operator) could additionally populate (depending on platform) some values to the config.
+Here are static contents of `cloud.conf` for OpenStack, which are generated by the installer:
+
+Example for OpenStack:
+
+```txt
+[Global]
+secret-name = openstack-credentials
+secret-namespace = kube-system
+```
+
+Resources deployed on the bootstrap machine will be destroyed by the bootstrap cleanup process, after which operator provisioned CCM will take care of worker `Nodes` initialization.
+
 ## Infrastructure Needed
 
 Additional infrastructure for OpenStack and vSphere may be required to test how CCM works with self-signed certificates. Current CI doesn't allow this.
@@ -691,3 +708,4 @@ Mandatory operator repository:
 - [The Kubernetes Cloud Controller Manager](https://medium.com/@m.json/the-kubernetes-cloud-controller-manager-d440af0d2be5) article
 https://hackmd.io/00IoVWBiSVm8mMByxerTPA#
 - [CSI support](https://github.com/openshift/enhancements/blob/master/enhancements/storage/csi-driver-install.md#ocp-45-kubernetes-118)
+- [CNI ]