Merge pull request #14558 from JacobTanenbaum/CIDR-work

Automatic merge from submit-queue (batch tested with PRs 14558, 16544).

Support multiple CIDR addresses for the pod SDN

Work in progress PR for multiple CIDR address work. addresses can be defined in the master config file by 


```
networkConfig:
  clusterNetworkCIDR: ""
  clusterNetworkConfig:
  - clusterNetworkCIDR: 10.128.0.0/28
  - clusterNetworkCIDR: 12.128.0.0/28
  externalIPNetworkCIDRs: null

```

or by passing a comma seporated list of them on the command line using --network-cidr="10.128.0.0/28,12.128.0.0/28"


besides general review - I could use some feedback on how to do a few things 

- [ ] How should I deal with a config file that defined ClusterNetworkCIDR the old way?

- [ ] In pkg/sdn/plugin/master.go there where checks to see if the cluster cidr was shrunk, I don't think that check is still required. It should be valid to break a large cluster cidr into it's smaller components. Is checking that all objects are allocated in defined places enough? As done by checkclusterobjects() in pkg/sdn/plugin/common.go

- [x] I need to change oc get clusternetwork to show a comma separated list 

- [ ] finish the unit testing


review of the work so far and input on the above questions are appreciated 
@knobunc @dcbw @danwinship @rajatchopra

api/docs/apis-network.openshift.io/v1.ClusterNetwork.adoc

@@ -19,7 +19,10 @@ Expand or mouse-over a field for more information about it.
 ++++
 <pre>
 <div style="margin-left:13px;"><span title="(string) APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources">apiVersion</span>:
-</div><div style="margin-left:13px;"><span title="(integer) HostSubnetLength is the number of bits of network to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods">hostsubnetlength</span>:
+</div><details><summary><span title="(array) ClusterNetworks is a list of ClusterNetwork objects that defines the global overlay network&#39;s L3 space by specifying a set of CIDR and netmasks that the SDN can allocate addressed from.">clusterNetworks</span>:
+</summary><div style="margin-left:13px;">- <span title="(string) CIDR defines the total range of a cluster networks address space.">CIDR</span>:
+</div><div style="margin-left:13px;">  <span title="(integer) HostSubnetLength is the number of bits of the accompanying CIDR address to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods.">hostSubnetLength</span>:
+</div></details><div style="margin-left:13px;"><span title="(integer) HostSubnetLength is the number of bits of network to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods">hostsubnetlength</span>:
 </div><div style="margin-left:13px;"><span title="(string) Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds">kind</span>:
 </div><details open><summary><span title="(v1.ObjectMeta) Standard object&#39;s metadata.">metadata</span>:
 </summary><details><summary>  <span title="(object) Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations">annotations</span>:

api/docs/oapi/v1.ClusterNetwork.adoc

@@ -19,7 +19,10 @@ Expand or mouse-over a field for more information about it.
 ++++
 <pre>
 <div style="margin-left:13px;"><span title="(string) APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources">apiVersion</span>:
-</div><div style="margin-left:13px;"><span title="(integer) HostSubnetLength is the number of bits of network to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods">hostsubnetlength</span>:
+</div><details><summary><span title="(array) ClusterNetworks is a list of ClusterNetwork objects that defines the global overlay network&#39;s L3 space by specifying a set of CIDR and netmasks that the SDN can allocate addressed from.">clusterNetworks</span>:
+</summary><div style="margin-left:13px;">- <span title="(string) CIDR defines the total range of a cluster networks address space.">CIDR</span>:
+</div><div style="margin-left:13px;">  <span title="(integer) HostSubnetLength is the number of bits of the accompanying CIDR address to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods.">hostSubnetLength</span>:
+</div></details><div style="margin-left:13px;"><span title="(integer) HostSubnetLength is the number of bits of network to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods">hostsubnetlength</span>:
 </div><div style="margin-left:13px;"><span title="(string) Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds">kind</span>:
 </div><details open><summary><span title="(v1.ObjectMeta) Standard object&#39;s metadata.">metadata</span>:
 </summary><details><summary>  <span title="(object) Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations">annotations</span>:

api/swagger-spec/oapi-v1.json

@@ -22781,9 +22781,8 @@
     "id": "v1.ClusterNetwork",
     "description": "ClusterNetwork describes the cluster network. There is normally only one object of this type, named \"default\", which is created by the SDN network plugin based on the master configuration when the cluster is brought up for the first time.",
     "required": [
-     "network",
-     "hostsubnetlength",
-     "serviceNetwork"
+     "serviceNetwork",
+     "clusterNetworks"
     ],
     "properties": {
      "kind": {
@@ -22813,6 +22812,31 @@
      "pluginName": {
       "type": "string",
       "description": "PluginName is the name of the network plugin being used"
+     },
+     "clusterNetworks": {
+      "type": "array",
+      "items": {
+       "$ref": "v1.ClusterNetworkEntry"
+      },
+      "description": "ClusterNetworks is a list of ClusterNetwork objects that defines the global overlay network's L3 space by specifying a set of CIDR and netmasks that the SDN can allocate addressed from."
+     }
+    }
+   },
+   "v1.ClusterNetworkEntry": {
+    "id": "v1.ClusterNetworkEntry",
+    "description": "ClusterNetworkEntry defines an individual cluster network. The CIDRs cannot overlap with other cluster network CIDRs, CIDRs reserved for external ips, CIDRs reserved for service networks, and CIDRs reserved for ingress ips.",
+    "required": [
+     "CIDR",
+     "hostSubnetLength"
+    ],
+    "properties": {
+     "CIDR": {
+      "type": "string",
+      "description": "CIDR defines the total range of a cluster networks address space."
+     },
+     "hostSubnetLength": {
+      "type": "integer",
+      "description": "HostSubnetLength is the number of bits of the accompanying CIDR address to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods."
      }
     }
    },

api/swagger-spec/openshift-openapi-spec.json

@@ -90742,15 +90742,21 @@
    "com.github.openshift.origin.pkg.network.apis.network.v1.ClusterNetwork": {
     "description": "ClusterNetwork describes the cluster network. There is normally only one object of this type, named \"default\", which is created by the SDN network plugin based on the master configuration when the cluster is brought up for the first time.",
     "required": [
-     "network",
-     "hostsubnetlength",
-     "serviceNetwork"
+     "serviceNetwork",
+     "clusterNetworks"
     ],
     "properties": {
      "apiVersion": {
       "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources",
       "type": "string"
      },
+     "clusterNetworks": {
+      "description": "ClusterNetworks is a list of ClusterNetwork objects that defines the global overlay network's L3 space by specifying a set of CIDR and netmasks that the SDN can allocate addressed from.",
+      "type": "array",
+      "items": {
+       "$ref": "#/definitions/com.github.openshift.origin.pkg.network.apis.network.v1.ClusterNetworkEntry"
+      }
+     },
      "hostsubnetlength": {
       "description": "HostSubnetLength is the number of bits of network to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods",
       "type": "integer",
@@ -90790,6 +90796,24 @@
      }
     ]
    },
+   "com.github.openshift.origin.pkg.network.apis.network.v1.ClusterNetworkEntry": {
+    "description": "ClusterNetworkEntry defines an individual cluster network. The CIDRs cannot overlap with other cluster network CIDRs, CIDRs reserved for external ips, CIDRs reserved for service networks, and CIDRs reserved for ingress ips.",
+    "required": [
+     "CIDR",
+     "hostSubnetLength"
+    ],
+    "properties": {
+     "CIDR": {
+      "description": "CIDR defines the total range of a cluster networks address space.",
+      "type": "string"
+     },
+     "hostSubnetLength": {
+      "description": "HostSubnetLength is the number of bits of the accompanying CIDR address to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pods.",
+      "type": "integer",
+      "format": "int64"
+     }
+    }
+   },
    "com.github.openshift.origin.pkg.network.apis.network.v1.ClusterNetworkList": {
     "description": "ClusterNetworkList is a collection of ClusterNetworks",
     "required": [

pkg/cmd/server/api/serialization_test.go

@@ -112,13 +112,33 @@ func fuzzInternalObject(t *testing.T, forVersion schema.GroupVersion, item runti
 					obj.NetworkConfig.ServiceNetworkCIDR = "10.0.0.0/24"
 				}
 			}
+			if len(obj.NetworkConfig.ClusterNetworks) == 0 {
+				clusterNetwork := []configapi.ClusterNetworkEntry{
+					{
+						CIDR:             "10.128.0.0/14",
+						HostSubnetLength: 9,
+					},
+				}
+				obj.NetworkConfig.ClusterNetworks = clusterNetwork
+			}
 
 			// TODO stop duplicating the conversion in the test.
 			kubeConfig := obj.KubernetesMasterConfig
 			noCloudProvider := kubeConfig != nil && (len(kubeConfig.ControllerArguments["cloud-provider"]) == 0 || kubeConfig.ControllerArguments["cloud-provider"][0] == "")
 			if noCloudProvider && len(obj.NetworkConfig.IngressIPNetworkCIDR) == 0 {
 				cidr := configapi.DefaultIngressIPNetworkCIDR
-				if !(configapi.CIDRsOverlap(cidr, obj.NetworkConfig.ClusterNetworkCIDR) || configapi.CIDRsOverlap(cidr, obj.NetworkConfig.ServiceNetworkCIDR)) {
+				setCIDR := true
+				if configapi.CIDRsOverlap(cidr, obj.NetworkConfig.ServiceNetworkCIDR) {
+					setCIDR = false
+				} else {
+					for _, clusterNetwork := range obj.NetworkConfig.ClusterNetworks {
+						if configapi.CIDRsOverlap(cidr, clusterNetwork.CIDR) {
+							setCIDR = false
+							break
+						}
+					}
+				}
+				if setCIDR {
 					obj.NetworkConfig.IngressIPNetworkCIDR = cidr
 				}
 			}

pkg/cmd/server/api/types.go

@@ -667,10 +667,12 @@ type UserAgentDenyRule struct {
 
 // MasterNetworkConfig to be passed to the compiled in network plugin
 type MasterNetworkConfig struct {
-	NetworkPluginName  string
-	ClusterNetworkCIDR string
-	HostSubnetLength   uint32
-	ServiceNetworkCIDR string
+	NetworkPluginName            string
+	DeprecatedClusterNetworkCIDR string
+	// ClusterNetworks contains a list of cluster networks that defines the global overlay networks L3 space.
+	ClusterNetworks            []ClusterNetworkEntry
+	DeprecatedHostSubnetLength uint32
+	ServiceNetworkCIDR         string
 	// ExternalIPNetworkCIDRs controls what values are acceptable for the service external IP field. If empty, no externalIP
 	// may be set. It may contain a list of CIDRs which are checked for access. If a CIDR is prefixed with !, IPs in that
 	// CIDR will be rejected. Rejections will be applied first, then the IP checked against one of the allowed CIDRs. You
@@ -683,6 +685,15 @@ type MasterNetworkConfig struct {
 	IngressIPNetworkCIDR string
 }
 
+// ClusterNetworkEntry defines an individual cluster network. The CIDRs cannot overlap with other cluster network CIDRs, CIDRs
+// reserved for external ips, CIDRs reserved for service networks, and CIDRs reserved for ingress ips.
+type ClusterNetworkEntry struct {
+	// CIDR defines the total range of a cluster networks address space.
+	CIDR string
+	// HostSubnetLength gives the number of address bits reserved for pod IPs on each node.
+	HostSubnetLength uint32
+}
+
 type ImageConfig struct {
 	// Format describes how to determine image names for system components
 	Format string

pkg/cmd/server/api/v1/conversions.go

@@ -90,14 +90,31 @@ func SetDefaults_MasterConfig(obj *MasterConfig) {
 			obj.NetworkConfig.ServiceNetworkCIDR = "10.0.0.0/24"
 		}
 	}
+	if len(obj.NetworkConfig.ClusterNetworks) == 0 {
+		obj.NetworkConfig.ClusterNetworks = []ClusterNetworkEntry{{CIDR: obj.NetworkConfig.DeprecatedClusterNetworkCIDR, HostSubnetLength: obj.NetworkConfig.DeprecatedHostSubnetLength}}
+
+		obj.NetworkConfig.DeprecatedClusterNetworkCIDR = ""
+		obj.NetworkConfig.DeprecatedHostSubnetLength = 0
+	}
 
 	// TODO Detect cloud provider when not using built-in kubernetes
 	kubeConfig := obj.KubernetesMasterConfig
 	noCloudProvider := kubeConfig != nil && (len(kubeConfig.ControllerArguments["cloud-provider"]) == 0 || kubeConfig.ControllerArguments["cloud-provider"][0] == "")
 
 	if noCloudProvider && len(obj.NetworkConfig.IngressIPNetworkCIDR) == 0 {
 		cidr := internal.DefaultIngressIPNetworkCIDR
-		if !(internal.CIDRsOverlap(cidr, obj.NetworkConfig.ClusterNetworkCIDR) || internal.CIDRsOverlap(cidr, obj.NetworkConfig.ServiceNetworkCIDR)) {
+		cidrOverlap := false
+		if internal.CIDRsOverlap(cidr, obj.NetworkConfig.ServiceNetworkCIDR) {
+			cidrOverlap = true
+		} else {
+			for _, entry := range obj.NetworkConfig.ClusterNetworks {
+				if internal.CIDRsOverlap(cidr, entry.CIDR) {
+					cidrOverlap = true
+					break
+				}
+			}
+		}
+		if !cidrOverlap {
 			obj.NetworkConfig.IngressIPNetworkCIDR = cidr
 		}
 	}

pkg/cmd/server/api/v1/swagger_doc.go

@@ -145,6 +145,16 @@ func (ClientConnectionOverrides) SwaggerDoc() map[string]string {
 	return map_ClientConnectionOverrides
 }
 
+var map_ClusterNetworkEntry = map[string]string{
+	"":                 "ClusterNetworkEntry defines an individual cluster network. The CIDRs cannot overlap with other cluster network CIDRs, CIDRs reserved for external ips, CIDRs reserved for service networks, and CIDRs reserved for ingress ips.",
+	"cidr":             "CIDR defines the total range of a cluster networks address space.",
+	"hostSubnetLength": "HostSubnetLength is the number of bits of the accompanying CIDR address to allocate to each node. eg, 8 would mean that each node would have a /24 slice of the overlay network for its pod.",
+}
+
+func (ClusterNetworkEntry) SwaggerDoc() map[string]string {
+	return map_ClusterNetworkEntry
+}
+
 var map_ControllerConfig = map[string]string{
 	"":                   "ControllerConfig holds configuration values for controllers",
 	"election":           "Election defines the configuration for electing a controller instance to make changes to the cluster. If unspecified, the ControllerTTL value is checked to determine whether the legacy direct etcd election code will be used.",
@@ -536,8 +546,9 @@ func (MasterConfig) SwaggerDoc() map[string]string {
 var map_MasterNetworkConfig = map[string]string{
 	"":                       "MasterNetworkConfig to be passed to the compiled in network plugin",
 	"networkPluginName":      "NetworkPluginName is the name of the network plugin to use",
-	"clusterNetworkCIDR":     "ClusterNetworkCIDR is the CIDR string to specify the global overlay network's L3 space",
-	"hostSubnetLength":       "HostSubnetLength is the number of bits to allocate to each host's subnet e.g. 8 would mean a /24 network on the host",
+	"clusterNetworkCIDR":     "ClusterNetworkCIDR is the CIDR string to specify the global overlay network's L3 space.  Deprecated, but maintained for backwards compatibility, use ClusterNetworks instead.",
+	"clusterNetworks":        "ClusterNetworks is a list of ClusterNetwork objects that defines the global overlay network's L3 space by specifying a set of CIDR and netmasks that the SDN can allocate addressed from.  If this is specified, then ClusterNetworkCIDR and HostSubnetLength may not be set.",
+	"hostSubnetLength":       "HostSubnetLength is the number of bits to allocate to each host's subnet e.g. 8 would mean a /24 network on the host.  Deprecated, but maintained for backwards compatibility, use ClusterNetworks instead.",
 	"serviceNetworkCIDR":     "ServiceNetwork is the CIDR string to specify the service networks",
 	"externalIPNetworkCIDRs": "ExternalIPNetworkCIDRs controls what values are acceptable for the service external IP field. If empty, no externalIP may be set. It may contain a list of CIDRs which are checked for access. If a CIDR is prefixed with !, IPs in that CIDR will be rejected. Rejections will be applied first, then the IP checked against one of the allowed CIDRs. You should ensure this range does not overlap with your nodes, pods, or service CIDRs for security reasons.",
 	"ingressIPNetworkCIDR":   "IngressIPNetworkCIDR controls the range to assign ingress ips from for services of type LoadBalancer on bare metal. If empty, ingress ips will not be assigned. It may contain a single CIDR that will be allocated from. For security reasons, you should ensure that this range does not overlap with the CIDRs reserved for external ips, nodes, pods, or services.",