IPPool CRD NV-IPAM

NV-IPAM updates: - Add support for IPPool CRD - Deprecate the configmap - Update templates with latest yamls configuration - Update documentation - Upgrade to v0.1.0 Signed-off-by: Fred Rolland <frolland@nvidia.com>
rollandf · Sep 11, 2023 · 5e58e3e · 5e58e3e
1 parent 1804ba4
commit 5e58e3e
Show file tree

Hide file tree

Showing 17 changed files with 281 additions and 157 deletions.
diff --git a/README.md b/README.md
@@ -219,12 +219,7 @@ spec:
   nvIpam:
     image: nvidia-k8s-ipam
     repository: ghcr.io/mellanox
-    version: v0.0.3
-    config: '{
-      "pools":  {
-        "my-pool": {"subnet": "192.168.0.0/24", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}
-      }
-    }'
+    version: v0.1.0
 ```
 
 Can be found at: `example/crs/mellanox.com_v1alpha1_nicclusterpolicy_cr-nvidia-ipam.yaml`
@@ -493,83 +488,6 @@ Specifying configuration either via Helm values when installing NVIDIA
 network operator, or by specifying them when directly creating NicClusterPolicy CR.
 These configurations eventually trigger the creation of a ConfigMap object in K8s.
 
-As an example, NVIDIA K8s IPAM plugin configuration is specified either via:
-
-__Helm values:__
-
-```yaml
-deployCR: true
-nvIpam:
-  deploy: true
-  image: nvidia-k8s-ipam
-  repository: ghcr.io/mellanox
-  version: v0.0.3
-  config: |-
-    {
-    "pools":  {
-      "rdma-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}
-      }
-    }
-```
-
-__NicClusterPolicy CR:__
-
-```yaml
-apiVersion: mellanox.com/v1alpha1
-kind: NicClusterPolicy
-metadata:
-  name: nic-cluster-policy
-spec:
-  nvIpam:
-    image: nvidia-k8s-ipam
-    repository: ghcr.io/mellanox
-    version: v0.0.3
-    config: '{
-      "pools":  {
-        "my-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}
-      }
-    }'
-```
-
-The configuration is then processed by the operator, eventually rendering and creating a _ConfigMap_, `nvidia-k8s-ipam-config`, within the
-namespace the operator was deployed. It contains the configuration for _nvidia k8s IPAM plugin_.
-
-For some advanced use-cases, it is desirable to provide such configurations at a later time.
-(e.g if network configuration is not known during Network Operator deployment time)
-
-To support this, it is possible to explicitly set such configuration to `nil` in Helm values
-or omit the `config` field of the relevant component while creating NicClusterPolicy CR.
-This will prevent Network Operator from
-creating such ConfigMaps, allowing the user to provide its own.
-
-Example (omitting nvidia k8s ipam config):
-
-__Helm values:__
-
-```yaml
-deployCR: true
-nvIpam:
-  deploy: true
-  image: nvidia-k8s-ipam
-  repository: ghcr.io/mellanox
-  version: v0.0.3
-  config: null
-```
-
-__NicClusterPolicy CR:__
-
-```yaml
-apiVersion: mellanox.com/v1alpha1
-kind: NicClusterPolicy
-metadata:
-  name: nic-cluster-policy
-spec:
-  nvIpam:
-    image: nvidia-k8s-ipam
-    repository: ghcr.io/mellanox
-    version: v0.0.3
-```
-
 > __Note__: It is the responsibility of the user to delete any existing configurations (ConfigMaps) if
 > they were already created by the Network Operator as well as deleting his own configuration when they
 > are no longer required.
diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml
@@ -248,6 +248,23 @@ rules:
   - get
   - list
   - watch
+- apiGroups:
+  - nv-ipam.nvidia.com
+  resources:
+  - ippools
+  verbs:
+  - create
+  - get
+  - list
+  - watch
+- apiGroups:
+  - nv-ipam.nvidia.com
+  resources:
+  - ippools/status
+  verbs:
+  - get
+  - patch
+  - update
 - apiGroups:
   - policy
   resources:

diff --git a/controllers/nicclusterpolicy_controller.go b/controllers/nicclusterpolicy_controller.go
@@ -75,6 +75,8 @@ type NicClusterPolicyReconciler struct {
 // +kubebuilder:rbac:groups=coordination.k8s.io,resources=leases,verbs=get;list;watch;create;update;patch;delete
 // +kubebuilder:rbac:groups=batch,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete
 // +kubebuilder:rbac:groups=config.openshift.io,resources=proxies,verbs=get;list;watch
+// +kubebuilder:rbac:groups=nv-ipam.nvidia.com,resources=ippools,verbs=get;list;watch;create;
+// +kubebuilder:rbac:groups=nv-ipam.nvidia.com,resources=ippools/status,verbs=get;update;patch;
 
 // Reconcile is part of the main kubernetes reconciliation loop which aims to
 // move the current state of the cluster closer to the desired state.

diff --git a/deployment/network-operator/README.md b/deployment/network-operator/README.md
@@ -561,9 +561,9 @@ optionally deployed components:
 | `nvIpam.deploy`           | bool   | `false`            | Deploy NVIDIA IPAM Plugin                                                                |
 | `nvIpam.image`            | string | `nvidia-k8s-ipam`  | NVIDIA IPAM Plugin image name                                                            |
 | `nvIpam.repository`       | string | `ghcr.io/mellanox` | NVIDIA IPAM Plugin image repository                                                      |
-| `nvIpam.version`          | string | `v0.0.3`           | NVIDIA IPAM Plugin image version                                                         |
+| `nvIpam.version`          | string | `v0.1.0`           | NVIDIA IPAM Plugin image version                                                         |
 | `nvIpam.imagePullSecrets` | list   | `[]`               | An optional list of references to secrets to use for pulling any of the Plugin image     |
-| `nvIpam.config`           | string | `"{"pools": {"rdma-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}}}"` | Network pool configuration as described in [nvidia-k8s-ipam](https://github.com/Mellanox/nvidia-k8s-ipam), the default defines a single IP Pool named `"rdma-pool"`|
+| `nvIpam.config`           | string | Deprecated         | This field is ignored. Configuration is done by using IPPool CRD                                                             |
 
 ## Deployment Examples
 

diff --git a/deployment/network-operator/crds/nv-ipam.nvidia.com_ippools.yaml b/deployment/network-operator/crds/nv-ipam.nvidia.com_ippools.yaml
@@ -0,0 +1,175 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.13.0
+  name: ippools.nv-ipam.nvidia.com
+spec:
+  group: nv-ipam.nvidia.com
+  names:
+    kind: IPPool
+    listKind: IPPoolList
+    plural: ippools
+    singular: ippool
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.subnet
+      name: Subnet
+      type: string
+    - jsonPath: .spec.gateway
+      name: Gateway
+      type: string
+    - jsonPath: .spec.perNodeBlockSize
+      name: Block Size
+      type: integer
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: IPPool contains configuration for IPAM controller
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: IPPoolSpec contains configuration for IP pool
+            properties:
+              gateway:
+                description: gateway for the pool
+                type: string
+              nodeSelector:
+                description: selector for nodes, if empty match all nodes
+                properties:
+                  nodeSelectorTerms:
+                    description: Required. A list of node selector terms. The terms
+                      are ORed.
+                    items:
+                      description: A null or empty node selector term matches no objects.
+                        The requirements of them are ANDed. The TopologySelectorTerm
+                        type implements a subset of the NodeSelectorTerm.
+                      properties:
+                        matchExpressions:
+                          description: A list of node selector requirements by node's
+                            labels.
+                          items:
+                            description: A node selector requirement is a selector
+                              that contains values, a key, and an operator that relates
+                              the key and values.
+                            properties:
+                              key:
+                                description: The label key that the selector applies
+                                  to.
+                                type: string
+                              operator:
+                                description: Represents a key's relationship to a
+                                  set of values. Valid operators are In, NotIn, Exists,
+                                  DoesNotExist. Gt, and Lt.
+                                type: string
+                              values:
+                                description: An array of string values. If the operator
+                                  is In or NotIn, the values array must be non-empty.
+                                  If the operator is Exists or DoesNotExist, the values
+                                  array must be empty. If the operator is Gt or Lt,
+                                  the values array must have a single element, which
+                                  will be interpreted as an integer. This array is
+                                  replaced during a strategic merge patch.
+                                items:
+                                  type: string
+                                type: array
+                            required:
+                            - key
+                            - operator
+                            type: object
+                          type: array
+                        matchFields:
+                          description: A list of node selector requirements by node's
+                            fields.
+                          items:
+                            description: A node selector requirement is a selector
+                              that contains values, a key, and an operator that relates
+                              the key and values.
+                            properties:
+                              key:
+                                description: The label key that the selector applies
+                                  to.
+                                type: string
+                              operator:
+                                description: Represents a key's relationship to a
+                                  set of values. Valid operators are In, NotIn, Exists,
+                                  DoesNotExist. Gt, and Lt.
+                                type: string
+                              values:
+                                description: An array of string values. If the operator
+                                  is In or NotIn, the values array must be non-empty.
+                                  If the operator is Exists or DoesNotExist, the values
+                                  array must be empty. If the operator is Gt or Lt,
+                                  the values array must have a single element, which
+                                  will be interpreted as an integer. This array is
+                                  replaced during a strategic merge patch.
+                                items:
+                                  type: string
+                                type: array
+                            required:
+                            - key
+                            - operator
+                            type: object
+                          type: array
+                      type: object
+                      x-kubernetes-map-type: atomic
+                    type: array
+                required:
+                - nodeSelectorTerms
+                type: object
+                x-kubernetes-map-type: atomic
+              perNodeBlockSize:
+                description: amount of IPs to allocate for each node, must be less
+                  than amount of available IPs in the subnet
+                type: integer
+              subnet:
+                description: subnet of the pool
+                type: string
+            required:
+            - gateway
+            - perNodeBlockSize
+            - subnet
+            type: object
+          status:
+            description: IPPoolStatus contains the IP ranges allocated to nodes
+            properties:
+              allocations:
+                description: IP allocations for Nodes
+                items:
+                  description: Allocation contains IP Allocation for a specific Node
+                  properties:
+                    endIP:
+                      type: string
+                    nodeName:
+                      type: string
+                    startIP:
+                      type: string
+                  required:
+                  - endIP
+                  - nodeName
+                  - startIP
+                  type: object
+                type: array
+            required:
+            - allocations
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
diff --git a/deployment/network-operator/templates/mellanox.com_v1alpha1_nicclusterpolicy_cr.yaml b/deployment/network-operator/templates/mellanox.com_v1alpha1_nicclusterpolicy_cr.yaml
@@ -176,8 +176,5 @@ spec:
     repository: {{ .Values.nvIpam.repository }}
     version: {{ .Values.nvIpam.version }}
     imagePullSecrets: {{ include "network-operator.nvIpam.imagePullSecrets" . }}
-    {{- if .Values.nvIpam.config | empty | not }}
-    config: {{ .Values.nvIpam.config | quote }}
-    {{- end }}
   {{- end }}
 {{ end }}
diff --git a/deployment/network-operator/templates/role.yaml b/deployment/network-operator/templates/role.yaml
@@ -352,6 +352,23 @@ rules:
   - get
   - list
   - watch
+- apiGroups:
+  - nv-ipam.nvidia.com
+  resources:
+  - ippools
+  verbs:
+  - create
+  - get
+  - list
+  - watch
+- apiGroups:
+  - nv-ipam.nvidia.com
+  resources:
+  - ippools/status
+  verbs:
+  - get
+  - patch
+  - update
 - apiGroups:
   - policy
   resources:

diff --git a/deployment/network-operator/values.yaml b/deployment/network-operator/values.yaml
@@ -235,15 +235,8 @@ nvIpam:
   deploy: false
   image: nvidia-k8s-ipam
   repository: ghcr.io/mellanox
-  version: v0.0.3
+  version: v0.1.0
   # imagePullSecrets: []
-  # network pool configuration as described in https://github.com/Mellanox/nvidia-k8s-ipam
-  config: |-
-    {
-    "pools":  {
-      "rdma-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}
-      }
-    }
 
 secondaryNetwork:
   deploy: true

diff --git a/example/crs/mellanox.com_v1alpha1_nicclusterpolicy_cr-nvidia-ipam.yaml b/example/crs/mellanox.com_v1alpha1_nicclusterpolicy_cr-nvidia-ipam.yaml
@@ -61,9 +61,4 @@ spec:
   nvIpam:
     image: nvidia-k8s-ipam
     repository: ghcr.io/mellanox
-    version: v0.0.3
-    config: '{
-      "pools":  {
-        "my-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}
-      }
-    }'
+    version: v0.1.0
diff --git a/hack/release.yaml b/hack/release.yaml
@@ -52,4 +52,4 @@ IpamPlugin:
 nvIpam:
   image: nvidia-k8s-ipam
   repository: ghcr.io/mellanox
-  version: v0.0.3
+  version: v0.1.0
diff --git a/hack/templates/crs/mellanox.com_v1alpha1_nicclusterpolicy_cr-nvidia-ipam.template b/hack/templates/crs/mellanox.com_v1alpha1_nicclusterpolicy_cr-nvidia-ipam.template
@@ -62,8 +62,3 @@ spec:
     image: {{ .NvIPAM.Image }}
     repository: {{ .NvIPAM.Repository }}
     version: {{ .NvIPAM.Version }}
-    config: '{
-      "pools":  {
-        "my-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}
-      }
-    }'