Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NVIPAMPool CRD #24

Merged
merged 5 commits into from
Sep 12, 2023
Merged

NVIPAMPool CRD #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9a0db98
NVIPAMPool CRD
rollandf Jul 24, 2023
2847204
IPPool controller
rollandf Aug 6, 2023
a95a67b
Node: watch IPPool CR
rollandf Aug 14, 2023
d6513cd
Migration from cm to CRs
rollandf Aug 15, 2023
251cb7e
Update README for IPPool CRD
rollandf Aug 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 14 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ PROTOC ?= $(LOCALBIN)/protoc/bin/protoc
PROTOC_GEN_GO ?= $(LOCALBIN)/protoc-gen-go
PROTOC_GEN_GO_GRPC ?= $(LOCALBIN)/protoc-gen-go-grpc
BUF ?= $(LOCALBIN)/buf
CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen

## Tool Versions
GOLANGCILINT_VERSION ?= v1.52.2
Expand All @@ -152,7 +153,7 @@ PROTOC_VER ?= 23.4
PROTOC_GEN_GO_VER ?= 1.31.0
PROTOC_GEN_GO_GRPC_VER ?= 1.3.0
BUF_VERSION ?= 1.23.1

CONTROLLER_GEN_VERSION ?= v0.13.0

.PHONY: envtest
envtest: $(ENVTEST) ## Download envtest-setup locally if necessary.
Expand Down Expand Up @@ -251,3 +252,15 @@ grpc-format: buf ## Format GRPC files
@echo "format protobuf files";
cd $(PROTO_DIR) && \
$(BUF) format -w --exit-code
.PHONY: controller-gen
controller-gen: $(CONTROLLER_GEN) ## Download controller-gen locally if necessary
$(CONTROLLER_GEN): | $(LOCALBIN)
GOBIN=$(LOCALBIN) go install sigs.k8s.io/controller-tools/cmd/controller-gen@$(CONTROLLER_GEN_VERSION)

.PHONY: generate
generate: controller-gen
$(CONTROLLER_GEN) object:headerFile="./boilerplate.go.txt" paths="./api/..."

.PHONY: manifests
manifests: controller-gen
$(CONTROLLER_GEN) crd paths="./api/..." output:dir="./deploy/crds"
160 changes: 107 additions & 53 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,8 @@ NVIDIA IPAM plugin consists of 3 main components:

### ipam-controller

A Kubernetes(K8s) controller that Watches on a predefined K8s ConfigMap for defined IP Pools.
It then proceeds in reconciling K8s Node objects by assiging each node via `ipam.nvidia.com/ip-blocks`
annotation a cluster unique range of IPs of the defined IP Pools.
A Kubernetes(K8s) controller that Watches on IPPools CRs in a predefined Namespace.
It then proceeds by assiging each node via IPPools Status a cluster unique range of IPs of the defined IP Pools.

### ipam-node

Expand All @@ -43,7 +42,7 @@ The daemon is responsible for:
A node daemon provides GRPC service, which nv-ipam CNI plugin uses to request IP address allocation/deallocation.
IPs are allocated from the provided IP Block assigned by ipam-controller for the node.
To determine the cluster unique IP Block for the defined IP Pool, ipam-node watches K8s API
for the Node object and extracts IP Block information from node annotation.
for the IPPool objects and extracts IP Block information from IPPool Status.

### nv-ipam

Expand All @@ -53,30 +52,65 @@ To allocate/deallocate IP address nv-ipam calls GRPC API of ipam-node daemon.
### IP allocation flow

1. User (cluster administrator) defines a set of named IP Pools to be used for IP allocation
of container interfaces via Kubernetes ConfigMap (more information in [Configuration](#configuration) section)
of container interfaces via IPPool CRD (more information in [Configuration](#configuration) section)

_Example_:

```json
{
"pools": {
"my-pool": {"subnet": "192.188.0.0/16", "perNodeBlockSize": 24, "gateway": "192.168.0.1"}
},
"nodeSelector": {
"kubernetes.io/os": "linux"
}
}
```yaml
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: IPPool
metadata:
name: pool1
namespace: kube-system
spec:
subnet: 192.168.0.0/16
perNodeBlockSize: 24
gateway: 192.168.0.1
nodeSelector:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/role
operator: In
values:
- worker
```

2. ipam-controller calculates and assigns unique IP Blocks for each Node via annotation
2. ipam-controller calculates and assigns unique IP Blocks for each Node via IPPool Status:

_Example_:

```yaml
annotations:
ipam.nvidia.com/ip-blocks: '{
"my-pool": {"startIP": "192.168.0.2", "endIP": "192.168.0.25", "gateway": "192.168.0.1", "subnet": "192.168.0.0/16"}
}'
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: IPPool
metadata:
name: pool1
namespace: kube-system
spec:
gateway: 192.168.0.1
nodeSelector:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/role
operator: In
values:
- worker
perNodeBlockSize: 24
subnet: 192.168.0.0/16
status:
allocations:
- endIP: 192.168.0.24
nodeName: host-a
startIP: 192.168.0.1
- endIP: 192.168.0.48
nodeName: host-b
startIP: 192.168.0.25
- endIP: 192.168.0.72
nodeName: host-c
startIP: 192.168.0.49
- endIP: 192.168.0.96
nodeName: k8s-master
startIP: 192.168.0.73

```

3. User specifies nv-ipam as IPAM plugin in CNI configuration
Expand All @@ -103,7 +137,7 @@ corresponding IP Pool that was allocated for the node

### ipam-controller configuration

ipam-controller accepts configuration using command line flags and K8s configMap
ipam-controller accepts configuration using command line flags and IPPools CRs.

#### Flags

Expand Down Expand Up @@ -140,10 +174,6 @@ Common flags:

Controller flags:

--config-name string
The name of the ConfigMap which holds controller configuration (default "nvidia-k8s-ipam-config")
--config-namespace string
The name of the namespace where ConfigMap with controller configuration exist (default "kube-system")
--health-probe-bind-address string
The address the probe endpoint binds to. (default ":8081")
--kubeconfig string
Expand All @@ -154,37 +184,43 @@ Controller flags:
Determines the namespace in which the leader election resource will be created. (default "kube-system")
--metrics-bind-address string
The address the metric endpoint binds to. (default ":8080")
--ippools-namespace string
The name of the namespace to watch for IPPools CRs. (default "kube-system")
```

#### ConfigMap
#### IPPool CR

ipam-controller accepts IP Pool configuration via configMap with a pre-defined key named `config`
ipam-controller accepts IP Pools configuration via IPPool CRs.
Multiple IPPool CRs can be created, with different NodeSelectors.

```yaml
apiVersion: v1
kind: ConfigMap
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: IPPool
metadata:
name: nvidia-k8s-ipam-config
name: my-pool
namespace: kube-system
data:
config: |
{
"pools": {
"my-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100 , "gateway": "192.168.0.1"}
},
"nodeSelector": {"kubernetes.io/os": "linux"}
}
spec:
subnet: 192.168.0.0/16
perNodeBlockSize: 100
gateway: 192.168.0.1
nodeSelector:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/role
operator: In
values:
- worker
```

* `pools`: contains a set of named IP Pools keyed by name
* `spec`: contains the IP pool configuration
* `subnet`: IP Subnet of the pool
* `gateway`: Gateway IP of the subnet
* `perNodeBlockSize`: the number of IPs of IP Blocks allocated to Nodes.
* `nodeSelector`: a map<string, string> of node selector labels, only nodes that match the provided labels will get assigned IP Blocks for the defined pools
* `nodeSelector`: A list of node selector terms. The terms are ORed. Each term can have a list of matchExpressions that are ANDed. Only the nodes that match the provided labels will get assigned IP Blocks for the defined pool.

> __Notes:__
>
> * pool name is composed of alphanumeric letters separated by dots(`.`) undersocres(`_`) or hyphens(`-`)
> * pool name is composed of alphanumeric letters separated by dots(`.`) underscores(`_`) or hyphens(`-`)
> * `perNodeBlockSize` minimum size is 2
> * `subnet` must be large enough to accommodate at least one `perNodeBlockSize` block of IPs

Expand Down Expand Up @@ -238,6 +274,8 @@ Node daemon flags:
The name of the Node on which the daemon runs
--store-file string
Path of the file which used to store allocations (default "/var/lib/cni/nv-ipam/store")
--ippools-namespace string
ykulazhenkov marked this conversation as resolved.
Show resolved Hide resolved
The name of the namespace to watch for IPPools CRs. (default "kube-system")

Shim CNI Configuration flags:

Expand Down Expand Up @@ -296,26 +334,30 @@ interface should have two IP addresses: one IPv4 and one IPv6. (default: network
> _NOTE:_ This command will deploy latest dev build with default configuration

```shell
kubectl apply -f https://raw.githubusercontent.com/Mellanox/nvidia-k8s-ipam/main/deploy/crds/nv-ipam.nvidia.com_ippools.yaml
kubectl apply -f https://raw.githubusercontent.com/Mellanox/nvidia-k8s-ipam/main/deploy/nv-ipam.yaml
```

### Create ipam-controller config
### Create IPPool CR

```shell
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: IPPool
metadata:
name: nvidia-k8s-ipam-config
name: my-pool
namespace: kube-system
data:
config: |
{
"pools": {
"my-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100 , "gateway": "192.168.0.1"}
},
"nodeSelector": {"kubernetes.io/os": "linux"}
}
spec:
subnet: 192.168.0.0/16
perNodeBlockSize: 100
gateway: 192.168.0.1
nodeSelector:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/role
operator: In
values:
- worker
EOF
```

Expand Down Expand Up @@ -345,7 +387,19 @@ EOF
View allocated IP Blocks:

```shell
kubectl get nodes -o=custom-columns='NAME:metadata.name,ANNOTATION:metadata.annotations.ipam\.nvidia\.com/ip-blocks'
kubectl get ippools.nv-ipam.nvidia.com -A -o jsonpath='{range .items[*]}{.metadata.name}{"\n"} {range .status.allocations[*]}{"\t"}{.nodeName} => Start IP: {.startIP} End IP: {.endIP}{"\n"}{end}{"\n"}{end}'

pool1
host-a => Start IP: 192.168.0.1 End IP: 192.168.0.24
host-b => Start IP: 192.168.0.25 End IP: 192.168.0.48
host-c => Start IP: 192.168.0.49 End IP: 192.168.0.72
k8s-master => Start IP: 192.168.0.73 End IP: 192.168.0.96

pool2
host-a => Start IP: 172.16.0.1 End IP: 172.16.0.50
host-b => Start IP: 172.16.0.51 End IP: 172.16.0.100
host-c => Start IP: 172.16.0.101 End IP: 172.16.0.150
k8s-master => Start IP: 172.16.0.151 End IP: 172.16.0.200
```

View network status of pods:
Expand Down
33 changes: 33 additions & 0 deletions api/v1alpha1/groupversion_info.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
Copyright 2023, NVIDIA CORPORATION & AFFILIATES
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

// +kubebuilder:object:generate=true
// +groupName=nv-ipam.nvidia.com

package v1alpha1
ykulazhenkov marked this conversation as resolved.
Show resolved Hide resolved

import (
"k8s.io/apimachinery/pkg/runtime/schema"
"sigs.k8s.io/controller-runtime/pkg/scheme"
)

var (
// GroupVersion is group version used to register these objects
GroupVersion = schema.GroupVersion{Group: "nv-ipam.nvidia.com", Version: "v1alpha1"}

// SchemeBuilder is used to add go types to the GroupVersionKind scheme
SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}

// AddToScheme adds the types in this group-version to the given scheme.
AddToScheme = SchemeBuilder.AddToScheme
)
72 changes: 72 additions & 0 deletions api/v1alpha1/ippool_type.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
/*
Copyright 2023, NVIDIA CORPORATION & AFFILIATES
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1alpha1

import (
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Subnet",type="string",JSONPath=`.spec.subnet`
// +kubebuilder:printcolumn:name="Gateway",type="string",JSONPath=`.spec.gateway`
// +kubebuilder:printcolumn:name="Block Size",type="integer",JSONPath=`.spec.perNodeBlockSize`
ykulazhenkov marked this conversation as resolved.
Show resolved Hide resolved

// IPPool contains configuration for IPAM controller
type IPPool struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec IPPoolSpec `json:"spec"`
Status IPPoolStatus `json:"status,omitempty"`
}

// IPPoolSpec contains configuration for IP pool
type IPPoolSpec struct {
// subnet of the pool
Subnet string `json:"subnet"`
// amount of IPs to allocate for each node,
// must be less than amount of available IPs in the subnet
PerNodeBlockSize int `json:"perNodeBlockSize"`
// gateway for the pool
Gateway string `json:"gateway"`
// selector for nodes, if empty match all nodes
NodeSelector *corev1.NodeSelector `json:"nodeSelector,omitempty"`
}

// IPPoolStatus contains the IP ranges allocated to nodes
type IPPoolStatus struct {
ykulazhenkov marked this conversation as resolved.
Show resolved Hide resolved
// IP allocations for Nodes
Allocations []Allocation `json:"allocations"`
}

// Allocation contains IP Allocation for a specific Node
type Allocation struct {
NodeName string `json:"nodeName"`
StartIP string `json:"startIP"`
EndIP string `json:"endIP"`
}

// +kubebuilder:object:root=true

// IPPoolList contains a list of IPPool
ykulazhenkov marked this conversation as resolved.
Show resolved Hide resolved
type IPPoolList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []IPPool `json:"items"`
}

func init() {
SchemeBuilder.Register(&IPPool{}, &IPPoolList{})
}