Skip to content

Commit

Permalink
Merge pull request #2466 from cyclinder/spideragent/cni_bin_init
Browse files Browse the repository at this point in the history 
spiderAgent: install cni,ovs and rdma in init-container
  • Loading branch information
@weizhoublue
weizhoublue authored Oct 28, 2023
2 parents 7fc60d5 + da274b2 commit d380f35
Show file tree
Hide file tree
Showing 15 changed files with 122 additions and 150 deletions.
65 changes: 36 additions & 29 deletions charts/spiderpool/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,35 +164,28 @@ helm install spiderpool spiderpool/spiderpool --wait --namespace kube-system \

### rdma parameters

| Name | Description | Value |
| ----------------------------------------------------------------- | ---------------------------------------------------------- | -------------------------------------- |
| `rdma.rdmaSharedDevicePlugin.install` | install rdma shared device plugin for macvlan cni | `false` |
| `rdma.rdmaSharedDevicePlugin.name` | the name of rdma shared device plugin | `spiderpool-rdma-shared-device-plugin` |
| `rdma.rdmaSharedDevicePlugin.image.registry` | the image registry of rdma shared device plugin | `ghcr.io` |
| `rdma.rdmaSharedDevicePlugin.image.repository` | the image repository of rdma shared device plugin | `mellanox/k8s-rdma-shared-dev-plugin` |
| `rdma.rdmaSharedDevicePlugin.image.pullPolicy` | the image pullPolicy of rdma shared device plugin | `IfNotPresent` |
| `rdma.rdmaSharedDevicePlugin.image.digest` | the image digest of rdma shared device plugin | `""` |
| `rdma.rdmaSharedDevicePlugin.image.tag` | the image tag of rdma shared device plugin | `latest` |
| `rdma.rdmaSharedDevicePlugin.image.imagePullSecrets` | the image imagePullSecrets of rdma shared device plugin | `[]` |
| `rdma.rdmaSharedDevicePlugin.podAnnotations` | the additional annotations | `{}` |
| `rdma.rdmaSharedDevicePlugin.podLabels` | the additional label | `{}` |
| `rdma.rdmaSharedDevicePlugin.resources.limits.cpu` | the cpu limit | `300m` |
| `rdma.rdmaSharedDevicePlugin.resources.limits.memory` | the memory limit | `300Mi` |
| `rdma.rdmaSharedDevicePlugin.resources.requests.cpu` | the cpu requests | `100m` |
| `rdma.rdmaSharedDevicePlugin.resources.requests.memory` | the memory requests | `50Mi` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.periodicUpdateInterval` | periodic Update Interval | `300` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.resourcePrefix` | resource prefix | `spidernet.io` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.resourceName` | resource Name | `hca_shared_devices` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.rdmaHcaMax` | rdma Hca Max | `500` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.vendors` | rdma device vendors, default to mellanox device | `15b3` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.deviceIDs` | rdma device IDs, default to mellanox device | `1017` |
| `rdma.rdmaCni.install` | install rdma cni used to isolate rdma device for sriov cni | `false` |
| `rdma.rdmaCni.image.registry` | the rdma-cni image registry | `ghcr.io` |
| `rdma.rdmaCni.image.repository` | the rdma-cni image repository | `k8snetworkplumbingwg/rdma-cni` |
| `rdma.rdmaCni.image.pullPolicy` | the rdma-cni image pullPolicy | `IfNotPresent` |
| `rdma.rdmaCni.image.digest` | the rdma-cni image digest | `""` |
| `rdma.rdmaCni.image.tag` | the rdma-cni image tag | `latest` |
| `rdma.rdmaCni.image.imagePullSecrets` | the rdma-cni image imagePullSecrets | `[]` |
| Name | Description | Value |
| ----------------------------------------------------------------- | ------------------------------------------------------- | -------------------------------------- |
| `rdma.rdmaSharedDevicePlugin.install` | install rdma shared device plugin for macvlan cni | `false` |
| `rdma.rdmaSharedDevicePlugin.name` | the name of rdma shared device plugin | `spiderpool-rdma-shared-device-plugin` |
| `rdma.rdmaSharedDevicePlugin.image.registry` | the image registry of rdma shared device plugin | `ghcr.io` |
| `rdma.rdmaSharedDevicePlugin.image.repository` | the image repository of rdma shared device plugin | `mellanox/k8s-rdma-shared-dev-plugin` |
| `rdma.rdmaSharedDevicePlugin.image.pullPolicy` | the image pullPolicy of rdma shared device plugin | `IfNotPresent` |
| `rdma.rdmaSharedDevicePlugin.image.digest` | the image digest of rdma shared device plugin | `""` |
| `rdma.rdmaSharedDevicePlugin.image.tag` | the image tag of rdma shared device plugin | `latest` |
| `rdma.rdmaSharedDevicePlugin.image.imagePullSecrets` | the image imagePullSecrets of rdma shared device plugin | `[]` |
| `rdma.rdmaSharedDevicePlugin.podAnnotations` | the additional annotations | `{}` |
| `rdma.rdmaSharedDevicePlugin.podLabels` | the additional label | `{}` |
| `rdma.rdmaSharedDevicePlugin.resources.limits.cpu` | the cpu limit | `300m` |
| `rdma.rdmaSharedDevicePlugin.resources.limits.memory` | the memory limit | `300Mi` |
| `rdma.rdmaSharedDevicePlugin.resources.requests.cpu` | the cpu requests | `100m` |
| `rdma.rdmaSharedDevicePlugin.resources.requests.memory` | the memory requests | `50Mi` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.periodicUpdateInterval` | periodic Update Interval | `300` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.resourcePrefix` | resource prefix | `spidernet.io` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.resourceName` | resource Name | `hca_shared_devices` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.rdmaHcaMax` | rdma Hca Max | `500` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.vendors` | rdma device vendors, default to mellanox device | `15b3` |
| `rdma.rdmaSharedDevicePlugin.deviceConfig.deviceIDs` | rdma device IDs, default to mellanox device | `1017` |

### multus parameters

Expand All @@ -216,6 +209,20 @@ helm install spiderpool spiderpool/spiderpool --wait --namespace kube-system \
| `multus.multusCNI.log.logLevel` | the multus-CNI daemonset pod log level | `debug` |
| `multus.multusCNI.log.logFile` | the multus-CNI daemonset pod log file | `/var/log/multus.log` |

### plugins parameters

| Name | Description | Value |
| -------------------------------- | ---------------------------------------------------------- | -------------------------------------------- |
| `plugins.installCNI` | install all cni plugins to each node | `false` |
| `plugins.installRdmaCNI` | install rdma cni used to isolate rdma device for sriov cni | `false` |
| `plugins.installOvsCNI` | install ovs cni to each node | `false` |
| `plugins.image.registry` | the image registry of plugins | `ghcr.io` |
| `plugins.image.repository` | the image repository of plugins | `spidernet-io/spiderpool/spiderpool-plugins` |
| `plugins.image.pullPolicy` | the image pullPolicy of plugins | `IfNotPresent` |
| `plugins.image.digest` | the image digest of plugins | `""` |
| `plugins.image.tag` | the image tag of plugins | `v0.8.0` |
| `plugins.image.imagePullSecrets` | the image imagePullSecrets of plugins | `[]` |

### clusterDefaultPool parameters

| Name | Description | Value |
Expand Down
18 changes: 9 additions & 9 deletions charts/spiderpool/templates/_helpers.tpl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -240,25 +240,25 @@ app.kubernetes.io/component: {{ .Values.multus.multusCNI.name | trunc 63 | trimS
name: multus
{{- end }}

#=================== rdma =====================
#=================== plugins =====================

{{/*
return the rdma cni image
return the plugins image
*/}}
{{- define "rdmacni.image" -}}
{{- $registryName := .Values.rdma.rdmaCni.image.registry -}}
{{- $repositoryName := .Values.rdma.rdmaCni.image.repository -}}
{{- define "plugins.image" -}}
{{- $registryName := .Values.plugins.image.registry -}}
{{- $repositoryName := .Values.plugins.image.repository -}}
{{- if .Values.global.imageRegistryOverride }}
{{- printf "%s/%s" .Values.global.imageRegistryOverride $repositoryName -}}
{{ else if $registryName }}
{{- printf "%s/%s" $registryName $repositoryName -}}
{{- else -}}
{{- printf "%s" $repositoryName -}}
{{- end -}}
{{- if .Values.rdma.rdmaCni.image.digest }}
{{- print "@" .Values.rdma.rdmaCni.image.digest -}}
{{- else if .Values.rdma.rdmaCni.image.tag -}}
{{- printf ":%s" .Values.rdma.rdmaCni.image.tag -}}
{{- if .Values.plugins.image.digest }}
{{- print "@" .Values.plugins.image.digest -}}
{{- else if .Values.plugins.image.tag -}}
{{- printf ":%s" .Values.plugins.image.tag -}}
{{- else -}}
{{- printf ":v%s" .Chart.AppVersion -}}
{{- end -}}
Expand Down
25 changes: 12 additions & 13 deletions charts/spiderpool/templates/daemonset.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,21 +80,20 @@ spec:
- linux
{{- end }}
initContainers:
{{- if .Values.rdma.rdmaCni.install }}
- name: rdma-cni
image: {{ include "rdmacni.image" . | quote }}
imagePullPolicy: {{ .Values.rdma.rdmaCni.image.pullPolicy }}
{{- if or .Values.plugins.installCNI .Values.plugins.installRdmaCNI .Values.plugins.installOvsCNI }}
- name: install-plugins
image: {{ include "plugins.image" . | quote }}
imagePullPolicy: {{ .Values.plugins.image.pullPolicy }}
env:
- name: INSTALL_CNI_PLUGINS
value: {{ .Values.plugins.installCNI | quote }}
- name: INSTALL_OVS_PLUGIN
value: {{ .Values.plugins.installOvsCNI | quote }}
- name: INSTALL_RDMA_PLUGIN
value: {{ .Values.plugins.installRdmaCNI | quote }}
command:
- "/bin/sh"
- "-c"
- |
BIN_LIST="rdma"
for ITEM in ${BIN_LIST} ; do
rm -f /host/opt/cni/bin/${ITEM}.old || true
( [ -f "/host/opt/cni/bin/${ITEM}" ] && mv /host/opt/cni/bin/${ITEM} /host/opt/cni/bin/${ITEM}.old ) || true
cp /usr/bin/${ITEM} /host/opt/cni/bin/${ITEM}
rm -f /host/opt/cni/bin/${ITEM}.old &>/dev/null || true
done
- "entrypoint.sh"
securityContext:
privileged: true
volumeMounts:
Expand Down
55 changes: 31 additions & 24 deletions charts/spiderpool/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -194,30 +194,6 @@ rdma:
## @param rdma.rdmaSharedDevicePlugin.deviceConfig.deviceIDs rdma device IDs, default to mellanox device
deviceIDs: "1017"

rdmaCni:
## @param rdma.rdmaCni.install install rdma cni used to isolate rdma device for sriov cni
install: false

image:
## @param rdma.rdmaCni.image.registry the rdma-cni image registry
registry: ghcr.io

## @param rdma.rdmaCni.image.repository the rdma-cni image repository
repository: k8snetworkplumbingwg/rdma-cni

## @param rdma.rdmaCni.image.pullPolicy the rdma-cni image pullPolicy
pullPolicy: IfNotPresent

## @param rdma.rdmaCni.image.digest the rdma-cni image digest
digest: ""

## @param rdma.rdmaCni.image.tag the rdma-cni image tag
tag: latest

## @param rdma.rdmaCni.image.imagePullSecrets the rdma-cni image imagePullSecrets
imagePullSecrets: []
# - name: "image-pull-secret"

## @section multus parameters
##
multus:
Expand Down Expand Up @@ -282,6 +258,37 @@ multus:
## @param multus.multusCNI.log.logFile the multus-CNI daemonset pod log file
logFile: "/var/log/multus.log"

## @section plugins parameters
##
plugins:
## @param plugins.installCNI install all cni plugins to each node
installCNI: false

## @param plugins.installRdmaCNI install rdma cni used to isolate rdma device for sriov cni
installRdmaCNI: false

## @param plugins.installOvsCNI install ovs cni to each node
installOvsCNI: false

image:
## @param plugins.image.registry the image registry of plugins
registry: ghcr.io

## @param plugins.image.repository the image repository of plugins
repository: spidernet-io/spiderpool/spiderpool-plugins

## @param plugins.image.pullPolicy the image pullPolicy of plugins
pullPolicy: IfNotPresent

## @param plugins.image.digest the image digest of plugins
digest: ""

## @param plugins.image.tag the image tag of plugins
tag: v0.8.0

## @param plugins.image.imagePullSecrets the image imagePullSecrets of plugins
imagePullSecrets: []

## @section clusterDefaultPool parameters
##
clusterDefaultPool:
Expand Down
4 changes: 3 additions & 1 deletion docs/usage/install/cloud/get-started-alibaba-zh_CN.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,11 @@ helm repo add spiderpool https://spidernet-io.github.io/spiderpool

helm repo update spiderpool

helm install spiderpool spiderpool/spiderpool --namespace kube-system --set ipam.enableStatefulSet=false --set multus.multusCNI.defaultCniCRName="ipvlan-eth0"
helm install spiderpool spiderpool/spiderpool --namespace kube-system --set ipam.enableStatefulSet=false --set multus.multusCNI.defaultCniCRName="ipvlan-eth0"
```

> 如果您的集群未安装 IPVlan, 你可以通过指定 Helm 参数 `--set plugins.installCNI=true` 安装 IPVlan。
>
> 如果您使用的是中国大陆的云厂商服务器，可以指定参数 `--set global.imageRegistryOverride=ghcr.m.daocloud.io` ，以帮助您更快的拉取镜像。
>
> Spiderpool 可以为控制器类型为：`Statefulset` 的应用副本固定 IP 地址。在公有云的 Underlay 网络场景中，云主机只能使用限定的 IP 地址，当 StatefulSet 类型的应用副本漂移到其他节点，但由于原固定的 IP 在其他节点是非法不可用的，新的 Pod 将出现网络不可用的问题。对此场景，将 `ipam.enableStatefulSet` 设置为 `false`，禁用该功能。
Expand Down
2 changes: 2 additions & 0 deletions docs/usage/install/cloud/get-started-alibaba.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ helm repo update spiderpool
helm install spiderpool spiderpool/spiderpool --namespace kube-system --set ipam.enableStatefulSet=false --set multus.multusCNI.defaultCniCRName="ipvlan-eth0"
```

> If IPVlan is not installed in your cluster, you can specify the Helm parameter `--set plugins.installCNI=true` to install IPVlan in your cluster.
>
> If you are using a cloud server from a Chinese mainland cloud provider, you can enhance image pulling speed by specifying the parameter `--set global.imageRegistryOverride=ghcr.m.daocloud.io`.
>
> Spiderpool allows for fixed IP addresses for application replicas with a controller type of `StatefulSet`. However, in the underlay network scenario of public clouds, cloud instances are limited to using specific IP addresses. When StatefulSet replicas migrate to different nodes, the original fixed IP becomes invalid and unavailable on the new node, causing network unavailability for the new Pods. To address this issue, set `ipam.enableStatefulSet` to `false` to disable this feature.
Expand Down
Loading

0 comments on commit d380f35

Please sign in to comment.