Support optional config #530

adrianchiris · 2023-05-22T11:42:57Z

No description provided.

adrianchiris · 2023-06-28T12:30:22Z

@ykulazhenkov @rollandf PTAL

example-cr.yaml

kind-cluster.yaml

nv-ipam-cm.yaml

rdma-dp-config

sriov-dp-config

value-overwrite.yaml

deployment/network-operator/values.yaml

adrianchiris · 2023-06-29T08:34:50Z

@ykulazhenkov addressed your comments, PTAL

ykulazhenkov

LGTM, thx

e0ne · 2023-06-29T14:18:49Z

api/v1alpha1/nicclusterpolicy_types.go

@@ -208,11 +212,11 @@ type IBKubernetesSpec struct {
 	UfmSecret string `json:"ufmSecret,omitempty"`
 }

+// NVIPAMSpec describes configuration options for nv-ipam
+// 1. Image information for nv-ipam
+// 2. Config for nv-ipam in JSON format


nit: this comment if a bit inaccurate because we specify a string representation of JSON here

what exactly would you like to appear here ?

maybe just 'Config for nv-ipam' is better option with a good examples and defaults. Anyway, I won't block PR for this or other nit

ack will update

e0ne · 2023-06-29T14:19:28Z

deployment/network-operator/README.md

@@ -561,7 +561,7 @@ optionally deployed components:
 | `nvIpam.repository`       | string | `ghcr.io/mellanox` | NVIDIA IPAM Plugin image repository                                                    |
 | `nvIpam.version`          | string | `v0.0.3`           | NVIDIA IPAM Plugin image version                                                       |
 | `nvIpam.imagePullSecrets` | list   | `[]`               | An optional list of references to secrets to use for pulling any of the Plugin image   |
-| `nvIpam.config`           | string | `''`               | Network pool configuration as described in https://github.com/Mellanox/nvidia-k8s-ipam |
+| `nvIpam.config`           | string | `{"pools": {"rdma-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}}}` | Network pool configuration as described in [nvidia-k8s-ipam](https://github.com/Mellanox/nvidia-k8s-ipam), the default defines a single IP Pool named `"rdma-pool"`|


nit: JSON shouble be coverd in double quotes

what do you mean ?

something like:
"{\"foo\": \"bar\"}"

dont u think it will hinder readability ?

also the value above is what we put in values.yaml as default there are no double quotes there.

I mean:
"{"pools": {"rdma-pool": {"subnet": "192.168.0.0/16", "perNodeBlockSize": 100, "gateway": "192.168.0.1"}}}"

also the value above is what we put in values.yaml as default there are no double quotes there.

that's because we use pipe symbol | thare which means multi-line string

ack, ill update

e0ne · 2023-06-29T14:22:16Z

manifests/state-multus-cni/0040-configmap.yml

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+{{ if .CrSpec.Config -}}


please, add deletion of this comfigmap logic to NicClusterPolicy controller for a consistancy: if CRD is changed and config is deleted we have to delete config map

There is a general mechanims at state level where resources are being deleted if needed.
do you really think we need a finer grained mechanism ?

if user sets config to nil, its expected that the next steps for him is to re-create the config map with the given name.
so it will be deleted by the user.

thinking about it, NOT deleting it may be actually better, because then the user may know the name of the config map it needs to re-create :).

There is a general mechanims at state level where resources are being deleted if needed. do you really think we need a finer grained mechanism ?
This

network-operator/pkg/state/state_multus_cni.go

Line 79 in 5fb589e

if cr.Spec.SecondaryNetwork == nil || cr.Spec.SecondaryNetwork.Multus == nil {

won't delete ConfigMap if Multus should be deployed

if user sets config to nil, its expected that the next steps for him is to re-create the config map with the given name. so it will be deleted by the user.

thinking about it, NOT deleting it may be actually better, because then the user may know the name of the config map it needs to re-create :).

I don't understans this use-case when user deletes operator-generated ConfigMap and creates it's own. My assumption is that all configuration should be done via NicClusterPolicy

I don't understans this use-case when user deletes operator-generated ConfigMap and creates it's own. My assumption is that all configuration should be done via NicClusterPolicy

the use-case is having configuration provided out of band. this was an internal request.
where:

user deploys network operator and provides nicclusterpolicy with nil config for e.g nv-ipam

at a later step when network provisioning occurs the user defines the needed config map for nv-ipam

this was the ask, in contrast to changing nicclusterpolicy at a later step which will probably mess up argo CD or smthn,

the use-case is having configuration provided out of band. this was an internal request.

in this case, please add a note about this feature in a README and this will be good to merge

won't delete ConfigMap if Multus should be deployed

Yes, and i think we dont need a finer grained mechansim to handle it (for now...)

added a section in README.

adrianchiris · 2023-07-02T11:47:17Z

README.md

 Check [Upgrade section in Helm Chart documentation](deployment/network-operator/README.md#upgrade) for details.
+
+## Externally Provided Configurations For Network Operator Sub-Components


@e0ne PTAL

we can either keep this section here or move it to its own document under /docs/
as its an advanced use case.

e0ne

LGTM. thanks for addressing my comments. It could be merged after rebase and CI passed

- Consolidate image spec with config to a common struct and use where applicable (device plugins, multus, nv-ipam) - set Config field as optional. use nil value to determine if unset. Signed-off-by: adrianc <adrianc@nvidia.com>

Run: `make generate && make manifests && make release-build` Signed-off-by: adrianc <adrianc@nvidia.com>

This is required to allow providing configurations for device plugins (RDMA, SR-IOV), nv-ipam out of band. That is, the user creates it prior to deploying these components with network operator. or alternatively restarting the relevant daemonset after config map creation. note that for multus, not providing configuration for configMap, will lead to multus using "auto" configuration same as it does today. Signed-off-by: adrianc <adrianc@nvidia.com>

Signed-off-by: adrianc <adrianc@nvidia.com>

define a single pool to be used for rdma networks to serve as default configuration. Signed-off-by: adrianc <adrianc@nvidia.com>

update README to reflect the new default for nvIpam config Signed-off-by: adrianc <adrianc@nvidia.com>

- Update project README - Update Helm README, adding quotes to default configuration of nv-ipam - Update comment in CRD Signed-off-by: adrianc <adrianc@nvidia.com>

adrianchiris · 2023-07-03T07:06:31Z

@e0ne PTAL

e0ne

LGTM. Thanks for addressing my comments

adrianchiris · 2023-07-04T11:08:20Z

@e0ne Feel free to merge this one when ready.

adrianchiris requested review from ykulazhenkov and e0ne May 22, 2023 11:44

adrianchiris added the on hold This enhancement is currently on hold pending additional clarification and evaluation label May 22, 2023

adrianchiris force-pushed the support-optional-config branch 2 times, most recently from dd763e1 to 871d33f Compare May 23, 2023 11:09

adrianchiris force-pushed the support-optional-config branch from 871d33f to fd81987 Compare June 28, 2023 12:29

adrianchiris removed the on hold This enhancement is currently on hold pending additional clarification and evaluation label Jun 28, 2023

ykulazhenkov reviewed Jun 29, 2023

View reviewed changes

deployment/network-operator/values.yaml Outdated Show resolved Hide resolved

adrianchiris force-pushed the support-optional-config branch from bbbaeff to 6c8432e Compare June 29, 2023 08:24

ykulazhenkov self-requested a review June 29, 2023 09:28

ykulazhenkov approved these changes Jun 29, 2023

View reviewed changes

e0ne requested changes Jun 29, 2023

View reviewed changes

adrianchiris commented Jul 2, 2023

View reviewed changes

adrianchiris force-pushed the support-optional-config branch 2 times, most recently from 4496082 to 2cf48d5 Compare July 2, 2023 12:49

e0ne reviewed Jul 2, 2023

View reviewed changes

adrianchiris added 9 commits July 3, 2023 10:03

optional nicclusterpolicy Config field

2c36ec3

- Consolidate image spec with config to a common struct and use where applicable (device plugins, multus, nv-ipam) - set Config field as optional. use nil value to determine if unset. Signed-off-by: adrianc <adrianc@nvidia.com>

Generate files

f27f978

Run: `make generate && make manifests && make release-build` Signed-off-by: adrianc <adrianc@nvidia.com>

Support optional config in helm chart

78953a8

Signed-off-by: adrianc <adrianc@nvidia.com>

Remove un-needed checks in shared device plugin state

41fdcf2

Signed-off-by: adrianc <adrianc@nvidia.com>

Update templates and generate them

855f893

Signed-off-by: adrianc <adrianc@nvidia.com>

deploy nv-ipam with default pool config

11c1d29

define a single pool to be used for rdma networks to serve as default configuration. Signed-off-by: adrianc <adrianc@nvidia.com>

Update network-operator helm README

be8bd94

update README to reflect the new default for nvIpam config Signed-off-by: adrianc <adrianc@nvidia.com>

Enhance Documentation

c4e9a99

- Update project README - Update Helm README, adding quotes to default configuration of nv-ipam - Update comment in CRD Signed-off-by: adrianc <adrianc@nvidia.com>

adrianchiris force-pushed the support-optional-config branch from 2cf48d5 to c4e9a99 Compare July 3, 2023 07:05

e0ne approved these changes Jul 3, 2023

View reviewed changes

e0ne merged commit 6c21718 into Mellanox:master Jul 4, 2023
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support optional config #530

Support optional config #530

adrianchiris commented May 22, 2023

adrianchiris commented Jun 28, 2023

adrianchiris commented Jun 29, 2023

ykulazhenkov left a comment

e0ne Jun 29, 2023

adrianchiris Jun 29, 2023

e0ne Jun 29, 2023

adrianchiris Jun 29, 2023

adrianchiris Jul 2, 2023

e0ne Jun 29, 2023

adrianchiris Jun 29, 2023

e0ne Jun 29, 2023

adrianchiris Jun 29, 2023

adrianchiris Jul 2, 2023

e0ne Jun 29, 2023

adrianchiris Jun 29, 2023

e0ne Jun 29, 2023

adrianchiris Jun 29, 2023 •

edited

Loading

e0ne Jun 29, 2023

adrianchiris Jun 29, 2023 •

edited

Loading

adrianchiris Jul 2, 2023

adrianchiris Jul 2, 2023

e0ne left a comment

adrianchiris commented Jul 3, 2023

e0ne left a comment

adrianchiris commented Jul 4, 2023 •

edited

Loading

		Check [Upgrade section in Helm Chart documentation](deployment/network-operator/README.md#upgrade) for details.

		## Externally Provided Configurations For Network Operator Sub-Components

Support optional config #530

Support optional config #530

Conversation

adrianchiris commented May 22, 2023

adrianchiris commented Jun 28, 2023

adrianchiris commented Jun 29, 2023

ykulazhenkov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianchiris Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianchiris Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e0ne left a comment

Choose a reason for hiding this comment

adrianchiris commented Jul 3, 2023

e0ne left a comment

Choose a reason for hiding this comment

adrianchiris commented Jul 4, 2023 • edited Loading

adrianchiris Jun 29, 2023 •

edited

Loading

adrianchiris Jun 29, 2023 •

edited

Loading

adrianchiris commented Jul 4, 2023 •

edited

Loading