feat: add support for safe driver loading feature #600

ykulazhenkov · 2023-09-20T08:22:00Z

On Node startup, the OFED container takes some time to compile and load the driver.
During that time, workloads might get scheduled on that Node.
When OFED is loaded, all existing PODs that use NVIDIA NICs will lose their network interfaces.
Some such PODs might silently fail or hang.
To avoid such a situation, before the OFED container is loaded,
the Node should get Cordoned and Drained to ensure all workloads are rescheduled.
The Node should be un-cordoned when the driver is ready on it.

The safe driver loading feature is implemented as a part of the upgrade flow,
meaning safe driver loading is a special scenario of the upgrade procedure,
where we upgrade from the inbox driver to the containerized OFED.

depends on: #685

ykulazhenkov · 2023-09-21T06:40:43Z

/retest-all

abdallahyas · 2023-09-21T07:23:45Z

@ykulazhenkov for the kind CI a rebase is needed to have the revert of the tighting patch applied, for the helm CI, the mofed container seem to have been delayed to be up, causing the 10 minute timeout of the CI to fail, Do you think we need to increase the timeout of the CI to wait for the nic cluster policy to become ready.

ykulazhenkov · 2023-11-08T15:01:13Z

/retest-all

ykulazhenkov · 2023-11-22T17:40:00Z

I updated PR code to use network-operator-init-container 0.0.2

ykulazhenkov · 2023-11-23T13:41:14Z

/retest-nic_operator_helm

ykulazhenkov · 2023-11-23T14:09:28Z

/retest-nic_operator_helm

ykulazhenkov · 2023-11-28T09:32:15Z

/retest-nic_operator_helm

main.go

pkg/state/state_ofed.go

api/v1alpha1/nicclusterpolicy_types.go

ykulazhenkov · 2023-11-29T09:47:25Z

I added check to Admission controller to make sure that initContainer and autoUpgrade are enabled when a user tries to enable safeLoad feature.
Also, I removed ofed.initContainer.enable flag from the CR (I kept it in Helm) - this flag has no extra meaning.

ykulazhenkov · 2023-11-29T14:59:12Z

/retest-nic_operator_helm

ykulazhenkov · 2023-11-30T07:41:20Z

/retest-nic_operator_helm

api/v1alpha1/nicclusterpolicy_types.go

ykulazhenkov · 2023-12-01T16:21:05Z

@adrianchiris @rollandf @almaslennikov I did a big change in the PR after discussion with Adrian: now we use enviroment variable to configure image for the init container. Please, take a look again

rollandf

Small nit, otherwise LGTM

api/v1alpha1/nicclusterpolicy_webhook_test.go

deployment/network-operator/README.md

pkg/config/config.go

adrianchiris · 2023-12-04T08:53:02Z

pkg/state/state_ofed.go

+			OSName:              nodeAttr[nodeinfo.AttrTypeOSName],
+			OSVer:               nodeAttr[nodeinfo.AttrTypeOSVer],
+			MOFEDImageName:      s.getMofedDriverImageName(cr, nodeAttr, reqLogger),
+			InitContainerConfig: s.getInitContainerConfig(cr, reqLogger, config.FromEnv().OFEDState.InitContainerImage),


i believe we would generally want to work with staticconfig provider[1] or just delete it and work with config.FromEnv()...

network-operator/main.go

Line 85 in c40b096

staticInfoProvider := staticconfig.NewProvider(staticconfig.StaticConfig{CniBinDirectory: cniBinDir})

same goes for the namespace we use in runtimespec (here and in other states)

thoughts ? (ill create a task to align so it will not affect this PR)

yeah, I was thinking about using a static provider, but I didn't find any benefits from it.
Maybe we don't need the static provider at all, because it doesn't make sense to cache env variables (we already read them only once)

maybe will be useful for UT ? (one less source of informaiton)

that is a state gets the CR, some providers, renderer, and do the work.

created #698 please add your preference there.

Signed-off-by: Yury Kulazhenkov <ykulazhenkov@nvidia.com>

ykulazhenkov · 2023-12-04T11:29:36Z

/retest-nic_operator_helm

e0ne

LGTM

ykulazhenkov added the on hold This enhancement is currently on hold pending additional clarification and evaluation label Sep 20, 2023

ykulazhenkov force-pushed the safe-driver-load branch 2 times, most recently from d72fe32 to 5054f10 Compare September 20, 2023 08:51

ykulazhenkov force-pushed the safe-driver-load branch from 5054f10 to de36e05 Compare September 21, 2023 07:33

almaslennikov mentioned this pull request Sep 22, 2023

Configure drain skip labels explicitly on the net-op side #577

Closed

almaslennikov approved these changes Sep 22, 2023

View reviewed changes

ykulazhenkov force-pushed the safe-driver-load branch 2 times, most recently from a1b8ea0 to 05bcf3e Compare November 1, 2023 11:39

ykulazhenkov changed the title ~~Add support for safe driver loading feature~~ feat: add support for safe driver loading feature Nov 1, 2023

ykulazhenkov force-pushed the safe-driver-load branch 3 times, most recently from 37834e9 to 6cf4d7a Compare November 8, 2023 14:39

ykulazhenkov marked this pull request as ready for review November 8, 2023 14:40

ykulazhenkov force-pushed the safe-driver-load branch 2 times, most recently from 2d368ce to 18d4400 Compare November 14, 2023 15:31

ykulazhenkov mentioned this pull request Nov 21, 2023

chore: update all dependencies for the operator #685

Merged

ykulazhenkov force-pushed the safe-driver-load branch 2 times, most recently from 09dbdff to 598d765 Compare November 22, 2023 17:34

ykulazhenkov marked this pull request as draft November 22, 2023 17:35

ykulazhenkov force-pushed the safe-driver-load branch from 598d765 to 4de254d Compare November 23, 2023 11:50

ykulazhenkov marked this pull request as ready for review November 23, 2023 13:40

ykulazhenkov force-pushed the safe-driver-load branch 2 times, most recently from c7b7be3 to 3eca43a Compare November 27, 2023 11:06

ykulazhenkov removed the on hold This enhancement is currently on hold pending additional clarification and evaluation label Nov 27, 2023

ykulazhenkov requested review from rollandf and adrianchiris November 27, 2023 11:06

rollandf reviewed Nov 28, 2023

View reviewed changes

main.go Outdated Show resolved Hide resolved

pkg/state/state_ofed.go Outdated Show resolved Hide resolved

api/v1alpha1/nicclusterpolicy_types.go Outdated Show resolved Hide resolved

ykulazhenkov force-pushed the safe-driver-load branch 3 times, most recently from f8a9925 to 76776e3 Compare November 28, 2023 14:34

rollandf approved these changes Nov 29, 2023

View reviewed changes

ykulazhenkov force-pushed the safe-driver-load branch from 76776e3 to c27eff8 Compare November 29, 2023 09:44

ykulazhenkov force-pushed the safe-driver-load branch from c27eff8 to e578586 Compare November 29, 2023 09:49

adrianchiris reviewed Nov 30, 2023

View reviewed changes

api/v1alpha1/nicclusterpolicy_types.go Outdated Show resolved Hide resolved

ykulazhenkov force-pushed the safe-driver-load branch 2 times, most recently from 7fb0594 to 003c802 Compare December 1, 2023 15:35

ykulazhenkov requested review from rollandf and almaslennikov December 1, 2023 16:18

rollandf approved these changes Dec 4, 2023

View reviewed changes

api/v1alpha1/nicclusterpolicy_webhook_test.go Outdated Show resolved Hide resolved

adrianchiris reviewed Dec 4, 2023

View reviewed changes

ykulazhenkov added 3 commits December 4, 2023 10:59

Add OFED safeLoad setting to NicClusterPolicy

7b56106

Signed-off-by: Yury Kulazhenkov <ykulazhenkov@nvidia.com>

Add support for network-operator-init-container

70f2a53

Signed-off-by: Yury Kulazhenkov <ykulazhenkov@nvidia.com>

Add support in Helm and docs

b2398da

Signed-off-by: Yury Kulazhenkov <ykulazhenkov@nvidia.com>

ykulazhenkov force-pushed the safe-driver-load branch from 003c802 to b2398da Compare December 4, 2023 09:07

adrianchiris approved these changes Dec 4, 2023

View reviewed changes

almaslennikov approved these changes Dec 4, 2023

View reviewed changes

e0ne approved these changes Dec 5, 2023

View reviewed changes

e0ne merged commit f64281f into Mellanox:master Dec 5, 2023
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for safe driver loading feature #600

feat: add support for safe driver loading feature #600

ykulazhenkov commented Sep 20, 2023 •

edited

Loading

ykulazhenkov commented Sep 21, 2023

abdallahyas commented Sep 21, 2023

ykulazhenkov commented Nov 8, 2023

ykulazhenkov commented Nov 22, 2023 •

edited

Loading

ykulazhenkov commented Nov 23, 2023

ykulazhenkov commented Nov 23, 2023

ykulazhenkov commented Nov 28, 2023

ykulazhenkov commented Nov 29, 2023 •

edited

Loading

ykulazhenkov commented Nov 29, 2023

ykulazhenkov commented Nov 30, 2023

ykulazhenkov commented Dec 1, 2023 •

edited

Loading

rollandf left a comment

adrianchiris Dec 4, 2023

ykulazhenkov Dec 4, 2023

adrianchiris Dec 4, 2023

adrianchiris Dec 4, 2023 •

edited

Loading

ykulazhenkov commented Dec 4, 2023

e0ne left a comment

feat: add support for safe driver loading feature #600

feat: add support for safe driver loading feature #600

Conversation

ykulazhenkov commented Sep 20, 2023 • edited Loading

ykulazhenkov commented Sep 21, 2023

abdallahyas commented Sep 21, 2023

ykulazhenkov commented Nov 8, 2023

ykulazhenkov commented Nov 22, 2023 • edited Loading

ykulazhenkov commented Nov 23, 2023

ykulazhenkov commented Nov 23, 2023

ykulazhenkov commented Nov 28, 2023

ykulazhenkov commented Nov 29, 2023 • edited Loading

ykulazhenkov commented Nov 29, 2023

ykulazhenkov commented Nov 30, 2023

ykulazhenkov commented Dec 1, 2023 • edited Loading

rollandf left a comment

Choose a reason for hiding this comment

adrianchiris Dec 4, 2023

Choose a reason for hiding this comment

ykulazhenkov Dec 4, 2023

Choose a reason for hiding this comment

adrianchiris Dec 4, 2023

Choose a reason for hiding this comment

adrianchiris Dec 4, 2023 • edited Loading

Choose a reason for hiding this comment

ykulazhenkov commented Dec 4, 2023

e0ne left a comment

Choose a reason for hiding this comment

ykulazhenkov commented Sep 20, 2023 •

edited

Loading

ykulazhenkov commented Nov 22, 2023 •

edited

Loading

ykulazhenkov commented Nov 29, 2023 •

edited

Loading

ykulazhenkov commented Dec 1, 2023 •

edited

Loading

adrianchiris Dec 4, 2023 •

edited

Loading