migrate nrt to ctrl runtime client #655

zwpaper · 2023-11-01T15:25:16Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

part of #485

Special notes for your reviewer:

after deleting the generated vendor, CI found noderesourcetopology crd file changed, v1alpha1 version was deleted, I checked and found nowhere that used the v1alpha1, so I deleted the version.

now we only have noderesourcetopology v1alpha2

Does this PR introduce a user-facing change?

migrate noderesourcetopology to ctrl runtime client

netlify · 2023-11-01T15:25:23Z

✅ Deploy Preview for kubernetes-sigs-scheduler-plugins canceled.

Name	Link
🔨 Latest commit	`0b5755d`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-scheduler-plugins/deploys/6550a66ff119c20008ad111c

Huang-Wei · 2023-11-01T17:36:30Z

/assign

zwpaper · 2023-11-03T06:53:27Z

@Huang-Wei CI passed, PTAL

Huang-Wei

Some nits. LGTM overall.

/approve
/lgtm

pkg/noderesourcetopology/cache/overreserve.go

Huang-Wei · 2023-11-04T16:25:58Z

pkg/noderesourcetopology/cache/overreserve.go

 		podLister:              podLister,
 		resyncMethod:           resyncMethod,
 	}
 	return obj, nil
 }

-func (ov *OverReserve) GetCachedNRTCopy(nodeName string, pod *corev1.Pod) (*topologyv1alpha2.NodeResourceTopology, bool) {
+func (ov *OverReserve) GetCachedNRTCopy(ctx context.Context, nodeName string, pod *corev1.Pod) (*topologyv1alpha2.NodeResourceTopology, bool) {


It seems ctx is not used in this function.

yup, either pass it through ov.Get (which will become ov.client.Get?) or discard it please (applies to similar places elsewhere in the code)

it is because we use the ctx here: https://github.com/kubernetes-sigs/scheduler-plugins/pull/655/files#diff-d469fc389aadc64511b0fcbf7af645119b04c14a1cc18f5d2307755df1c41a2eR57

and this function is part of the cache Interface here: https://github.com/kubernetes-sigs/scheduler-plugins/pull/655/files#diff-4792874d567196ef3ce188f2f3ccda73d46457d99d95111e04a0072dcf183375R35

so we have to change it to both DiscardReserved and OverReserve

BTW, Passthrough also used the ctx in GetCachedNRTCopy: https://github.com/kubernetes-sigs/scheduler-plugins/pull/655/files#diff-5747013d93a36239aaa4d659c8de3b9a2ef08bbd0f842377c25a1e810007b3e6R44

right, my bad. It would be nice to change it as
func (ov *OverReserve) GetCachedNRTCopy(_ context.Context, nodeName string, pod *corev1.Pod) (*topologyv1alpha2.NodeResourceTopology, bool) { }
because ctx is indeed not needed in the body, but this is borderline a nit and totally not blocking

k8s-ci-robot · 2023-11-04T16:32:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei, zwpaper

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Huang-Wei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Huang-Wei · 2023-11-04T16:33:43Z

/hold

cc @ffromani @swatisehgal @Tal-or for awareness.

The background is that scheduler-plugins start to use more CRDs as time goes, and the current way is to generate typed client for each CRD and leverage in the plugin codebase. The codegen part is annoying and raise the bar of using/understanding scheduler plugin. So the overall goal to turn to dynamic client by leverage controller runtime's utilities.

cc-ing you is b/c it seems you're working on pushing NRT to upstream. Hope this work doesn't conflict with your work.

Tal-or

Thank you for the PR.
I left some notes from my side, and I would wait for @ffromani and @swatisehgal to review this PR as well.

pkg/noderesourcetopology/cache/overreserve.go

Tal-or · 2023-11-05T09:46:04Z

pkg/noderesourcetopology/pluginhelpers.go

@@ -43,32 +45,26 @@ const (
 )

 func initNodeTopologyInformer(tcfg *apiconfig.NodeResourceTopologyMatchArgs, handle framework.Handle) (nrtcache.Interface, error) {
-	topoClient, err := topoclientset.NewForConfig(handle.KubeConfig())
+	scheme := runtime.NewScheme()
+	utilruntime.Must(clientgoscheme.AddToScheme(scheme))


I would handle this error instead of panic

this error in practice won't happen IMO. If we treat it as an error, @Tal-or do you mean we can find a way to proceed?

I'm not sure why this commit (which seems to want to focus on test code) needs to add manage schemas

@Huang-Wei Not sure if it's possible to proceed, but at least throwing a detailed error which bubble up and explained what happen, would be more user friendly.

This is how the controller runtime client can know what to do with the types, this is a common usage from kube-builder, which is one of the most famous libs using controller runtime.

you can check: https://book.kubebuilder.io/cronjob-tutorial/empty-main

I agree with @Huang-Wei . The error is not recoverable, so it's fine to use utilruntime.Must and fail badly in the (I reckon) exceptionally rare case on which AddToScheme fails.

The only thing which doesn't fully click to me is why we need to fix the schema in this precise code location. Disclosure: I didn't review the latest changes, so this is possibly already improved.

Hi @ffromani, I did not update the codes here.

we must handle the scheme somewhere, as the main scheduler, we added in main.go, and for this nrt plugin, I add it here when initing it, so that we are not adding this scheme if nrt plugin is not used.

as for utilruntime.Must, WDYT @Tal-or ?

@ffromani raised a very good point. The AddToScheme logic should only happen once per its usage:

for running scheduler, it's should in main's init() or equivalent place. @zwpaper I know this is generic code problem, but if you can add this logic in this PR, it'd be much appreciated 🙇‍♂️

for UTs, it's fine to init in each UT as we treat each UT as a client

as for utilruntime.Must, WDYT @Tal-or ?

@zwpaper since the other reviewers are in favor of utilruntime.Must that's ok

@Huang-Wei moved all scheme initial into init func, PTAL

# rg -B 1 -g '!vendor/' -g '!*_test.go' -- utilruntime.Must pkg/noderesourcetopology/plugin.go 45-func init() { 46: utilruntime.Must(clientgoscheme.AddToScheme(scheme)) 47: utilruntime.Must(topologyv1alpha2.AddToScheme(scheme)) pkg/networkaware/topologicalsort/topologicalsort.go 45-func init() { 46: utilruntime.Must(clientgoscheme.AddToScheme(scheme)) 47: utilruntime.Must(agv1alpha.AddToScheme(scheme)) pkg/networkaware/networkoverhead/networkoverhead.go 66-func init() { 67: utilruntime.Must(clientgoscheme.AddToScheme(scheme)) 68- 69: utilruntime.Must(agv1alpha1.AddToScheme(scheme)) 70: utilruntime.Must(ntv1alpha1.AddToScheme(scheme)) pkg/generated/clientset/versioned/scheme/register.go 54- v1.AddToGroupVersion(Scheme, schema.GroupVersion{Version: "v1"}) 55: utilruntime.Must(AddToScheme(Scheme)) pkg/generated/clientset/versioned/fake/register.go 54- v1.AddToGroupVersion(scheme, schema.GroupVersion{Version: "v1"}) 55: utilruntime.Must(AddToScheme(scheme)) cmd/controller/app/server.go 37-func init() { 38: utilruntime.Must(clientgoscheme.AddToScheme(scheme)) 39- 40: utilruntime.Must(schedulingv1a1.AddToScheme(scheme)) apis/config/scheme/scheme.go 44-func AddToScheme(scheme *runtime.Scheme) { 45: utilruntime.Must(config.AddToScheme(scheme)) 46: utilruntime.Must(v1.AddToScheme(scheme)) 47: utilruntime.Must(v1beta2.AddToScheme(scheme)) 48: utilruntime.Must(v1beta3.AddToScheme(scheme)) apis/scheduling/scheme/scheme.go 31-func AddToScheme(scheme *runtime.Scheme) { 32: utilruntime.Must(schedv1alpha1.AddToScheme(scheme

pkg/noderesourcetopology/pluginhelpers.go

pkg/noderesourcetopology/score_test.go

ffromani

there could be some impl details to iterate over, but design wise moving to controller-runtime seems OK. Doesn't help us in general in our journey towards k8s core merge (or its proposal, at least), but its' manageable.

What puzzles me more is if this approach is still compatible with #599 .
Note that #599 is not necessarily something which I want to merge, but still the problem it attacks is real, and we didn't reach a conclusion which side should carry a fix. In addition, it would be nice to have the option to workaround the issue also on scheduler side if needed.

pkg/noderesourcetopology/cache/overreserve.go

ffromani · 2023-11-06T09:49:09Z

pkg/noderesourcetopology/cache/overreserve.go

 		podLister:              podLister,
 		resyncMethod:           resyncMethod,
 	}
 	return obj, nil
 }

-func (ov *OverReserve) GetCachedNRTCopy(nodeName string, pod *corev1.Pod) (*topologyv1alpha2.NodeResourceTopology, bool) {
+func (ov *OverReserve) GetCachedNRTCopy(ctx context.Context, nodeName string, pod *corev1.Pod) (*topologyv1alpha2.NodeResourceTopology, bool) {


yup, either pass it through ov.Get (which will become ov.client.Get?) or discard it please (applies to similar places elsewhere in the code)

ffromani · 2023-11-06T09:50:03Z

pkg/noderesourcetopology/cache/passthrough.go

 )

 type Passthrough struct {
-	lister listerv1alpha2.NodeResourceTopologyLister
+	client ctrlclient.Client


likewise, we're not using embedding here so better be consistent in OverReserve as well

I have changed the OverReserve to private client to keep it consistent.

ffromani · 2023-11-06T09:52:47Z

pkg/noderesourcetopology/pluginhelpers.go

@@ -43,32 +45,26 @@ const (
 )

 func initNodeTopologyInformer(tcfg *apiconfig.NodeResourceTopologyMatchArgs, handle framework.Handle) (nrtcache.Interface, error) {
-	topoClient, err := topoclientset.NewForConfig(handle.KubeConfig())
+	scheme := runtime.NewScheme()
+	utilruntime.Must(clientgoscheme.AddToScheme(scheme))


I'm not sure why this commit (which seems to want to focus on test code) needs to add manage schemas

ffromani · 2023-11-06T09:53:28Z

test/integration/noderesourcetopology_cache_test.go

+			scheme := runtime.NewScheme()
+			utilruntime.Must(clientgoscheme.AddToScheme(scheme))
+			utilruntime.Must(topologyv1alpha2.AddToScheme(scheme))


while this is probably OK, it seems wasteful to do each time. Can we do it once when test runs?

the tests run independently, how can we init once and run cross all the tests? maybe we have to refactor to use a test.M?

I sort of agree with @zwpaper, in the perspective to making each UT hermetic.

ffromani · 2023-11-06T09:53:34Z

test/integration/noderesourcetopology_cache_test.go

+	scheme := runtime.NewScheme()
+	utilruntime.Must(clientgoscheme.AddToScheme(scheme))
+	utilruntime.Must(topologyv1alpha2.AddToScheme(scheme))


ffromani · 2023-11-06T09:53:46Z

test/integration/noderesourcetopology_test.go

+	scheme := runtime.NewScheme()
+	utilruntime.Must(clientgoscheme.AddToScheme(scheme))
+	utilruntime.Must(topologyv1alpha2.AddToScheme(scheme))
+	extClient, err := ctrlclient.New(globalKubeConfig, ctrlclient.Options{Scheme: scheme})


Huang-Wei · 2023-11-10T02:40:26Z

What puzzles me more is if this approach is still compatible with #599 .

@ffromani this PR won't block #599 as #599 touched scheduler config API, and the introduced informer are imposed on core api objects (pods).

Signed-off-by: Zhang Wei <kweizh@gmail.com>

Huang-Wei

LGTM overall.

A potential future improvement is to instantiate the scheme once (in main.go) for non-test code, and then pass to all plugins that need it using a WithScheme option. But this doesn't block on this PR's merge.

Huang-Wei · 2023-11-13T17:52:06Z

pkg/networkaware/topologicalsort/topologicalsort_test.go

@@ -28,7 +28,7 @@ import (
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/util/rand"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
-	"k8s.io/client-go/kubernetes/scheme"
+	clientgoscheme "k8s.io/client-go/kubernetes/scheme"


nit: revert this?

the scheme name is in conflict with the var scheme for init schemes, changed to keep consistency

ffromani

LGTM
(non-binding to let other tagged reviewers review)

ffromani · 2023-11-14T16:16:53Z

pkg/noderesourcetopology/cache/overreserve.go

 		podLister:              podLister,
 		resyncMethod:           resyncMethod,
 	}
 	return obj, nil
 }

-func (ov *OverReserve) GetCachedNRTCopy(nodeName string, pod *corev1.Pod) (*topologyv1alpha2.NodeResourceTopology, bool) {
+func (ov *OverReserve) GetCachedNRTCopy(ctx context.Context, nodeName string, pod *corev1.Pod) (*topologyv1alpha2.NodeResourceTopology, bool) {


right, my bad. It would be nice to change it as
func (ov *OverReserve) GetCachedNRTCopy(_ context.Context, nodeName string, pod *corev1.Pod) (*topologyv1alpha2.NodeResourceTopology, bool) { }
because ctx is indeed not needed in the body, but this is borderline a nit and totally not blocking

ffromani · 2023-11-14T16:17:56Z

pkg/noderesourcetopology/cache/overreserve.go

-		nrtCandidate, err := ov.nrtLister.Get(nodeName)
-		if err != nil {
+		nrtCandidate := &topologyv1alpha2.NodeResourceTopology{}
+		if err := ov.client.Get(context.Background(), types.NamespacedName{Name: nodeName}, nrtCandidate); err != nil {


I wonder if we should use the same context for all the Gets - and I think we should (and we should also make it cancellable as further extension). But this change is better served in a followup PR.

yup, other PRs also have some leftovers. The general idea would be use context wherever you can:

the chained calls in PreFitler/Filter can use the context that is provided by scheduler framework

for calls outside PreFilter/Filter/..., we should instantiate one at the start of main() (or make some change in scheduler framework to expose it), and pass it to all plugins

created an issue to track this #664

Huang-Wei · 2023-11-14T18:54:04Z

/lgtm
/unhold

k8s-ci-robot requested review from swatisehgal and Tal-or November 1, 2023 15:25

zwpaper marked this pull request as draft November 1, 2023 15:25

zwpaper changed the title ~~WIP migrate nrt to ctrl runtime~~ WIP migrate nrt to ctrl runtime client Nov 1, 2023

zwpaper mentioned this pull request Nov 1, 2023

migrate controllers and clients to leverage controller-runtime #485

Closed

17 tasks

k8s-ci-robot assigned Huang-Wei Nov 1, 2023

zwpaper force-pushed the refactor-ctrl-runtime-capacity branch from c3ee7ed to 723706e Compare November 2, 2023 16:45

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 2, 2023

zwpaper changed the title ~~WIP migrate nrt to ctrl runtime client~~ migrate nrt to ctrl runtime client Nov 3, 2023

zwpaper marked this pull request as ready for review November 3, 2023 00:08

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 3, 2023

k8s-ci-robot requested a review from seanmalloy November 3, 2023 00:08

zwpaper force-pushed the refactor-ctrl-runtime-capacity branch 3 times, most recently from 0524bcf to d447dfc Compare November 3, 2023 04:35

Huang-Wei reviewed Nov 4, 2023

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 4, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 4, 2023

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 4, 2023

Huang-Wei mentioned this pull request Nov 4, 2023

Instantiate one NRT informer across NRT plugin instances #657

Closed

Tal-or reviewed Nov 5, 2023

View reviewed changes

NoicFank mentioned this pull request Nov 6, 2023

Multiple informers for the same resource(NodeResourceTopologies) may be serving at the same time #656

Closed

4 tasks

ffromani reviewed Nov 6, 2023

View reviewed changes

zwpaper force-pushed the refactor-ctrl-runtime-capacity branch from c14580c to f4b63d2 Compare November 9, 2023 15:33

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 9, 2023

zwpaper added 4 commits November 12, 2023 17:44

controller: migrate nrt to ctrl runtime

6ed4997

tests: migrate nrt tests to ctrl runtime

4f943be

vendor: delete generated nrt client

76b044c

Signed-off-by: Zhang Wei <kweizh@gmail.com>

crd: drop noderesourcetopology v1alpha1 in crd

d7a2cf4

Signed-off-by: Zhang Wei <kweizh@gmail.com>

zwpaper force-pushed the refactor-ctrl-runtime-capacity branch from f4b63d2 to 9471d79 Compare November 12, 2023 09:45

refactor: move scheme initial into init functions in each plugins

0b5755d

zwpaper force-pushed the refactor-ctrl-runtime-capacity branch from 9471d79 to 0b5755d Compare November 12, 2023 10:18

Huang-Wei reviewed Nov 13, 2023

View reviewed changes

ffromani reviewed Nov 14, 2023

View reviewed changes

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Nov 14, 2023

k8s-ci-robot merged commit 862667a into kubernetes-sigs:master Nov 14, 2023
10 checks passed

zwpaper mentioned this pull request Nov 15, 2023

Provide Context to scheduler plugins #664

Open

ffromani mentioned this pull request Nov 15, 2023

nrt: cache: add support for dedicated informer #599

Merged

migrate nrt to ctrl runtime client #655

migrate nrt to ctrl runtime client #655

Conversation

zwpaper commented Nov 1, 2023 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

netlify bot commented Nov 1, 2023 • edited Loading

✅ Deploy Preview for kubernetes-sigs-scheduler-plugins canceled.

Huang-Wei commented Nov 1, 2023

zwpaper commented Nov 3, 2023

Huang-Wei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 4, 2023

Huang-Wei commented Nov 4, 2023 • edited Loading

Tal-or left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tal-or Nov 12, 2023 • edited Loading

Choose a reason for hiding this comment

zwpaper Nov 12, 2023 • edited Loading

Choose a reason for hiding this comment

ffromani left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Huang-Wei commented Nov 10, 2023

Huang-Wei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zwpaper Nov 15, 2023 • edited Loading

Choose a reason for hiding this comment

ffromani left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Huang-Wei commented Nov 14, 2023

zwpaper commented Nov 1, 2023 •

edited

Loading

netlify bot commented Nov 1, 2023 •

edited

Loading

Huang-Wei commented Nov 4, 2023 •

edited

Loading

Tal-or left a comment •

edited

Loading

Tal-or Nov 12, 2023 •

edited

Loading

zwpaper Nov 12, 2023 •

edited

Loading

zwpaper Nov 15, 2023 •

edited

Loading