feat: support use index when to list by labelSelector #2287

likakuli · 2022-07-30T06:36:22Z

Signed-off-by: likakuli 1154584512@qq.com

What type of PR is this?
/kind feature

What this PR does / why we need it:
Support use index when to list object by labelSelector

Which issue(s) this PR fixes:
Fixes # #2234

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Signed-off-by: likakuli <1154584512@qq.com>

karmada-bot · 2022-07-30T06:36:27Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign rainbowmango after the PR has been reviewed.
You can assign the PR to them by writing /assign @rainbowmango in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Garrybest · 2022-08-01T02:57:36Z

pkg/util/manager/manager.go

+
+	"k8s.io/client-go/rest"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/manager"


Using controllerruntime "sigs.k8s.io/controller-runtime" here because alias.go is provided.

Garrybest · 2022-08-01T02:58:14Z

pkg/util/manager/manager.go

+}
+
+type labelIndexerManager struct {
+	manager.Manager


controllerruntime.Manager

got it, thx

Garrybest · 2022-08-01T02:58:35Z

pkg/util/manager/manager.go

+//		https://github.com/kubernetes/kubernetes/pull/109334
+//      https://github.com/kubernetes-sigs/controller-runtime/pull/1838
+func NewLabelIndexerManager(config *rest.Config, options manager.Options) (manager.Manager, error) {
+	manager, err := manager.New(config, options)


controllerruntime.NewManager

Garrybest · 2022-08-01T03:00:51Z

pkg/util/manager/manager.go

+	// register label indexer
+	ctx := context.Background()
+	for obj, indexerFunc := range typeIndexerFuncMap {
+		err := lim.Manager.GetFieldIndexer().IndexField(ctx, obj, "metadata.labels", indexerFunc)


transfer metadata.labels to a constant.

Garrybest · 2022-08-01T03:06:57Z

pkg/util/manager/manager.go

+// typeIndexerFuncMap contains internal type and its indexerFunc used to set up indexer.
+var typeIndexerFuncMap = map[client.Object]client.IndexerFunc{
+	&workv1alpha2.ResourceBinding{}:        indexForResourceBinding(),
+	&workv1alpha1.ResourceBinding{}:        indexForResourceBinding(),


I think there is no need to index workv1alpha1.ResourceBinding, workv1alpha1.ClusterResourceBinding either.

Garrybest · 2022-08-01T03:15:44Z

cmd/agent/app/agent.go

@@ -148,7 +149,7 @@ func run(ctx context.Context, karmadaConfig karmadactl.KarmadaConfig, opts *opti
 		return fmt.Errorf("failed to generate execution space name for member cluster %s, err is %v", opts.ClusterName, err)
 	}

-	controllerManager, err := controllerruntime.NewManager(controlPlaneRestConfig, controllerruntime.Options{
+	controllerManager, err := manager.NewLabelIndexerManager(controlPlaneRestConfig, controllerruntime.Options{


I don't know why we establish our own manager and cache in util/manager. I think we only need to add indexer to this controllerManager. And I have implement the respective indexer in controllerManager in https://github.com/karmada-io/karmada/blob/master/pkg/controllers/cluster/common.go

And here is the list function. The field client.Client already use a cache here. I think there is no need to establish our own cache, right? It could make the code cleaner.

karmada/pkg/controllers/cluster/taint_manager.go

Lines 68 to 74 in 616033f

if err := tc.List(ctx, rbList, client.MatchingFieldsSelector{

Selector: fields.OneTermEqualSelector(rbClusterKeyIndex, cluster.Name),

}); err != nil {

klog.ErrorS(err, "Failed to list ResourceBindings", "cluster", cluster.Name)

return controllerruntime.Result{Requeue: true}, err

}

for i := range rbList.Items {

thanks for your suggestion.

I think there are three reasons why add custom manager and cache.
There are many callers to call list func with labelSelector and the first reason is just don't want to modify too many callers.

The second is that when controller-runtime and client-go support use multi-indexer for labelSelector then what we
need to do is just use original manager and cache instead of our custom types.

The third is that it's a non-business logic. User just do what they need to do in its controller.

Combine your views, I think we don't need a new manager and only need to establish a new cache to meet @likakuli's three reasons.Because code about cache index only exists in controller-runtime/pkg/cache, don't need to edit manager.

We can add new cache's constructor in here

karmada/cmd/controller-manager/app/controllermanager.go

Lines 111 to 118 in 7cce95f

controllerManager, err := controllerruntime.NewManager(config, controllerruntime.Options{

Logger: klog.Background(),

Scheme: gclient.NewSchema(),

SyncPeriod: &opts.ResyncPeriod.Duration,

LeaderElection: opts.LeaderElection.LeaderElect,

LeaderElectionID: opts.LeaderElection.ResourceName,

LeaderElectionNamespace: opts.LeaderElection.ResourceNamespace,

LeaseDuration: &opts.LeaderElection.LeaseDuration.Duration,

Or we can directly add indexer to controllerManager using https://github.com/karmada-io/karmada/blob/master/pkg/controllers/cluster/common.go. It's ok for me.

Got it👌. We don't want to modify so many List callers, so we build our own cache which implement the List function.

Actually, I'm not sure about how much performace it will improve? I think all objects are in cache, and it seems like there have not been any perform issue so far. So is it worthy to implement our own cache and manager here? Just some questions.

Our own cache will add index of some specific labels to optimize the List from O(n) to O(1).You can see https://github.com/kubernetes-sigs/controller-runtime/blob/f46919744bee01060c9084a285e049afffd38c9d/pkg/cache/internal/cache_reader.go#L135-L162
When List with labelselector,it use range and costs O(n).We now convert labelselector to filedselector for optimization.

@Garrybest Do you think it is worth optimizing now? I want to finish it.

I think it's necessary. What we need to do is thinking about when and how to do it. Also i think we should provide some best practice for users just like k8s. How many clusters we can support? How many work we can support? e.g. Optimizing may help to increase the cap.

Glad to see this feature. Meanwhile, I hope we could change the basic cache or manager as little as possible.

cc @RainbowMango who may have some ideas about this patch.

Garrybest · 2022-08-01T03:21:29Z

/assign

yy158775 · 2022-08-01T06:17:15Z

pkg/util/manager/manager.go

+	}
+}
+
+func indexForWork() client.IndexerFunc {


When List work objects with label FederatedResourceQuotaNamespaceLabel and FederatedResourceQuotaNameLabel

karmada/pkg/controllers/federatedresourcequota/federated_resource_quota_status_controller.go

Lines 65 to 72 in ad94a94

if err := c.List(context.TODO(), workList, &client.ListOptions{

LabelSelector: labels.SelectorFromSet(labels.Set{

util.FederatedResourceQuotaNamespaceLabel: quota.Namespace,

util.FederatedResourceQuotaNameLabel: quota.Name,

}),

}); err != nil {

klog.Errorf("Failed to list workList created by federatedResourceQuota(%s), error: %v", req.NamespacedName.String(), err)

}

According to your index function,if the work object has three labels it may be cause failure.
Eg:
workobject:
[
labels:
"FederatedResourceQuotaNamespaceLabel":1,
"FederatedResourceQuotaNameLabel":2,
"ServiceNameLabel":3,
]
the index value is "FederatedResourceQuotaNamespaceLabel=1&FederatedResourceQuotaNameLabel=2&ServiceNameLabel=3",

but List function calculate the index value is "FederatedResourceQuotaNamespaceLabel=1&FederatedResourceQuotaNameLabel=2".

I'am afraid this will cause can't find the objects L in such situation.

Yes. Is this possible that work both have the three labels above?

feat: support use index when to list by labelselector

3a1d43f

Signed-off-by: likakuli <1154584512@qq.com>

karmada-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Jul 30, 2022

karmada-bot requested review from lonelyCZ and RainbowMango July 30, 2022 06:36

karmada-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 30, 2022

Garrybest reviewed Aug 1, 2022

View reviewed changes

karmada-bot assigned Garrybest Aug 1, 2022

yy158775 reviewed Aug 1, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support use index when to list by labelSelector #2287

feat: support use index when to list by labelSelector #2287

likakuli commented Jul 30, 2022

karmada-bot commented Jul 30, 2022

Garrybest Aug 1, 2022

Garrybest Aug 1, 2022

likakuli Aug 1, 2022

Garrybest Aug 1, 2022

Garrybest Aug 1, 2022

Garrybest Aug 1, 2022

Garrybest Aug 1, 2022

Garrybest Aug 1, 2022

likakuli Aug 1, 2022

yy158775 Aug 1, 2022

likakuli Aug 1, 2022

Garrybest Aug 1, 2022

yy158775 Aug 2, 2022

yy158775 Aug 5, 2022

likakuli Aug 5, 2022

Garrybest Aug 5, 2022

Garrybest commented Aug 1, 2022

yy158775 Aug 1, 2022 •

edited

Loading

likakuli Aug 1, 2022

	if err := tc.List(ctx, rbList, client.MatchingFieldsSelector{
	Selector: fields.OneTermEqualSelector(rbClusterKeyIndex, cluster.Name),
	}); err != nil {
	klog.ErrorS(err, "Failed to list ResourceBindings", "cluster", cluster.Name)
	return controllerruntime.Result{Requeue: true}, err
	}
	for i := range rbList.Items {

	controllerManager, err := controllerruntime.NewManager(config, controllerruntime.Options{
	Logger: klog.Background(),
	Scheme: gclient.NewSchema(),
	SyncPeriod: &opts.ResyncPeriod.Duration,
	LeaderElection: opts.LeaderElection.LeaderElect,
	LeaderElectionID: opts.LeaderElection.ResourceName,
	LeaderElectionNamespace: opts.LeaderElection.ResourceNamespace,
	LeaseDuration: &opts.LeaderElection.LeaseDuration.Duration,

	if err := c.List(context.TODO(), workList, &client.ListOptions{
	LabelSelector: labels.SelectorFromSet(labels.Set{
	util.FederatedResourceQuotaNamespaceLabel: quota.Namespace,
	util.FederatedResourceQuotaNameLabel: quota.Name,
	}),
	}); err != nil {
	klog.Errorf("Failed to list workList created by federatedResourceQuota(%s), error: %v", req.NamespacedName.String(), err)
	}

feat: support use index when to list by labelSelector #2287

Are you sure you want to change the base?

feat: support use index when to list by labelSelector #2287

Conversation

likakuli commented Jul 30, 2022

karmada-bot commented Jul 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Garrybest commented Aug 1, 2022

yy158775 Aug 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yy158775 Aug 1, 2022 •

edited

Loading