feat: make concurrency of the worker configurable

[zqzten]

What this PR does / why we need it:
As to gain better performance on different reconciliations, the concurrency of worker should be made configurable for different controllers.

In short, this PR does two things below:

• Introduce a new MaxConcurrentReconciles option to worker for inner controller concurrency settings.
• Introduce a new Concurrency section to KubeFedConfg to expose the concurrency settings to users.

Which issue(s) this PR fixes:
Fixes #1392

[k8s-ci-robot]

Welcome @zqzten!

It looks like this is your first PR to kubernetes-sigs/kubefed 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kubefed has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[zqzten]

@hectorj2f Mind take a look? Advice on further design is needed.

[swiftslee]

I'd prefer the first piece of advice. How do we define the kubefedconfig custom resource if we choose the second advice？Cause we don't know what sync controllers we will run.

[zqzten]

I'd prefer the first piece of advice. How do we define the kubefedconfig custom resource if we choose the second advice？Cause we don't know what sync controllers we will run.

I agree that configuring sync controller for each type in KubeFedConfig is not applicable, but what if we can configure that in FTC? In addition, there's still a choice between one config for all of the core controllers (sync & status) and seperated config for each core controller and higher level controller (like RSP) of kubefed.

[zqzten]

To keep it simple as well as extendable, currently I exposed the MaxConcurrentReconciles option of the two core controllers' workers to user, because they do the most work of KubeFed and their performance is subject to the scale of clusters KubeFed managed.

Other concurrency settings can alse be exposed in the Concurrency section of KubeFedConfg in the future when needed.

[zqzten]

@jimmidyson @makkes This PR is ready for review.

[zqzten]

/cc @hectorj2f

[hectorj2f]

@zqzten Could you highlight the performance benefits of this solution ? and why would you propose these changes ?

[zqzten]

@hectorj2f We are using KubeFed in some large-scale cluster management cases, such as a public cloud service. It's common that operations on some clusters may take a long time (or even timeout) and block the controller processing resources belong to other users' clusters. Increasing the concurrency of sync controller mitigated this problem because now it has chance to process other resources on the fly.

In addition, it's apparent that properly increase the concurrency of a controller can improve its performance because it can simply process more events at the same time. The MaxConcurrentReconciles is a commonly used option in controller-runtime (as its name comes from the controller-runtime library too) to improve the controller's performance.

[hectorj2f]

@zqzten Thanks for the explanation. @irfanurrehman @jimmidyson Do you have any concerns about these properties ?

[irfanurrehman]

Thanks @zqzten for doing this.
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: irfanurrehman, jimmidyson, zqzten

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [irfanurrehman,jimmidyson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment