Proposal: Cluster Authorization

[everpeace]

What type of PR is this?

/kind design

What this PR does / why we need it:

This PR adds a simple design proposal of Cluster Authorization which can enforce that karmada-agents runs with least privilege in control plane as much as possible

Which issue(s) this PR fixes:

Ref ##4485 (comment)

Special notes for your reviewer:

This is a rough draft. I would be glad to have your active feedbacks.

Please also refer to my working POC: #4948

Does this PR introduce a user-facing change?:

[karmada-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign garrybest after the PR has been reviewed.
You can assign the PR to them by writing /assign @garrybest in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[RainbowMango]

Thanks @everpeace
cc @XiShanYongYe-Chang

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (fa4d6d3) 51.93% compared to head (8ac9628) 51.92%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4593      +/-   ##
==========================================
- Coverage   51.93%   51.92%   -0.01%     
==========================================
  Files         248      248              
  Lines       24634    24634              
==========================================
- Hits        12793    12791       -2     
- Misses      11144    11145       +1     
- Partials      697      698       +1     

Flag	Coverage Δ
unittests	51.92% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[XiShanYongYe-Chang]

Hi @everpeace, I'm sorry for replying late.
Thanks for your wonderful proposal!

I have read it once and feel that this is a great proposal. There are some parts where I still need to further understand before asking you questions.

It seems that it needs to rely on the integration and release of kubernetes-sigs/controller-runtime#2664 before it can continue to advance the current proposal.

Ask @chaosi-zju to help take a look with this proposal.

[everpeace]

It seems that it needs to rely on the integration and release of kubernetes-sigs/controller-runtime#2664 before it can continue to advance the current proposal.

Yes, my POC branch relies on the unmerged/unreleased code of controller-runtime. So, if we can't wait for controller-runtime, could we use pkg/util/lifted temporary??

[XiShanYongYe-Chang]

@everpeace thanks for your quick response!

Yes, my POC branch relies on the unmerged/unreleased code of controller-runtime. So, if we can't wait for controller-runtime, could we use pkg/util/lifted temporary??

Now we don't need to rush to decide how to do it specifically. We can let everyone take a look at this proposal first.

In addition, may I ask if you are currently using or planning to use Karmada?

[everpeace]

@XiShanYongYe-Chang

Now we don't need to rush to decide how to do it specifically. We can let everyone take a look at this proposal first.

Sure.

In addition, may I ask if you are currently using or planning to use Karmada?

Yeah, it's still pre-POC phase actually, though. I'm looking into Karmada to sync/distribute our custom resources (for ML training jobs) into multiple kubernetes clusters across multiple clouds, multiple regions, and on-prems. Particularly, on-prem clusters are managed independently by multiple project teams (not platform teams). So, that's(on-prem part) the reason why I want make karmada-agent with the least privilege in control plane.

Moreover, from the view point of ML training platform, I would like to introduce "Bring your own cluster" model into our platform. Thus, I will open an issue to make karmada-agent works only　for selected namespaces in control plane as my next step.

[XiShanYongYe-Chang]

my POC branch depends on the controller-runtime PR which supports authorization webhook: kubernetes-sigs/controller-runtime#2664

Hi @everpeace Is there any progress on this mission?

[everpeace]

Is there any progress on this mission?

Unfortunately no... Nobody gave a review on my PR(kubernetes-sigs/controller-runtime#2664) in a month.

I'm wondering how to promote my PR in the controller-runtime community.

[XiShanYongYe-Chang]

In that case, should we consider using the alternative first?

[everpeace]

should we consider using the alternative first?

Do you mean Generating fine-grained RBAC entries for karmada-agent? Controlling agent authorization via RBAC is not clean and easy to be buggy　(because Role/ClusterRole are resource name dependent). So, in my personal view, I would not recommend this alternative, honestly.

So, if we can't wait for controller-runtime, could we use pkg/util/lifted temporary??

could we use lifted temporarily?? Authroization webhooks are already supported in kube-apiserver. The problem is only in library (controller-runtime).

[XiShanYongYe-Chang]

Let's listen to @RainbowMango 's opinions.

[everpeace]

kubernetes-sigs/controller-runtime#2664 (comment) gave dismissive feedback, unfortunately. This means, my PR would not be merged into container-runtime. So, let me to put authorization webhook implementation inside of karmada.

[XiShanYongYe-Chang]

So, let me to put authorization webhook implementation inside of karmada.

I'm not sure if this implementation is appropriate in Karmada. Originally I wanted to use the features provided by upstream directly. Let me understand. Can we consider alternatives in parallel?

[everpeace]

Originally I wanted to use the features provided by upstream directly.

Yeah, I wanted it, too. But upstream would reject hosting aurhotization webhook implementation code in controller-runtime repo😿.

Can we consider alternatives in parallel?

Sure, absolutely. Let me update my POC branch first then. But, I would like to insist to authroization webhook solution instead of generating RBACs because RBAC solution is not clean and easy to regress when agent code was updated.

[everpeace]

@XiShanYongYe-Chang I updated my poc branch and opened a WIP PR just to make our discussion easier: #4948. I will close #4948 once this proposal agreed. Please take a look.

kubernetes authorization webhook handler which depends on controller-runtime is implemented in pkg/util/authorization_webhook package. see this commit: a009de6.