Enhance Operator Controller to Utilize User-Provided Service Accounts for Cluster Extensions

[varshaprasad96]

Description:

Enable operator controller to utilize a service account (SA) provided by the user during the creation of the Cluster Extension (CE). This change aims to limit the operator controller's privileges, thereby eliminating the need for admin access and preventing actions on behalf of users.

Design Considerations:

The proposed change introduces complexities regarding how K8s objects are cached by the operator controller. Presently, after a helm release is applied, the controller client, (which would then utilise a SA), dynamically establishes watches for each GVKs contained in the bundle.

With the introduction of SAs, we face significant challenges concerning permissions and object caching:

1. How can we ensure that a client with a specific SA has access to informers for GVKs created by another CE?
2. This might necessitate setting up separate informers for each GVK, per each SA or do we opt for a shared cache?

Open Questions:

1. How will scalability be impacted when managing thousands of Cluster Extensions within a cluster if we have informers for each GVK per CE? Is that something we even need to worry about? (Looking into current operators installed using OLM v0 can provide an idea)

2. Caching vs. Polling: Given that the bundle objects are currently monitored for health checks, should we consider switching to a polling mechanism?
   a. How would moving to polling affect scenarios where a user deletes a specific bundle resource?

3. Configuration Options:
   a. Should we offer a choice between polling and caching based on user preference?
   b. If users are willing to accept potential delays in the reconciler’s response time to the deletion (or modification) of bundle objects in exchange for not having a shared cache, should we then prioritize polling? Alternatively, should we default to caching for users requiring immediate reaction times?

4. Can a shared cache across clients with different access permissions due to SAs raise security concerns?

Related:

• BundleDeployment: add and implement required serviceAccountName and installNamespace fields rukpak#857
• actionConfigGetter: allow custom mapping from object to rest.Config helm-operator-plugins#317

cc: @joelanford