001 design document on how to manage organizations

[NissesSenap]

Description

A design document on how to manage organizations, teams and users through a CRD.

Relevant issues/tickets

#408

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

• This change requires a documentation update
• I have added tests that prove my fix is effective or that my feature works
• I have added a test case that will be used to verify my changes
• Verified independently on a cluster by reviewer

Verification steps

[NissesSenap]

I have documented a few curl commands to play with the grafana API and after using them I actually understood how the API works.
Part of these commands could potentially be a part of future e2e tests but I didn't feel that they had a place directly in the design spec. Try your self against your own grafana instance.

Grafana API testing

HEADER='content-type:application/json'
NAMESPACE="grafana-operator-system"
PASSWORD=$(kubectl -n $NAMESPACE get secrets grafana-admin-credentials --template={{.data.GF_SECURITY_ADMIN_PASSWORD}} | base64 -d)

List org in a base setup, as we can see the name of the default org is Main Org.

$ curl -H $HEADER "http://admin:$PASSWORD@localhost:3000/api/org/"
{"id":1,"name":"Main Org.","address":{"address1":"","address2":"","city":"","zipCode":"","state":"","country":""}}

All users in current org

$ curl -H $HEADER "http://admin:$PASSWORD@localhost:3000/api/org/users/lookup"
[{"userId":1,"login":"admin","avatarUrl":"/avatar/46d229b033af06a191ff2267bca9ae56"}]

Create org

DATA='{"name":"org2"}'
curl -H $HEADER -d $DATA -X POST "http://admin:$PASSWORD@localhost:3000/api/orgs"

Create global user

DATA='{"name":"User2","email":"user2@graf.com","login":"user2","password":"userpassword","OrgId":3}'
curl -H $HEADER -X POST -d $DATA "http://admin:$PASSWORD@localhost:3000/api/admin/users"

Add the same user to the org that we created earlier to another organiaztion.

DATA='{"loginOrEmail":"user2","role":"Viewer"}'
curl -H $HEADER -d $DATA -X POST "http://admin:$PASSWORD@localhost:3000/api/orgs/2/users"

Switch to org 2

curl -H $HEADER -X POST "http://admin:$PASSWORD@localhost:3000/api/user/using/2"

Create a dashboard in org id=2 assuming that you changed your organization with the command above.

DATA='{"dashboard":{"id":null,"uid":null,"title":"Production Overview","tags":["templated"],"timezone":"browser","schemaVersion":16,"version":0,"refresh":"25s"},"folderId":0,"folderUid":"l3KqBxCMz","message":"Made changes to xyz","overwrite":false}'
curl -H $HEADER -d $DATA -X POST "http://admin:$PASSWORD@localhost:3000/api/dashboards/db"

So if in the crd you define the same username in multiple organiaztions we would have to do the following

• verify if that users exists
• If not create the user to the specefied org
• If the crd defines it in a second org
• check if the user exists
• check if that organiaztion already is defined. If not add access to that specific organiaztion

We cant allow a CR to define stuff like theme and isDisabled in multiple organiaztions since the API would start to fight against eachother.
The only thing they would be allowed to define in multiple organiaztions are role because that can differ between the orgs.

[github-actions]

This PR hasn't been updated for a while, marking as stale

[Voigtus]

So incredibly sorry for not getting back to this topic earlier.

I really like the ideas, and am still unsure, which of the two alternatives would be the better one.

In my mind I don't have a clear view, where the linkage between Dashboard and Organization should be situated. Should the Organization "search" it's Dashboards, or should the Dasbhoards search their corresponding Organization.
Right now I'm leaning to the side of Organization searching it's Dashboards via labels, but this changes kinda hourly.

What I think needs special attention is the fact, that the controllers for the new CR(s) needs to have an eye on the Grafana instance, where Orgs, Teams and Users have been deployed to, so that the CRs can be reconciled. So there is a dependency that maybe needs to be targeted in the status in the object, for example:

apiVersion: integreatly.org/v1alpha1
kind: GrafanaOrganization # Not a good name, need a better one.
metadata:
  name: organization-example
spec:
  orgniaztions:
    - name: "Main Org."
      grafanaLabelSelector: # Required
      dashboardNamespaceSelector: # Will assume same ns unless specefied
      teamsLabel: <my-team-label>
      userLabel: <my-user-label>
status:
     reconciledSuccessfully: false
     grafanaInstances:
       - name: instance-a
         applied: true
       - name: instance-b
         applied: false

So if instance-a is restarted, it needs to find be set to 'applied: false', so the controller for the GrafanaOrganization reapplies it after the instance is finished restarting. Or needs to start the reconciliation cycle for the GrafanaOrganization.

Another question is: Shall there be the possibility to add a URL or service instead of a grafanaLabelSelector, to add Organizations to non "operator-managed" instances.

[padrepitufo]

can this be re-opened? or is there further work happening elsewhere?

[NissesSenap]

@padrepitufo i reopend it. Please feel free to come with any feedback.

[NissesSenap]

Thanks for bringing this up @Voigtus. I didn't think of how dashboards (or any other resource for that matter) should be applied in a multi org solution.

To be honest the more I think about this I think it's a bad idea. To complexity this would bring to our CRD:s and the operator it self is huge compared to the current solution.
I have started to write multiple designs on how this could be solved but I don't find a way where I think that it will be easy to use.

I think we should go back to the base question which we haven't really discussed. Why do grafana need multiple organizations in a kubernetes environment?
Currently I understand it since we don't have multiple namespace-support #599 but when that is implemented what is the need?

From a really big organization point of view when it was painful to spin up new grafana servers I fully understand that you want this feature. But in a gitops world where the only thing you need to do to spin up a new grafana instance is to update your repo and you can get a grafana per team/service I don't see the need.

Switching organiaztions vs login in to multiple endpoints is about as painful from a end user point of view.

If you or someone else is able to prove me wrong I'm more then happy to look at that design proposal.

On the other side I don't see a big problem to create team/user managment from a CRD within the main organiaztion.
At the same time why do we need teams? If you are big enough to need a organiaztion you are probably big enough acutally want your own grafana instance.

So finally users, this something I think is useful since I think multiple organizations wants to be able to create a service account so they can visualize there dashboards on a TV.
But at the same time it's just one curl command and there is possible workarounds for this. #575

So please share your thoughts on why we have or haven't need for these sets of features.
We would love to get both negative and positive feedback on this.

And if you do think it's needed please create a design that will scale and still be easy to use. We are happy to talk about different solutions around this area.

[Voigtus]

In my opinion a multi-organization is still valid, as long as there is no multi-namespace and mutli-instance support.
One other part could be the "ressource saturation" part. Why spin up 10 mostly idling instances, if you can have one that has a good usage? (Playing devils advocate here. I'm right on your thought train, but can already feel the discussion and arguments.)

Thinking about users. Don't lean towards "User-management", let this be done by something that does it better, like AD, Keycloak and what else there is in the world.
What I would see as a "necessity" is the creation of a serviceaccount with read-rights either in instances or orgs.

My thoughts are currently for a prototype with incremental "feature creation"

1. This should be as "small" as possible. I.e. only creating an Org
2. Create a serviceaccount with read and/or write rights (depending on CR)
3. Then create the addition of Dashboards.
   • My thinking here is let the Dashboard be created by the current process, grab the .json by the Organization controller and let it be applied to the Organization.
   • Every Org must state in the CR which dashboards it wants to use. So no interference with the other CR, just plain API calls going all over the place.
4. Add inline Dashboards, so you can add a Dashboard to an Organization inline. This has no fancy features like subfolder. Just a plain Dashboard.
5. Add Dashboards via CM to an Organization. Just reference the configmap in the Organization and "poof" there is a dashboard. This would even allow multiple Orgs to get a set of shared dashboards. Might be the chance as well to add and test the creation of dashboards via API and not by using config files.
6. Add Dashboards via CR to an Organization.
   • Reference the Organization which it should be applied to in the GrafanaDatasource. Kind of an opt-in solution (you want a dashboard, make the concious decision and talk to the owner of the dashboard)
   • Add a field "applyToAllOrganizations" to GrafanaDatasource => default set of Dashboards for everyone (kinda like Oprah: you get a Dashboard, and you get a Dashboard...)
7. Add Datasources to Organization (kind of like step 3-6 but with Datasources)
8. Teams: scrap it. That is waaaay to "fine granular" and currently there is only the wish for Orgs to be supported. And why implement and hassle with a jack of all trades if not necessary.

Just my 2 cents. Step 3-5 can be scrapped if there is the wish to keep everything relating Dashboards in the Dashboard Controller. But I guess the best alternative is an opt-in solution for Dashboards and Datasources. Less work for us, more "clear" view in a git, where a Dashboard is coming from and who uses it.

What might be challenging in Step 6 is that the Dashboards should be reconcilable but don't create the necessity to restart the instance if these are only applied by API to the Organizations. Next step could be to switch up the whole process of Dashboard creation to API, but I don't know, wether this should be a part of this story.

[NissesSenap]

@Voigtus we are currently working on multi-namespace and mutli-instance support and it will most likely be merged before anyone starts to work on implementing this feature.

Do you think this is worth it assuming that multi-namespace and mutli-instance support is available?

About you ideas if you or someone else can come with concrete example how the CRD:s would be managed to solve this we will definitely look at it. But I won't personally take any more of my time in trying to design this in a foreseeable future. But I'm happy to discuss this feature if someone else drives the question.

[Voigtus]

I was just having a thought.
Does the problem "how do I manage the CR" not also apply to Datasources and Dashboards in a Multi-Instance Architecture?
How do you want to "marry" Dashboards with specific instances? Shall the instance have the lead, or should the CR know where it should be applied to?
Since this is the same question but with another CR my guess would be: The core mechanism could be the same for Organizations.

The by far most simplistic way to implement Organizations would be to add the following to the Grafana CRD:

spec:
  organization:
    - name: orgA
      createViewer: true
      dashboards:
        - dashboardA
        - dashboardC
      datasources:
        - datasourceA
        - datasourceB
    - name: orgB
      createViewer: false

Then after the current reconciliation cycle, the organizations are created by getting all ressources via HTTP-API of the grafana instance and then are applied to organizations.
What are the drawbacks:

• All Datasources and Dashboards need to be created first in the main-org.
• API calls to Instance to get already present information in other resources (duplication)

I have a general question regarding the way multi-instance-namespace support will be supported. Can I hit you up on slack @NissesSenap , or is there a discussion going on anywhere where I can read the current state?

[NissesSenap]

In grafana.spec we have

dashboardLabelSelector	<[]Object>
dashboardNamespaceSelector	<Object>

so we should be able to support Multi-Instance Architecture without any issues.

You can follow along with Multi-Instance work at: #599

Sure you can hit me up at slack DM or in the grafana-operator channel. But if it's a specific question about multi-namespace I would recommend to post it in the PR or in the issue: #174

[github-actions]

This PR hasn't been updated for a while, marking as stale