ClusterRole's and ClusterRoleBindings should include namespace in the name

[jlewi]

See #373

ClusterRole's and ClusterRoleBinding's aren't namespaced scoped. So if we try to create multiple Kubeflow deployments in different namespaces they can interfere.

ClusterRoleBinding's are the bigger problem because we create namespace scoped service accounts, so the bindings for the different deployments are different.

If we include the namespace name in the ClusterRoleBinding this would make the names unique and allow us to better support multiple deployments.

[zjj2wry]

serviceaccount is an array. when there are multiple serviceaccount in clusterrolebindding, we do not know which partition to use.

[jlewi]

What I'm suggesting is that we create 1 clusterrolebinding per Kubeflow deployment and make the name unique so that if we have multiple Kubeflow deployments running in different namespaces there is still 1 clusterrolebinding per Kubeflow deployment.

[zjj2wry]

kubectl create clusterrole ambassador
kubectl create  clusterrolebinding ambassador --clusterrole=ambassador --sa={ns-1}:ambassador --sa={ns-2}:ambassador
kubectl create rolebinding {ns-1}-ambassador --clusterrole=ambassador --sa={ns-1}:ambassador
kubectl create rolebinding {ns-2}-ambassador --clusterrole=ambassador --sa={ns-2}:ambassador

I think using rolebinding here may be a good approach, because clusterrolebinding is also a cluster-wide resource.
kubectl get rolebinding --all-namespaces | grep ambassador

[ankushagarwal]

I tried using role bindings for ambassador, but it doesn't seem to work because by default ambassador tries to watch for services at the cluster level instead of the namespace level. When we use role binding, we can only give ambassador permissions within a single namespace and that fails.

So we might have to go with @jlewi 's proposal and create a separate clusterolebinding per deployment. eg: ns1-ambassador,ns2-ambassador where ns1 and ns2 are two kubeflow deployments on the same cluster

[ankushagarwal]

I spoke too soon, we can restrict ambassador to a single namespace by setting the env variable AMBASSADOR_SINGLE_NAMESPACE : https://www.getambassador.io/reference/configuration

[ankushagarwal]

Please take a look at #408