Support cross-namespace cluster operation

[colhom]

I'm thinking that it's quite critical for etcd-operator to support creating clusters in any namespace, rather than just the namespace that the operator is deployed to.

This is because the entire Kubernetes access-control mechanism (service accounts, RBAC roles) are scoped by the namespace. The operator is one class of infrastructure, and the clusters it manages are a separate class of infrastructure. Forcing them to be in the same namespace, and hence bound to the same RBAC rules and service account associations is a severe hindrance. It does not allow differentiating access to the operator deployment itself from access to etcd cluster(s). Furthermore, it does also not allow parameterizing Kubernetes API access between the operator and the etcd clusters it manages, as both are always in the same namespace.

A great example would be an infrastructure team managing the etcd-operator deployment itself and delegating operator-manged etcd cluster deployments to separate application teams. I feel this is likely to be a common use case.

Requiring that the operator run in the same namespace as all the clusters it manages makes expressing this delegation via Kubernetes roles (not to mention external PKI systems) very difficult, if not impossible.

[hongchaodeng]

This is because the entire Kubernetes access-control mechanism (service accounts, RBAC roles) are scoped by the namespace.

This is not true. RBAC is not scoped by namespace level. See https://github.com/coreos/etcd-operator/blob/master/doc/user/rbac.md . Operator should have different roles from etcd pods, even if they are within the same namespace.

…

On Fri, Mar 3, 2017 at 4:38 PM colhom ***@***.***> wrote: I'm thinking that it's quite critical for etcd-operator to support creating clusters in any namespace, rather than just the namespace that the operator is deployed to. This is because the entire Kubernetes access-control mechanism (service accounts, RBAC roles) are scoped by the namespace. The operator is one class of infrastructure, and the clusters it manages are a separate class of infrastructure. Forcing them to be in the same namespace, and hence bound to the same RBAC rules and service account associations is a severe hindrance. It does not allow differentiating access to the operator deployment itself from access to etcd cluster(s). Furthermore, it does also not allow parameterizing Kubernetes API access *between* the operator and the etcd clusters it manages, as both are always in the same namespace. A great example would be an infrastructure team managing the etcd-operator deployment itself and delegating operator-manged etcd cluster deployments to separate application teams. I feel this is likely to be a common use case. Requiring that the operator run in the same namespace as all the clusters it manages makes expressing this delegation via Kubernetes roles (not to mention external PKI systems) very difficult, if not impossible. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#859>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4NNNQwAD0KINrJg4cxiM7jXFk3rWgYks5riLKGgaJpZM4MS5wf> .

[colhom]

@hongchaodeng this is very true. What I meant to say is that imagine you were running a Kubernetes clusters shared between teams. Traditionally you, as the sysadmin, would be responsible for all the Kubernetes operators.

When a team asks for access to the cluster, you would most likely hand out a namespace. Using the namespace you can provide a nice sandbox to your users:

• common resource limits for that team (scoped by namespace)
• Kubernetes API access (this is done manually now in configuring the namespace)
  • For service accounts, via the default-token which allows API access only in that namespace.
  • For users, you would create RBAC rules which allow all new users full access to the new namespace
• Monitoring (maybe prometheus operator, retrieval scoped at top level by namespace)
• Log aggregation (hopefully someday via a logging operator)

Notice how the theme that all these things are provided by cluster-wide services but are scoped by namespace. The convention seems to be that operators traditionally run in *-system namespaces but can be expected to provide service for the whole cluster across namespaces. This is a useful abstraction, as it reflects that Kubernetes clusters are often managed by a single team, but used by many teams.

As you might expect in this deployment scenario, another service a sysadmin may want to provide is managed etcd clusters using etcd operator.

In the same way you would most likely only run a single prometheus operator or logging operator for the entire cluster, I feel there will be the expectation to architect deployments to only run a single etcd-operator to provide managed etcd clusters to all namespaces. Definitely not the only way to do things, but we need to support this model.

Part of the motivation here comes also comes from looking forward to when the etcd-operator is responsible for processing cert requests and revokes from an external PKI system. This will further complicate the deployment of a new etcd-operator as it will involve "blessing" a new certificate signing identity in the external PKI system, and then hooking that certificate request pipeline up to etcd operator. In many organizations, this is not a process to be taken lightly. I feel this further underlines the simple need to have a single etcd operator deployment, managed by a sysadmin/devops-like team, which can provide managed etcd clusters to any other namespace. If etcd operator can't run clusters outside it's own namespace, I feel this very common deployment scenario will be impossible.

\cc @brianredbeard

[xiang90]

@colhom I agree this is a nice to have for the reasons you mentioned and some other reasons I heard from other people.

[mfacenet]

I'll give this a +1, I'm using the etcd-operator to run kubernetes masters/etcd on top of an existing k8s cluster, in this use case we separate the hyperkube/etcd components into its own namespace. Specifically, it would be beneficial for us to have one operator to update/maintain/etc while utilizing it across multiple namespaces.

\cc @brianredbeard @xiang90 @colhom

[xiang90]

moved to future based on current priority.

[gjcarneiro]

Given #1777 is merged, is this still an issue, or should it be closed?