enabling auth not supported

[junghoahnsc]

Hello,

I'm trying to enable auth by

$ etcdctl user add root
$ etcdctl auth enable

Then, all etcd pods are restarted and auth is disabled. I checked the logs from operator, it showed

time="2017-05-03T20:48:19Z" level=warning msg="health check of etcd member (http://10.96.7.65:2379) failed: etcd health probing failed for http://10.96.7.65:2379: etcdserver: user name is empty" cluster-name=etcd-cluster pkg=cluster

Should I add guest role for this health check?
And I'm not sure why auth is disabled after pods were restarted.

[xiang90]

when you disable auth, health checking will fail. we fixed this in etcd itself a while ago. after we bump etcd version, it should be resolved.

[xiang90]

well. not exactly, we probably should not use key command to do health checking.

/cc @hongchaodeng

[junghoahnsc]

I'm using etcd v3.1.6. I think guest role should be created automatically when I enabled auth:
https://github.com/coreos/etcd/blob/v3.1.6/etcdserver/auth/auth.go#L376

Then, health checking should also work? But 'guest' role doesn't seem to be created when I enabled it.

[xiang90]

@junghoahnsc etcd operator needs the "root" access for backing up etcd, and doing health checking. We need to figure out a way to assign the operator the root role.

[junghoahnsc]

oh I see.

[mcameron]

Any news on etcd auth support in etcd-operator?

[xiang90]

@mcameron Sorry. No. We do not have enough time to work on this. Do you have any interest in helping with this feature?

[raoofm]

@xiang90 @hongchaodeng any timeline for this. To me it is a show blocker.

How can anyone use etcd without enabling auth? Is it assumed that everyone will use client-cert-auth feature for authentication? How about authorization and managing roles for clients? It is a very important feature for us without which we cannot go live.

I think atleast the milestone should be defined as there are many extra features being added which are good to have but this feature is a core feature in etcd and has been parked for a few months now. Gentle request to prioritize it please.

To me, if passing back the root creds to operator requires few changes, isn't it easier to just allow to pass config to enable auth and root creds as part of bootstrap in EtcdCluster

[hongchaodeng]

OK. We can prioritize it.

[raoofm]

@hongchaodeng thank you

[raoofm]

@hongchaodeng @xiang90 any timeline for this? This is really needed.

[hongchaodeng]

Please be patient, we are already working hard in December. We need to align company goals with community goals all the time.

Apologize for the delay. If you have time, feel free to work on it.

[hongchaodeng]

@raoofm @junghoahnsc
Enabling auth will need to give etcd operator a user.
Can you provide more concrete use cases how you would setup the user, and how you would manage the passwords for etcd operator user?

[raoofm]

@hongchaodeng
I would imagine it as etcd-operator will use two predefined secrets etcdUser and etcdPassword. These secrets can be provided with the etcd-operator deployment.

Here is the flow

• etcd-operator is deployed with credentials as secrets.
• EtcdCluster is deployed.
• There are 2 possible cases here
  • auth is not enabled
    • etcd-operator will still use the credentials but those will be ignored by etcd cluster as auth is not enabled.
  • root user is created on cluster and v3 auth is enabled.
    • etcd-operator will use the credentials from secrets to interact with the cluster. Same for backup-operator.

Ofcourse, the admin who created root user and enabled v3 auth also needs to create a new user with the same credentials as defined in the etcd-operator secrets and give appropriate role to the user (not sure if it needs root role)

As for our usecase we manage our secrets as bitnami sealed secrets or it could be k8s encrytped secrets.

[hongchaodeng]

the admin who created root user and enabled v3 auth also needs to create a new user with the same credentials as defined in the etcd-operator secrets

Why not just use the certificate CN as auth user for etcd-operator and its liveness/readiness probes in each etcd pod? This way the current code will work out of box (with user/role setup properly). This will require --client-cert-auth to be set, nonetheless.

@raoofm Does your use case work with that?

[raoofm]

I think that would do. I'm wondering if roles could be defined for the CN user the same way as a normal basic auth user. I guess it is possible, need to try it out. In that case we need the role to be created by the admin manually.
What if the CN is root 😄 ?

[hongchaodeng]

What if the CN is root ?

What's the question?

[daniel-garcia]

@hongchaodeng
Has development started on this? If not, I have time to implement this feature.

Also I'm not finding a --client-cert-auth flag in etcdctl. Is this a new feature? or does it need to be implemented?

[hongchaodeng]

Hi @daniel-garcia .
This doesn't need any implementation yet. See #1051 (comment).

[raoofm]

What's the question?

I mean, if we give the CN of the client cert as root then etcd-operator will work with etcd as a root user?
So what that means is a root user can authenticate itself to etcd cluster via two means password/CN

If all this is true then either CN for etcd-operator should be always root as root role is always present by default or some admin needs to create the etcd-operator role manually for the user that is given as CN while created the client cert.

[hongchaodeng]

Hi @raoofm

if we give the CN of the client cert as root then etcd-operator will work with etcd as a root user?

Right. I would recommend to set root in CN:

etcd-operator/example/tls/certs/etcd-client.json

Line 2 in f7b24d7

"CN": "etcd client",

This would be simple to start with. We can iterate on this.

[xiang90]

@daniel-garcia

what is your planned design/implementation? can you talk about it? @hongchaodeng's suggestion is to grant the operator root role through TLS cert so that nothing is needed.

[daniel-garcia]

Initially I thought about adding the root user and password as a secret and have etcd operator manage and sync that. The liveness probe would use the provided user/pass if provided.

I misunderstood the solution by @hongchaodeng; I understand it now. I've tried the proposed solution and it appears to work, AFAICT. I am seeing a restart of each of the etcd member nodes after enabling auth but seems to be stable after that. I'll do some more testing this weekend.

[hongchaodeng]

Related: --peer-require-cn could be added as another layer of auth? This is new feature in 3.3

[daniel-garcia]

I'm using etcd 3.1.8 and at least in this version an error is returned on in cluster health check

etcdserver: user name is empty

despite the operator and client cert containing root in the CN and a root user configured.

[daniel-garcia]

Verified that the approach works with etcd 3.2.8.

[xiang90]

@daniel-garcia

probably you hit a bug in etcd. would you mind helping with documenting how to enable auth with etcd operator? The TLS requirement is reasonable. Thanks.

[daniel-garcia]

It looks like I wrote too early to say that it worked. With etcd 3.1.8, etcd operator worked in the sense that the V3 API was available for a short amount of time until the cluster health checks failed; at that point the operator kills the cluster and things stop working. With etcd 3.2.8, enabling auth works and the cluster health checks work because they present the client cert. Basic auth over SSL, however, doesn't work.

When I provide etcdctl the username/password and the CA cert (or import the CA cert to the system), the etcd server responds with:

transport: remote error: tls: bad certificate

This error looks a lot like etcd-io/etcd#8603 but I don't have sufficient evidence that they are related.

[hongchaodeng]

@daniel-garcia

Mind creating a new issue? This thread is too long.
Also share the way to reproduce it?

[raoofm]

@hongchaodeng @daniel-garcia This must be becuase of #1416

As the client-cert-auth is enabled by default on the etcd cluster by the etcd-operator, the cert presented by your client (etcdctl) is a CA cert which will fail client-cert-auth.

I raised an issue #1416 to have an option to be able to set flags on etcd. I think it is ok to enable client-cert-auth but there should be an option to disable it. Because without that option a cluster which has existing basic auth clients cannot migrate to start using etcd-operator as it will break all existing clients.

[hongchaodeng]

Thanks. Now I understand the problem.

Basic auth over SSL, however, doesn't work.

This is discussed in issue #1416 and resolved in etcd etcd-io/etcd#8584 . It is new feature coming in etcd 3.3: https://github.com/coreos/etcd/releases/tag/v3.3.0-rc.3

[raoofm]

So anybody who wants v3 auth feature should wait for #1731

[hongchaodeng]

I would encourage to try out the 3.3 RC and communicate issues