Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

confd bypasses local proxy, all instances connect directly to single cluster member #319

Closed
fasaxc opened this issue Aug 13, 2015 · 7 comments

Comments

@fasaxc
Copy link

fasaxc commented Aug 13, 2015

I'm doing some scale testing of my application with a 5-node etcd cluster and 500+ client hosts. I have an etcd proxy on each of my client hosts. confd seems to initially connect to the local proxy but then, on subsequent requests, it connects directly to a node in the cluster. Worse, all the confd instances seem to pick the same cluster node to talk to, which causes excessive load on that node and makes it unresponsive.

# /confd --version
confd 0.9.0
@bacongobbler
Copy link
Collaborator

sounds like an issue with go-etcd, as that is the library we are using to connect.

@xiang90
Copy link

xiang90 commented Aug 13, 2015

try not to sync the cluster when you know you are reaching out a proxy.

@Lukasa
Copy link

Lukasa commented Aug 18, 2015

As @xiang90 says, I think this is strictly an issue with the way confd uses go-etcd. On L31 of backends/etcd/client.go confd calls Client.SetCluster. This, behind the scenes, syncs the cluster.

This call causes go-etcd to query the etcd cluster and find all the machines in the cluster, and then sets the cluster information to that. I think simply removing this call would resolve the problem.

I'm going to try playing around with that change, to see what happens.

@Lukasa
Copy link

Lukasa commented Aug 25, 2015

Ok, that appears to work, but we now encounter a problem when passing an authority without a scheme (i.e. just 'host:port', such as '127.0.0.1:4001'). Previously, the call to SetCluster would call go-etcd's Client.createHttpPath method, which can add a scheme to the front. This now doesn't happen. @kelseyhightower any objections to me bringing that method into confd so that I can use it here?

@geku
Copy link

geku commented Jan 26, 2016

For us this problem prevents an etcd cluster migration to new node. We have an etcd cluster and etcd-proxies, when etcd nodes get migrated to new nodes, the proxies still work and connect to the new etcd peers but confd stops working as it somehow misses the new etcd peers and returns a ERROR 501: All the given peers are not reachable.

In case the etcd would trust the proxy and not try to discover the peers itself, it would work.

@HeavyHorst
Copy link
Contributor

This should be fixed in the latest master because of the switch to the new etcd client libs.

@kelseyhightower
Copy link
Owner

This should now be fixed on master now that we are using the new etcd client libs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants