proxy mark an unavailable endpoint available

[gouyang]

When the endpoints unavailable, it still "marked endpoint http://10.66.15.25:2379 available", this is weird.

# etcd -proxy on  --listen-client-urls http://10.73.2.5:8002 --initial-cluster 'default=http://10.66.15.25:2380' --proxy-refresh-interval 3000 --debug
2015-09-16 13:06:35.855059 I | etcdmain: etcd Version: 2.2.0+git
2015-09-16 13:06:35.855094 I | etcdmain: Git SHA: c082488
2015-09-16 13:06:35.855098 I | etcdmain: Go Version: go1.4.2
2015-09-16 13:06:35.855107 I | etcdmain: Go OS/Arch: linux/amd64
2015-09-16 13:06:35.855113 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 24
2015-09-16 13:06:35.855117 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2015-09-16 13:06:35.855143 N | etcdmain: the server is already initialized as proxy before, starting as etcd proxy...
2015-09-16 13:06:35.855230 I | etcdmain: proxy: using peer urls [http://10.66.15.25:2380] from cluster file "default.etcd/proxy/cluster"
2015-09-16 13:06:35.857572 E | etcdmain: failed to notify systemd for readiness
2015-09-16 13:06:35.857586 I | etcdmain: proxy: listening for client requests on http://10.73.2.5:8002
2015-09-16 13:06:43.869935 I | proxy: failed to direct request to http://10.66.15.25:2379: dial tcp 10.66.15.25:2379: connection refused
2015-09-16 13:06:43.869956 I | proxy: marked endpoint http://10.66.15.25:2379 unavailable
2015-09-16 13:06:43.869966 I | proxy: unable to get response from 1 endpoint(s)
2015-09-16 13:06:44.863526 W | etcdserver: could not get cluster response from http://10.66.15.25:2380: Get http://10.66.15.25:2380/members: dial tcp 10.66.15.25:2380: connection refused
2015-09-16 13:06:44.863539 W | etcdmain: proxy: could not retrieve cluster information from the given urls
2015-09-16 13:06:44.871128 I | proxy: zero endpoints currently available
2015-09-16 13:06:45.877311 I | proxy: zero endpoints currently available
2015-09-16 13:06:46.878094 I | proxy: zero endpoints currently available
2015-09-16 13:06:47.864706 W | etcdserver: could not get cluster response from http://10.66.15.25:2380: Get http://10.66.15.25:2380/members: dial tcp 10.66.15.25:2380: connection refused
2015-09-16 13:06:47.864719 W | etcdmain: proxy: could not retrieve cluster information from the given urls
2015-09-16 13:06:47.879124 I | proxy: zero endpoints currently available
2015-09-16 13:06:48.870127 I | proxy: marked endpoint http://10.66.15.25:2379 available
2015-09-16 13:06:48.879807 I | proxy: zero endpoints currently available
2015-09-16 13:06:49.880564 I | proxy: zero endpoints currently available
2015-09-16 13:06:50.865848 W | etcdserver: could not get cluster response from http://10.66.15.25:2380: Get http://10.66.15.25:2380/members: dial tcp 10.66.15.25:2380: connection refused
2015-09-16 13:06:50.865859 W | etcdmain: proxy: could not retrieve cluster information from the given urls

# etcd -version
etcd Version: 2.2.0+git
Git SHA: c082488
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Reproduce steps:

1. run etcd cluster on one machine A
2. publish flannel network settings to etcd cluster on machine A
3. run etcd proxy to use the cluster on machine B
4. run flannel with etcd proxy on machine B
5. stop etcd cluster on machine A
6. restart flanneld on machine B
7. check etcd proxy logs on machine B

[yichengq]

It marks the endpoint available after 5s that it detected the endpoint was unavailable to give the endpoint another chance. Does this make sense to you?

The log doesn't express it clearly, and should be improved.

[gouyang]

@yichengq It's okay if it retry the endpoint after 5s, please improve the log to match that.

[xiang90]

@gouyang Can you send us a pull request to fix it? Thanks!