Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix firewall to allow etcd client traffic between controllers #287

Merged
merged 1 commit into from
Aug 22, 2018

Conversation

dghubble
Copy link
Member

@dghubble dghubble commented Aug 22, 2018

  • Broaden internal-etcd firewall rule to allow etcd client traffic (2379) from other controller nodes
  • Previously, kube-apiservers were only able to connect to their node's local etcd peer. While master node outages were tolerated, reaching a healthy peer took longer than necessary in some cases
  • Reduce time needed to bootstrap a cluster

@dghubble
Copy link
Member Author

dghubble commented Aug 22, 2018

This was most evident from running kubectl get cs (ignore scheduler and controller manager) several times on a GCP cluster. Notice, whichever apiserver handles the request, you'll see it views its own etcd peer as healthy and the others as unreachable. In reality, each etcd peer in the 3-etcd cluster is fine.

etcd-0               Healthy     {"health":"true"}                                                                                                                                            
etcd-2               Unhealthy   Get https://yavin-etcd2.domain.com.:2379/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)   
etcd-1               Unhealthy   Get https://yavin-etcd1.domain.com.:2379/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

With the fix, the results align with other multi-master clusters.

etcd-0               Healthy     {"health":"true"}                                                                                                                                            
etcd-1               Healthy     {"health":"true"}                                                                                                                                            
etcd-2               Healthy     {"health":"true"}                                                                                                                                            

Note, you can still shutdown masters if a quorum of nodes are up and expect kubectl to work. So this doesn't directly impact availability, but was definitely undesired.

* Broaden internal-etcd firewall rule to allow etcd client
traffic (2379) from other controller nodes
* Previously, kube-apiservers were only able to connect to their
node's local etcd peer. While master node outages were tolerated,
reaching a healthy peer took longer than neccessary in some cases
* Reduce time needed to bootstrap a cluster
@dghubble dghubble merged commit e58b424 into master Aug 22, 2018
@dghubble dghubble deleted the fix-etcd-firewall branch September 3, 2018 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant