-
Notifications
You must be signed in to change notification settings - Fork 267
How should I configure DNS for etcd in Tectonic Installer for DigitalOcean? #632
Comments
Update: I've tried to assign a DNS address also for my second node, but it fails to join the cluster. I see messages like the following in its log:
What am I doing wrong? |
Does the DNS records get created correctly? You should have A records for all your etcd nodes and SRV records for etcd-server and etcd-client. |
@alekssaul I think they should be working. I have A records socialfoodie-etcd-0.socialfoodie.club and socialfoodie-etcd-1.socialfoodie.club as well as SRV records _etcd-client._tcp.socialfoodie.club and _etcd-server._tcp.socialfoodie.club. What do you think of the error message? Before I assigned a DNS record to the second etcd node, it would complain about not being able to resolve its address (socialfoodie-etcd-1.socialfoodie.club), so at least that issue disappeared. |
Code in the PR looks ok to me.. The SRV records do too. Generally, you want to have odd number of etcd nodes (3 & 5 are typical HA recommendations). Did you tear down the environment and re-create it when you made the DNS changes? |
@alekssaul I guess I didn't, although I rebooted the etcd nodes. |
Can you please try to terraform destroy and terraform apply again. |
@alekssaul OK, in the process. Might take a bit of time though, since I must update the Route 53 domain registration to point to the newly created hosted zone's name servers. I wish the latter weren't necessary, but haven't seen any way of avoiding it :( |
@alekssaul I've brought up a new cluster, now the roles are reversed. The second member reports a healthy cluster ( |
@alekssaul The node that doesn't work logs messages like these:
|
Also may want to consider using; https://www.terraform.io/docs/providers/do/r/domain.html instead of Route53 since DO seem to have dns-as-a-service solution. The challenge is, DO_domain provider doesn't seem to support SRV records. You can work around it by using static discovery like; tectonic-installer/platforms/metal/cl/bootkube-controller.yaml.tmpl Lines 10 to 17 in eb3f75c
|
@alekssaul Thanks, a DO native solution would be better, but I don't think we're seeing any issues due to DNS are we? I mean, etcd complains of cluster ID mismatch. |
clusterID mismatch seem to be related to a configuration difference between the etcd cluster nodes per etcd-io/etcd#3710 . |
- remove Route53 dependency - fixes coreos#632 by changing the `--name=etcd` flag. - uses DO DNS for DNS record creation
It looks like you had |
@alekssaul Thanks :) Trying your PR now. It's funny that etcd nodes should be misconfigured, although what you're saying such as the name being non-unique makes sense, since I copied the configuration from the AWS setup. |
@alekssaul The PR works perfectly! I made some minor modifications, but overall it led to a healthy etcd cluster up and running on DO and without need for Route 53. I guess the static discovery approach of etcd cluster members could be problematic though if one should auto scale the cluster? It's not a case for me at the moment, but worth thinking about. |
Closing this for now as it works pretty well. |
I asked this originally on the mailing list, but was asked to raise an issue instead in order to reach the right people:
In implementing the Tectonic Installer for DigitalOcean, I've come across the problem that only the first etcd node comes up successfully since other nodes are not given a DNS address. I've copied this aspect from the AWS etcd module's DNS configuration.
Even though this is how it's done already in the AWS setup, I just cannot see how it's supposed to work, since each etcd node is configured as if it would have a corresponding DNS address. Can someone please enlighten me as to how etcd nodes should be configured wrt. DNS? I would appreciate any help!
The text was updated successfully, but these errors were encountered: