nsqd: expand DNS records for "service discovery"

[ekristen]

In the age of service discovery where tools like Consul, SkyDNS, and other services exist to do service discovery via DNS because networks and IP address change, I think it would make sense for NSQ to take advantage of this.

This proposal is for anytime a DNS entry is given to an argument like -lookupd-tcp-address that nsq take all the records returned from the DNS query and make the additional connections automatically.

For example, if using consul and I provide -lookupd-tcp-address=nsqlookup.service.consul and the nsqlookup.service.consul has 3 IP addresses in the DNS record, nsqd should make 3 separate connections just as if -lookup-tcp-address had been given 3 times.

Thoughts?

[jehiah]

@ekristen #601 is relevant to this conversation as that PR added the ability to dynamically configure nsqd lookup addresses via HTTP. That means it's now possible to write glue code that synchronizes nsqd lookupd settings dynamically with any other 3rd party discovery mechanism.

[jehiah]

To clarify, our goal with #601 was to somewhat defer making a choice on supporting any specific service discovery methods as there is broad lack of consensus on a singular approach to do that. Instead we chose as a first step to expose the foundational capability that could be used to integrate with any discovery mechanism.

I'm not opposed to a path forward on built in support for (some) service discovery approaches. This is a good spot for discussion on that. Thoughts @mreiferson ?

[mreiferson]

@jehiah yea, I agree. For example, implementing some DNS based resolution would be reasonably agnostic to the service discovery implementation as most expose some mechanism to deliver results in that way.

[ekristen]

Well by expanding DNS it also just allows anyone who wants to use DNS, like with Route53 on AWS with a Private Zone to just setup a record that contains all nsqlookupd servers and viola. It doesn't necessarily have to be in the name of support "service discovery".

[mreiferson]

@ekristen I think we agree 😁

[ekristen]

Awesome @mreiferson +1

[harlow]

+1 to DNS discovery. This would be great addition.

[harlow]

I've created a "sidecar" app that runs alongside each nsqd instance in our cluster. It pings DNS for A records and then pushes the IP addresses to the /config/nsqlookupd_tcp_addresses endpoint.

https://github.com/harlow/nsqd-discovery

@mreiferson re: #708 I'll get a PR into nsqd docs that explains how to use the new config endpoints. They seem like a decent stop-gap until a decision is made whether or not to add DNS discovery to nsqd proper.

[mreiferson]

@harlow this is great, I'll keep an eye out for that PR!

[harlow]

@mreiferson does it make sense to add /config/nsqlookupd_tcp_addresses endpoint for nsqadmin too?

[mreiferson]

probably :)

[fixe]

+1 for adding the endpoint to nsqadmin.