security: bind HTTP and HTTPS to different ports

[schmichael]

Use Case

There are use cases where having HTTP and HTTPS with http_verify_clients simultaneously available is useful, for example:

ELB Health Checks

via @weargoggles on #2587 (comment)

ELB can run its health check with a different port and protocol to the one it uses as the backend. So:

• Consul is configured to have both HTTP and HTTPS ports
• ELB is just speaking TCP for the incoming connections
• the health check is HTTP, and the firewall is configured so that only the load balancer can access Consul's insecure HTTP port

I want to be able to do the same thing in Nomad; forward TCP connections (leaving TLS to Nomad) based on the health determined by a plain HTTP check.

I am using Consul to bootstrap and advertise. The configuration in the Consul health check you mention works perfectly for me, and it would work even if the 'skip verification' option was not set, because the certificates I'm using are valid. Perhaps that should be configurable; but I don't know that it can be controlled by verify_https_client because it's Consul being configured there.

Cons

The obvious con is that the HTTP port must be secured using external tooling and the network must be trusted for any use over HTTP.

Alternative: Allow subset of operations for unauthenticated clients

https_verify_clients could request but not require client certificates and allow a subset of operations (perhaps only a healthcheck endpoint) for requests without certificates.

While this is technically the best solution it introduces extra security checks throughout our API and may add complexity to eventual ACLs.

Implementation

TBD

[weargoggles]

I've tried running with the client verification config now in nomad master, but because I can't health-check nomad servers from a load balancer with it enabled, it's not viable right now.

I've worked around this by binding the Nomad HTTP listener to the loopback address, switching off TLS for HTTP, and offloading TLS verification to https://github.com/square/ghostunnel . Happy to write it up in more detail if anybody else needs it.

[schmichael]

@weargoggles Sorry you're blocked from using client verification. Two questions:

• Would enabling TCP liveness checks for HTTPS be a sufficient workaround?
  • I didn't implement that initially because it either spams the logs or takes some obnoxious log filtering code to prevent spamming.
• Would the alternative listed above work for you if implemented? Sadly I doubt we'll get that into 0.6.

[weargoggles]

• I think you are talking about the consul service check? If so, no, I don't see that I would get any benefit from that.
• Consul's separate listeners work well for me, and having that style in nomad would relieve me of the need to run an authenticating proxy component on each server, but it is not any sort of urgent.