autoencrypt: helpful error for clients with wrong dc

[lkysow]

If clients have set a different datacenter than the servers they're connecting with for autoencrypt, give a helpful error message. Fixes #10205

The change is to check when we're about to call ForwardRPC if the datacenter matches our current DC. There's no use-case for forwarding to a remote DC so we can short-circuit right there.

Testing & Reproduction steps

# server.hcl
server = true
bootstrap_expect = 1
data_dir = "/tmp/consul-server"
auto_encrypt {
  allow_tls = true
}
tls {
  defaults {
    ca_file = "certs/consul-agent-ca.pem"
    cert_file = "certs/dc1-server-consul-0.pem"
    key_file = "certs/dc1-server-consul-0-key.pem"
  }
}

# client.hcl
datacenter = "different"
retry_join = ["127.0.0.1:8301"]
ports {
  http = 9501
  dns = 9600
  serf_lan = 9301
}
data_dir = "/tmp/consul-client"
auto_encrypt {
  tls = true
}

Old logs:

[ERROR] agent.auto_config: AutoEncrypt.Sign RPC failed: addr=172.25.20.152:8300 error="rpcinsecure error making call: No path to datacenter"
[ERROR] agent.auto_config: No servers successfully responded to the auto-encrypt request

New logs:

2022-10-01T21:58:08.525-0700 [ERROR] agent.auto_config: No servers successfully responded to the auto-encrypt request
2022-10-01T21:59:55.456-0700 [ERROR] agent.auto_config: AutoEncrypt.Sign RPC failed: addr=127.0.0.1:8300 error="rpcinsecure: error making call: mismatched datacenter (client_dc='different' server_dc='dc1'); check client has same datacenter set as servers"

PR Checklist

• updated test coverage
• external facing docs updated
• not a security concern

[lkysow]

I thought adding :'s here would be nice because rpcinsecure is meant to indicate the area the code is in. It's not an "rpcinsecure error" which makes me think rpcinsecure is having an issue, it's an error while making an rpcinsecure call.

[ishustava]

Nice!! I definitely hit those errors in the past and it took me a while to figure out what was wrong.