consul/connect: add support for connect mesh gateways

[shoenig]

This PR implements first-class support for Nomad running Consul
Connect Mesh Gateways. Mesh gateways enable services in the Connect
mesh to make cross-DC connections via gateways, where each datacenter
may not have full node inter-connectivity.

Note: in this context "datacenter" means Consul datacenter.

Consul docs with more information:
https://www.consul.io/docs/connect/gateways/mesh-gateway

The following group level service block can be used to establish
a Connect mesh gateway.

service {
  connect {
    gateway {
      mesh {
        // no configuration
      }
    }
  }
}

Services can make use of a mesh gateway by configuring so in their
upstream blocks, e.g.

service {
  connect {
    sidecar_service {
      proxy {
        upstreams {
          destination_name = "<service>"
          local_bind_port  = <port>
          datacenter       = "<datacenter>"
          mesh_gateway {
            mode = "<mode>"
          }
        }
      }
    }
  }
}

Typical use of a mesh gateway is to create a bridge between DCs.
A mesh gateway should then be configured with a service port that is
mapped from a host_network configured on a WAN interface in Nomad agent
config, e.g.

client {
  host_network "public" {
    interface = "eth1"
  }
}

Create a port mapping in the group.network block for use by the mesh
gateway service from the public host_network, e.g.

network {
  mode = "bridge"
  port "mesh_wan" {
    host_network = "public"
  }
}

Use this port label for the service.port of the mesh gateway, e.g.

service {
  name = "mesh-gateway"
  port = "mesh_wan"
  connect {
    gateway {
      mesh {}
    }
  }
}

Currently Envoy is the only supported gateway implementation in Consul.
By default Nomad client will run the latest official Envoy docker image
supported by the local Consul agent. The Envoy task can be customized
by setting meta.connect.gateway_image in agent config or by setting
the connect.sidecar_task block.

Gateways require Consul 1.8.0+, enforced by the Nomad scheduler.

Closes #9446

[shoenig]

e2e tests coming in followup PR

leaving CL out until we decide if this is going in 1.1.1 or 1.1.2

[tgross]

This looks great! I left one question about the default upstreams mode which might have some code impact depending on what the right behavior is supposed to be, but other than that this LGTM.

[tgross]

The intended default behavior is a little unclear to me. It seems from the text that unset, "", and "default" should all result in using whatever Consul has set, which seems reasonable.

But we don't have the literal value "default" listed in the modes below. And the Canonicalize method in the api package looks like we're canonicalizing the unset value to "none", and the Validate method in the nomad/structs package looks like we're rejecting an unset or "default" value. Whereas the connectMeshGateway function does account for "default".

[shoenig]

Wow good catch! The source of truth here is "whatever consul does"
https://www.consul.io/docs/connect/config-entries/service-defaults#meshgateway
which is accepting, "", "none", "local", and "remote". We should have the empty string pass through, rather than canonicalize it to one of the modes, so that folks can use a Consul service-default config entry to provide a default behavior of their choosing. I'll fix the docs and the Canonicalize method.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.