Securing connection between Envoy and upstream service: client authentication

[tsaarni]

It is possible to use TLS for the connection between Envoy and upstream server, but currently only the server is authenticated. After external client validation #2250 is introduced, it would make sense to add client authentication also to the internal TLS connection (highlighted with red dashed box in the figure).

Use case: only authorized clients shall be able to access resources of the upstream server. Requests from unauthorized clients running within the same cluster shall be rejected.

Unauthorized clients are other applications that share the same cluster. This could potentially include vulnerable software that could be remotely exploited to access the server.

Edit 2020-03-17: updated the picture to highlight that envoy should get own client certificate which is different from the certificate of the external client.

[jpeach]

Envoy can't use the client certificate from the authorized client to connect to the upstream server because it doesn't have the private key. If you need to provide the client certificate to the upstream server, then probably TCP proxy (terminating TLS at the upstream) is the right approach.

There may be some merit to issuing client certificates to each Envoy proxy and programming it to use them to establish upstream connections, but that would require the upstream service to opt in to the scheme, and I suspect that few would do that.

[tsaarni]

Using external client certificate is of course impossible, but in my case we do issue certificates internally within the cluster to secure cluster-internal communication. This follows the same principle as securing the xDS TLS connection between Contour and Envoy.

Similarly as the upstream services can be configured with server certificates (which Envoy verifies currently) in many cases they can be also configured with CA certificate to validate Envoy's certificate.

Some existing applications have that opt-in feature in their configuration. When it comes to services that users are developing themselves, in my experience it is very common for the REST/web frameworks to provide opt-in support to configure client certificate validation.

[jpeach]

OK, confirming this is the second case, where each envoy proxy uses its own client certificate to connect to the backend.

[codeyq]

+1 on this feature request. Securing out-cluster and in-cluster with different set of mTLS key certs.

[jpeach]

I think this is reasonably do-able with the recent changes to certgen and bootstrap, though it is a bit trickier if you want to have a unique client certificate for each envoy (which would be ideal). I think that we could have the bootstrapper lay down a envoy client certificate that would automatically be configured and be presented on request. The tricky part is that certgen doesn't know how many envoy instances need certificates.

[tsaarni]

In my opinion single client certificate for all Envoys should be good enough.

Although sharing credentials is of course generally discouraged, here all Envoys should be considered as replicas of the same "distributed" TLS client, all equally authorized to connect to the upstream server. Separate certificates would be needed if upstream needs to tell each Envoy apart, or if revoking individual client cert needs to be done without impacting others, which I don't think as realistic scenario either.

In practice, it would be also problematic to get each Envoy pod to mount different secret volume.

EDIT: Yet another alternative is having unique client certificate for connecting to each upstream server. In my use case, also that is unnecessary.

[codeyq]

In my opinion single client certificate for all Envoys should be good enough.

+1. One client cert for inter-cluster mTLS is sufficient in my use case but I agree one per upstream is more ideal. Not sure about the technical difficulties though.