Support DNS in on-demand XDS

[nmittler]

The current DNS proxy does not support on-demand mode. This is because it simply assumes that all the workloads and services it needs are already in the ProxyState.

Since DNS requests are the first thing done when making a request, the ProxyState would be empty (except for local workloads). This means that the DNS proxy will have to perform the on-demand handshake. However, the requirements for DNS on-demand differ slightly from what we have today:

1. On-demand by hostname: We'll need the on-demand system to support queries by FQDN end-to-end. The hostname can match either services or workloads. And if we find a match, we should return all related resources. For example, if the match is for a service, we should also return all workloads for that service. Similarly if the match is a workload, all associated services should be returned.
2. Batched requests: The DNS proxy will attempt a number of hostname aliases when looking up the requested host. For example, if mysvc.myns is queried, the DNS proxy will lookup all of the following until it finds a matching service: mysvc.myns, mysvc.myns.svc, mysvc.myns.svc.cluster.local.

I believe with these 2 things in place, the new flow for the DNS proxy could be something like:

1. Generate potential aliases for the requested hostname.
2. Check aliases in ProxyState. If found, done.
3. Send a batch on-demand request for all hostname aliases.
4. Check aliases in ProxyState. If found, done.
5. Forward to upstream DNS resolver.

[nmittler]

Discussed with @howardjohn offline. There is some concern regarding the needs of DNS WRT the current on-demand XDS API. Specifically:

• Since the DNS logic only has a requested hostname (no namespace), it needs a way of requesting any service/workload with a given hostname. This seems to be in violation of the XDS spec, where typically a given subscription yields a single resource.
• Since the DNS logic has to match the requested hostname to a canonical hostname, it needs to try a bunch of possible aliases. This would result in a potentially large number of unused subscriptions. While this may not be a violation of the XDS spec, we should look into the impact this would have on the system.

Need further discussion to find a path forward.