Intercept and respond to DNS queries for Virtual Services using Envoy's DNS filter

[bcelenza]

Tell us about your request

As the owner of a service in App Mesh, I would like to use simple hostnames to call my dependencies.

For example, if I have a dependent service registered by the name database.foo.mesh.local in my Route 53 private hosted zone, I'd like to define that service using the following App Mesh resources.

First, I create the VirtualNode which uses the fully-qualified domain name (FQDN) for it's service discovery setting.

$ aws appmesh create-virtual-node --mesh-name foo \
    --virtual-node-name database-node \
    --spec '{ "serviceDiscovery": { "dns": { "hostname": "database.foo.mesh.local" }}}'

Then, I create a VirtualService named after the simpler hostname, and use the previously defined VirtualNode as the provider for that VirtualService.

$ aws appmesh create-virtual-service --mesh-name foo \
    --virtual-service-name database \
    --spec '{ "provider": { "virtualNode": { "virtualNodeName": "database-node" }}'

For dependent services who specify this VirtualService as a backend, they should be able to make a request to http://database successfully.

Which integration(s) is this request for?
This could be Fargate, ECS, EKS, EC2, Kubernetes, something else.

All

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Allowing services to be called by simple hostnames instead of the registered FQDN allows service owners to reduce complexity in naming for their applications, particularly as their application spans multiple stages, AWS accounts, etc. where the private hosted zone namespace may change, but the hostnames are the same.

The reason this does not work today is that the hostname by itself does not automatically resolve to an IP address in DNS. While you can successfully resolve database.foo.mesh.local due to the DNS A records in the Route 53 private hosted zone, database is not resolvable by default.

This same issue would exist for any fully qualified domain name (FQDN) which is not resolvable by DNS, as indicated in #71.

Are you currently working around this issue?

One way to work around this issue to to bundle a fake DNS entry in the /etc/hosts file for the ECS container, EC2 instance, etc. For example, in /etc/hosts you may define an entry for the above example like:

10.10.10.10 database

The IP address added to /etc/hosts does not matter so long as it's not a local loopback address. The entry in /etc/hosts will allow the calling application to resolve to an IP address and attempt to send an HTTP request to that IP address, which will be intercepted by Envoy Proxy where the Host header will be matched and the request forwarded to the endpoints that resolve for the VirtualNode service discovery settings.

If you're using a FQDN, you can setup a Route53 private hosted zone for your VPC with an A record pointing to a standard non-loopback IP address (e.g. 10.10.10.10).

Additional context

Envoy Proxy does not currently handle UDP based traffic, and cannot intercept DNS queries. This issue is currently marked as blocked on:

1. UDP proxying support in Envoy
2. DNS filtering support in Envoy

[rizblie]

@bcelenza Can you please clarify that the intent of this feature request is that the name of the virtual service should not need to be resolvable in DNS? IMHO the title and description do not make this clear. Thanks.

If you can confirm this, then we can reject+close #330 .

[bcelenza]

Hey @rizblie , the original intent for this issue was to use Envoy's DNS responder capabilities to intercept and respond to DNS queries for Virtual Services in App Mesh. You're right that the title and description aren't super clear in that regard. I'm no longer part of this project at AWS, so I'll leave it up to others on the team to potentially rename the issue.

[herrhound]

I updated the name of the issue to match to the original intent, and also assigned it to the roadmap project as we are actively researching it.

[imileti]

Hi @rajal-amzn, just wondering if anyone can give a little update on this issue? The proposed workarounds don't scale nicely with creating new services. Tnx

[rajal-amzn]

@imileti We are working on enabling Envoy's DNS filter in AppMesh right now. Though we cannot commit to a specific date here, you can expect updates in the next quarter. And I will post the proposal here in a couple of days.

[imileti]

@rajal-amzn amazing, thank you.

[rajal-amzn]

Hey all,
We are adding the following fields to enable intercepting DNS queries in App Mesh.

• DNS mode: This would be used to specify whether DNS requests need to be intercepted by the Envoy proxy. This can be specified at the Mesh level which means that the changes are applied to all Virtual Services in the mesh. When this option is enabled, all Virtual Services would get a default DNS value. This DNS mode can be overridden per Virtual Service.
• DNS discovery: This would be used to override the default DNS value set when DNS mode is set to intercept for a particular Virtual Service.

API Model

Following represents the updated API model for Mesh.

{
    "type" : "AWS::AppMesh::Mesh",
    "properties": {
        "meshName": "SampleMesh",
        "spec" : {
            # (*NEW*) (OPTIONAL) Specifies whether to generate DNS entry for all Virtual Services in the mesh. 
            "dns": {
                # (*NEW*) (REQUIRED) Allowed values are INTERCEPT/DISABLE. Default value is DISABLE
                "mode": "INTERCEPT"
            }
        }
    }
}

Following represents the updated API model for Virtual Services.

{
    "type" : "AWS::AppMesh::VirtualService", 
    "properties" : 
    { 
        "meshName" : "SampleMesh", 
        "virtualServiceName" : "SampleVirtualService",
        "spec" :
        {
            ...
            # (*NEW*) (OPTIONAL) Specifies the DNS config for this Virtual Service
            "dns" : {
                # (*NEW*) (OPTIONAL)  Overrides the DNS mode for this particular VirtualService
                "mode": "INTERCEPT" 
                # (*NEW*) (OPTIONAL)  Specifies the Service Discovery for this particular VirtualService
                # Valid only when mode is INTERCEPT for this VirtualService.
                "discovery": { 
                    # (*NEW*) (OPTIONAL) Atleast one of ipv4Address or ipv6Address must be specified
                    "ipv4" : [  
                        "10.10.0.1"
                    ],
                    "ipv6": [ 
                        "2001:8a:c1::2800:7"
                    ]
                }
            }
        }
    }
}

When DNS is set to INTERCEPT, Envoy would be configured to answer DNS requests for the specified Virtual Services in the mesh based on either the default DNS value or the overridden value. In addition to this, we also require changes to the routing table to redirect DNS requests to Envoy. To enable this, we are proposing the following changes to the ECS Proxy Configuration.

ProxyConfiguration:
        Type: 'APPMESH'
        ContainerName: 'envoy'
        ProxyConfigurationProperties:
          - Name: 'ProxyIngressPort'
            Value: '15000'
          - Name: 'ProxyEgressPort'
            Value: '15001'
           # (*NEW*) (Optional) Specifies the Envoy port where DNS requests are redirected to
          - Name: 'ProxyDNSPort'  
            Value: '15002'
           # (*NEW*) (Optional) Specifies the protocol of the DNS requests that need to be redirected. Currently only UDP protocol is supported in Envoy.
          - Name: 'ProxyDNSProtocols' #Optional 
            Value: 'UDP' # Comma separated values

We'll be updating this issue once we have the feature enabled in our Preview Channel. Until then, we'd love to hear your feedback on this proposal in the comments.

Thanks!

[whitelancer]

We'd love to see this finally make its way into production/live! Having the ability to customize simpler DNS names is critical to our goals.

[schealex]

any update on this? We are trying to get the dns resolution for virtual services working aswell but it's quite a task currently :/