Using upstream hostname for request #294

dkryptr · 2017-05-12T16:40:19Z

My team and I are working on figuring out a service discovery architecture for our multitude of Serverless stacks.

I've created two EC2 instances in AWS: one with Consul and the other with Fabio. I have a Serverless stack deployed with a single endpoint: /status. I registered a service in Consul with the address being the API Gateway URL (xxxxxxx.execute-api....com) and the port to 443. I set the tag to urlprefix-/status proto=https.

Now, maybe I'm confused on how all of this works, but how come I can't do curl <fabio-ip>:9999/status? When I run that command, I get a 403 status code and a body saying bad request from CloudFront. Adding a Trace header prints Routing to service status on https://xxxxxxx.execute-api....com:443/status. If I run curl https://xxxxxxx.execute-api....com:443/status, I get back a response from my lambda function as I expect.

It seems in order for it work, I need to add a Host header matching the API Gateway URL: curl -H 'Host: xxxxxxx.execute-api....com' <fabio-ip>:9999/status. That works fine and returns back a response from my lambda function, but that defeats the whole purpose. If I have clients go to curl the api, I can't expect them to know the API Gateway URL since Fabio and Consul are meant to abstract it.

So, my main question is, what am I doing wrong? Am I thinking about this in the wrong way? Am I missing some configuration of some sort? Thanks.

The text was updated successfully, but these errors were encountered:

magiconair · 2017-05-12T17:26:25Z

That should work. You're sure that your urlprefix- tag does not contain the hostname?

mitchelldavis · 2017-05-12T17:37:55Z

To piggy back on @CGreenburg question. We're hoping to get Fabio to work like a reverse proxy (I think). HTTP -> HTTPS in this case so our users can simply hit Fabio and Fabio routes the traffic to the backend services. (An obvious use case for the tool.) However, we're stubbing our toes somewhere.

http://<fabio machine>/<service name>/<path> --> https://<service address>/<path>

I think I understand that a straight through proxy wouldn't work in this case because the traffic would need to be terminated in Fabio and then continued on to the service and back in order to handle the certificates.
We were able to get this to work if we added the Host: <service address> to the request which is prohibitive. If the software knew the host header then it could go strait there.

What are we missing? Are we fundamentally missing something on how this should all work, or missing something in the Fabio configuration?

mitchelldavis · 2017-05-12T17:47:50Z

Here is my configuration for a Serverless REST API in Fabio:

/registrar	https://<garble>.amazonaws.com:443	proto=https tlsskipverify=true	100.00%

(purposefully garbling the url there.)

I get a Bad Request from Amazon when I run even though this is the use case we're looking for...

curl http://<ip>:9999/registrar

I should also note that I'm using the latest version of the magiconair/fabio docker container to run Fabio.

magiconair · 2017-05-12T18:49:40Z

@mitchelldavis Pls open another ticket in the future. I don't mind answering several questions but I'd prefer not to morph or merge issues. You probably need to add strip=/servicename to the urlprefix tag since fabio doesn't do automatic path stripping.

dkryptr · 2017-05-12T18:59:46Z

@magiconair yes, the tag does not contain the hostname. I've noticed when running curl -v <fabio-ip>:9999/status, the Host header gets set to <fabio-ip>:9999. If I set the Host header to only the ip, I get the same result.

I registered a service pointing to www.google.com port 443 with a tag urlprefix-/google proto=https strip=/google. When I run curl -v -L <fabio-ip>:9999/google, the Host gets set to <fabio-ip>:9999 but I don't get the html from google. If I run curl -v -L -H 'Host: <fabio-ip>' http://<fabio-ip>:9999/google without the port specified in the host header, everything works like a charm.

So.... I'm unsure as to why the API Gateway URL is so picky about the Host header...

magiconair · 2017-05-12T19:00:40Z

That looks like a bug.

mitchelldavis · 2017-05-12T19:12:53Z

@magiconair Sorry to confuse. @CGreenburg sits behind me, and we're working through the same problem.

magiconair · 2017-05-12T19:23:08Z

Indeed - that should have been obvious. Sorry for pushing :)

magiconair · 2017-05-12T19:31:23Z

What does your consul service record look like?

mitchelldavis · 2017-05-12T20:04:32Z

[
    {
        "Node": "api",
        "Address": "opesqzhtvl.execute-api.us-west-2.amazonaws.com",
        "TaggedAddresses": null,
        "ServiceID": "registrar",
        "ServiceName": "registrar",
        "ServiceTags": [
            "urlprefix-/registrar proto=https strip=/registrar"
        ],
        "ServiceAddress": "",
        "ServicePort": 443,
        "ServiceEnableTagOverride": false,
        "CreateIndex": 517,
        "ModifyIndex": 517
    }
]

magiconair · 2017-05-12T20:05:43Z

Can you try setting the ServiceAddress ?

mitchelldavis · 2017-05-12T20:17:41Z

I did and same result:

[
    {
        "Node": "api",
        "Address": "opesqzhtvl.execute-api.us-west-2.amazonaws.com",
        "TaggedAddresses": null,
        "ServiceID": "registrar",
        "ServiceName": "registrar",
        "ServiceTags": [
            "urlprefix-/registrar proto=https strip=/registrar"
        ],
        "ServiceAddress": "opesqzhtvl.execute-api.us-west-2.amazonaws.com",
        "ServicePort": 443,
        "ServiceEnableTagOverride": false,
        "CreateIndex": 745,
        "ModifyIndex": 745
    }
]

magiconair · 2017-05-12T20:20:26Z

And the service is accessible under https://opesqzhtvl.execute-api.us-west-2.amazonaws.com/ ?

mitchelldavis · 2017-05-12T20:20:58Z

Yes. We can curl it directly no problem.

mitchelldavis · 2017-05-12T20:21:32Z

you should be able to curl it at: https://opesqzhtvl.execute-api.us-west-2.amazonaws.com/dev

magiconair · 2017-05-12T20:24:01Z

Can you use access logging to log the target url:

fabio -log.access.format '$upstream_request_url` -log.access.target stdout

mitchelldavis · 2017-05-12T20:26:44Z

I was doing that already actually:

"192.168.16.145 - [12/May/2017:20:24:33 +0000]
        upstream_addr: opesqzhtvl.execute-api.us-west-2.amazonaws.com:443
        upstream_host: opesqzhtvl.execute-api.us-west-2.amazonaws.com$;
        upstream_request_url: https://opesqzhtvl.execute-api.us-west-2.amazonaws.com:443/dev
        upstream_request_uri: /dev
        upstream_request_scheme: https
        upstream_service: registrar"

(That extra dollar sign on the upstream_host was a typo on the log.access.format in the properties)

magiconair · 2017-05-12T20:32:02Z

I'm getting a 403 since I don't have the access token.

$ curl -i https://opesqzhtvl.execute-api.us-west-2.amazonaws.com/dev
HTTP/1.1 403 Forbidden
Content-Type: application/json
Content-Length: 42
Connection: keep-alive
Date: Fri, 12 May 2017 20:30:45 GMT
x-amzn-RequestId: e07ebbf7-3751-11e7-a9b1-0b55e7aa1c29
x-amzn-ErrorType: MissingAuthenticationTokenException
X-Cache: Error from cloudfront
Via: 1.1 4f41781811f1a69022318a8d308fd9f3.cloudfront.net (CloudFront)
X-Amz-Cf-Id: rVQRObFNqJFaHmIOQjObLbqxZtrESrprFT4Ji_OaQvG2NPwMSgxDnA==

{"message":"Missing Authentication Token"}

mitchelldavis · 2017-05-12T20:35:48Z

That's expected, Sorry, we're not going to give you access.

magiconair · 2017-05-12T20:42:13Z

ok, late here (AMS, NL). I'll look at it probably on Monday.

mitchelldavis · 2017-05-12T20:42:36Z

No Worries. Thank you!

mitchelldavis · 2017-05-16T15:40:12Z

@magiconair, anymore thoughts on this?

magiconair · 2017-05-16T17:15:19Z

@mitchelldavis got sidetracked. I try to have a look later tonight or tomorrow morning.

mitchelldavis · 2017-05-18T12:16:07Z

@magiconair, We got a response back from the Cloud Front Engineers at AWS and it turns out that it's not working with the API Gateway because we're passing the wrong host header. In this case, we're passing the IP of the fabio machine as the host header. Cloud Front can't do anything with that host header and is expecting a header with the URL of the service we're trying to hit.

Does Fabio attempt to "re-write" the host header to the URL it's routing to?

magiconair · 2017-05-18T19:39:53Z

No, it uses the host header for routing and then passes the request on as is.

magiconair · 2017-05-18T19:53:22Z

OK, so the problem is this:

# b.com advertises 'urlprefix-/status' with Address: b.com
client -> http://a.com/status -> fabio -> http://b.com/status 'Host: a.com'

b.com is supposed to either advertise a.com/status and accept requests with that hostname or advertise /status and accept requests with any hostname.

Your case is different. You register the service without a hostname but the upstream service only accepts a specific hostname. fabio cannot do this right now.

I can see two options:

tell fabio through an option to replace the Host header with the hostname of the upstream server
tell fabio which hostname to use when making the upstream request.

I think this should only work if the upstream service advertises a route with /.

mitchelldavis · 2017-05-18T20:03:42Z

You've nailed it @magiconair. That's the issue. Do you see this as a viable pull request that won't destroy existing functionality? If so, I may be able to spend some cycles trying to get it implemented.

WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP ----------------------------------------------- This patch is work in progress and demonstrates the desired behavior and adds a test to verify it. The feature needs to be made configurable per route. Fabio does not modify the Host header when forwarding the request to the upstream server. This patch enables a mode where for a specific route fabio will use the hostname of the upstream server as the host header. ----------------------------------------------- WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP Fixes #294

magiconair · 2017-05-18T20:50:33Z

The following line to proxy/proxy_http.go adds the behavior you want but for all services. This needs to be exposed and made configurable which probably isn't difficult but I'd like to think about it a bit more. You can hack the work around in there for now to make it work.

diff --git a/proxy/http_proxy.go b/proxy/http_proxy.go
index 4056d07..2e5d84f 100644
--- a/proxy/http_proxy.go
+++ b/proxy/http_proxy.go
@@ -99,6 +99,7 @@ func (p *HTTPProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        } else {
                targetURL.RawQuery = t.URL.RawQuery + "&" + r.URL.RawQuery
        }
+       r.Host = targetURL.Host

        // TODO(fs): The HasPrefix check seems redundant since the lookup function should
        // TODO(fs): have found the target based on the prefix but there may be other

magiconair · 2017-05-18T20:51:11Z

I've pushed a branch which is WIP but contains an integration test to verify the behavior. The added line does what you want but now it needs to become configurable.

mitchelldavis · 2017-05-22T15:41:37Z

This is awesome. Thank you for looking into this!

mitchelldavis · 2017-05-24T12:23:30Z

@magiconair, I was thinking that a tag would be a great way to configure this. Just like the tlsskipverify boolean tag, we could add a usedesthostheader boolean tag that simply surrounds the code you outlined above in an if statement. Do you think that would work? Also, I'm not a GO guru, so could you give me a few hints on what needs to be updated in order for me to get this pull request started?

WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP ----------------------------------------------- This patch is work in progress and demonstrates the desired behavior and adds a test to verify it. The feature needs to be made configurable per route. Fabio does not modify the Host header when forwarding the request to the upstream server. This patch enables a mode where for a specific route fabio will use the hostname of the upstream server as the host header. ----------------------------------------------- WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP WIP Fixes #294

mitchelldavis · 2017-05-26T13:03:13Z

Thanks @magiconair, I'm currently working on a pull request. I'm having trouble getting the vault tests to pass, but we'll get it in when we can.

magiconair · 2017-05-26T13:30:14Z

Can you try with vault 0.6.4?

magiconair · 2017-05-26T13:31:26Z

The way I usually run the tests is VAULT_EXE=~/vault-0.6.4/vault make test

mitchelldavis · 2017-05-27T20:27:26Z

@magiconair, changing to vault 0.6.4 worked. Thank you so much for your guidance and I was able to run the changes against AWS API Gateway and it worked like a charm.

This patch adds support for a 'host=dst' option on a route which will trigger the proxy to use the hostname of the target host for the outgoing request instead of the one provided by incoming request. This allows fabio to act as a reverse proxy for an external site.

magiconair · 2017-05-31T21:02:26Z

merged PR #301 to master

magiconair · 2017-05-31T21:08:05Z

@mitchelldavis Thanks for this patch!

mitchelldavis · 2017-06-01T17:47:18Z

You're very welcome! What kind of timeline are we looking at to get this released in a docker container?

magiconair · 2017-06-01T20:25:07Z

Early next week

magiconair changed the title ~~Serverless architecture using Fabio and Consul~~ Using upstream hostname for request May 18, 2017

magiconair added the enhancement label May 18, 2017

mitchelldavis pushed a commit to mitchelldavis/fabio that referenced this issue May 27, 2017

Using upstream hostname for request (fabiolb#294)

a4f6717

magiconair closed this as completed May 31, 2017

This was referenced Sep 26, 2017

Propagate FQDN for TLS verification with proto=https #362

Open

Should rewrite the Host Header #75

Closed

magiconair mentioned this issue Oct 10, 2017

[feature]: proxy route option #356

Closed

Using upstream hostname for request #294

Using upstream hostname for request #294

Comments

dkryptr commented May 12, 2017

magiconair commented May 12, 2017

mitchelldavis commented May 12, 2017 • edited Loading

mitchelldavis commented May 12, 2017 • edited Loading

magiconair commented May 12, 2017

dkryptr commented May 12, 2017

magiconair commented May 12, 2017

mitchelldavis commented May 12, 2017

magiconair commented May 12, 2017

magiconair commented May 12, 2017

mitchelldavis commented May 12, 2017

magiconair commented May 12, 2017

mitchelldavis commented May 12, 2017

magiconair commented May 12, 2017

mitchelldavis commented May 12, 2017

mitchelldavis commented May 12, 2017

magiconair commented May 12, 2017 • edited Loading

mitchelldavis commented May 12, 2017 • edited Loading

magiconair commented May 12, 2017

mitchelldavis commented May 12, 2017

magiconair commented May 12, 2017

mitchelldavis commented May 12, 2017

mitchelldavis commented May 16, 2017

magiconair commented May 16, 2017

mitchelldavis commented May 18, 2017

magiconair commented May 18, 2017

magiconair commented May 18, 2017

mitchelldavis commented May 18, 2017

magiconair commented May 18, 2017

magiconair commented May 18, 2017

mitchelldavis commented May 22, 2017

mitchelldavis commented May 24, 2017

mitchelldavis commented May 26, 2017

magiconair commented May 26, 2017

magiconair commented May 26, 2017

mitchelldavis commented May 27, 2017

magiconair commented May 31, 2017

magiconair commented May 31, 2017

mitchelldavis commented Jun 1, 2017

magiconair commented Jun 1, 2017

mitchelldavis commented May 12, 2017 •

edited

Loading

mitchelldavis commented May 12, 2017 •

edited

Loading

magiconair commented May 12, 2017 •

edited

Loading

mitchelldavis commented May 12, 2017 •

edited

Loading