[THREESCALE-4393] Add support to use Basic Authentication with the forward proxy #1409

tkan145 · 2023-08-31T06:13:10Z

What

Fixes https://issues.redhat.com/browse/THREESCALE-4393

This PR support the use of Basic Auth with forward proxy in the following scenarios:

APIcast <> forward HTTP proxy with Basic Auth <> HTTP upstream
APIcast <> forward HTTP proxy with Basic Auth <> HTTPS upstream

Verification steps

Add the following line to examples/forward-proxy/tinyproxy.conf

BasicAuth foo bar

APIcast <> forward HTTP proxy with Basic Auth <> HTTP upstream

Update examples/forward-proxy/apicast-config.json with the following:

{                                                             
    "services": [                                             
        {                                                     
            "backend_version": "1",                           
            "proxy": {                                        
                "hosts": ["one"],                             
                "api_backend": "http://postman-echo.com",       
                "backend": {                                  
                    "endpoint": "http://127.0.0.1:8081",      
                    "host": "backend"                         
                },                                            
                "policy_chain": [                             
                    {                                         
                        "name": "apicast.policy.apicast"      
                    },                                        
                    {                                         
                        "name": "apicast.policy.http_proxy",  
                        "configuration": {                    
                            "http_proxy": "http://foo:bar@proxy:443/"
                        }                                     
                    }                                         
                ],                                            
                "proxy_rules": [                              
                    {                                         
                        "http_method": "GET",                 
                        "pattern": "/",                       
                        "metric_system_name": "hits",         
                        "delta": 1,                           
                        "parameters": [],                     
                        "querystring_parameters": {}          
                    }                                         
                ]                                             
            }                                                 
        }                                                     
    ]                                                         
}

run docker-compose.forward-proxy.yml

make forward-proxy-gateway

Get APIcast IP

❯ docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS         PORTS                NAMES
bab24ff74d69   apicast-test            "container-entrypoin…"   9 seconds ago    Up 9 seconds   8080/tcp, 8090/tcp   apicast_build_0-gateway-run-a32123385dce
dec323e010f9   nginx:1.23.4            "/docker-entrypoint.…"   10 seconds ago   Up 9 seconds   80/tcp, 443/tcp      apicast_build_0-upstream-1
a0b1ffe5b514   apicast_build_0-proxy   "/usr/bin/tinyproxy …"   10 seconds ago   Up 9 seconds   0/tcp                apicast_build_0-proxy-1

❯ APICAST_IP=$(docker inspect apicast_build_0-gateway-run-a32123385dce | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)

Send request to APICast

❯ curl -v -H "Host: one" http://${APICAST_IP}:8080/get?user_key=foo
*   Trying 127.0.0.1...                                                                                                         
* TCP_NODELAY set                                                                                                               
* Connected to localhost (127.0.0.1) port 8080 (#0)                                                                             
> GET /get?user_key= HTTP/1.1                                                                                                   
> Host: one                                                                                                                     
> User-Agent: curl/7.61.1                                                                                                       
> Accept: */*                                                                                                                   
>                                                                                                                               
< HTTP/1.1 200 OK                                                                                                               
< Server: openresty                                                                                                             
< Date: Thu, 31 Aug 2023 06:12:41 GMT                                                                                           
< Content-Type: application/json; charset=utf-8                                                                                 
< Content-Length: 370                                                                                                           
< Connection: keep-alive                                                                                                        
< Via: 1.1 tinyproxy (tinyproxy/1.11.1)                                                                                         
< ETag: W/"172-1BPDY6s8yDG5TJHwqYbKsiphJMI"                                                                                     
< set-cookie: sails.sid=s%3AnwD0yJG_9FLjXIev7zU9n_6U3CFFN5KD.QOtPZ4Jy84tU46c8%2FiYS%2FCsaolhffNCqFLeGPbcH%2FcM; Path=/; HttpOnly
<                                                                                                                               
{                                                                                                                               
  "args": {                                                                                                                     
    "user_key": ""                                                                                                              
  },                                                                                                                            
  "headers": {                                                                                                                  
    "x-forwarded-proto": "http",                                                                                                
    "x-forwarded-port": "80",                                                                                                   
    "host": "postman-echo.com",                                                                                                 
    "x-amzn-trace-id": "Root=1-64f02f59-455a8e1e1cf53ad137ba4e7a",                                                              
    "via": "1.1 tinyproxy (tinyproxy/1.11.1)",                                                                                  
    "user-agent": "curl/7.61.1",                                                                                                
    "accept": "*/*"                                                                                                             
  },                                                                                                                            
  "url": "http://postman-echo.com/get?user_key="                                                                                
* Connection #0 to host localhost left intact

We can see that the the request is going through the proxy

APIcast <> forward HTTP proxy with Basic Auth <> HTTPS upstream

Update examples/forward-proxy/apicast-config.json with the following:

{                                                             
    "services": [                                             
        {                                                     
            "backend_version": "1",                           
            "proxy": {                                        
                "hosts": ["one"],                             
                "api_backend": "https://postman-echo.com",       
                "backend": {                                  
                    "endpoint": "http://127.0.0.1:8081",      
                    "host": "backend"                         
                },                                            
                "policy_chain": [                             
                    {                                         
                        "name": "apicast.policy.apicast"      
                    },                                        
                    {                                         
                        "name": "apicast.policy.http_proxy",  
                        "configuration": {                    
                            "https_proxy": "http://foo:bar@proxy:443/"
                        }                                     
                    }                                         
                ],                                            
                "proxy_rules": [                              
                    {                                         
                        "http_method": "GET",                 
                        "pattern": "/",                       
                        "metric_system_name": "hits",         
                        "delta": 1,                           
                        "parameters": [],                     
                        "querystring_parameters": {}          
                    }                                         
                ]                                             
            }                                                 
        }                                                     
    ]                                                         
}

run docker-compose.forward-proxy.yml

make forward-proxy-gateway

Get APIcast IP

❯ docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS         PORTS                NAMES
f0b18fa16dda   apicast-test            "container-entrypoin…"   9 seconds ago    Up 9 seconds   8080/tcp, 8090/tcp   apicast_build_0-gateway-run-a32123385dce
7b3107db8aa4   nginx:1.23.4            "/docker-entrypoint.…"   10 seconds ago   Up 9 seconds   80/tcp, 443/tcp      apicast_build_0-upstream-1
f0b18fa16dda   apicast_build_0-proxy   "/usr/bin/tinyproxy …"   10 seconds ago   Up 9 seconds   0/tcp                apicast_build_0-proxy-1

❯ APICAST_IP=$(docker inspect apicast_build_0-gateway-run-a32123385dce | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)

Send request to APICast

❯ curl -v -H "Host: one" http://${APICAST_IP}:8080/get?user_key=foo
*   Trying 127.0.0.1...                                                                                                            
* TCP_NODELAY set                                                                                                                  
* Connected to localhost (127.0.0.1) port 8080 (#0)                                                                                
> GET /get?user_key= HTTP/1.1                                                                                                      
> Host: one                                                                                                                        
> User-Agent: curl/7.61.1                                                                                                          
> Accept: */*                                                                                                                      
>                                                                                                                                  
< HTTP/1.1 200 OK                                                                                                                  
< Server: openresty                                                                                                                
< Content-Type: application/json; charset=utf-8                                                                                    
< Transfer-Encoding: chunked                                                                                                       
< Connection: keep-alive                                                                                                           
< set-cookie: sails.sid=s%3ALAz2UZyLXVxBZ8VY2XjC90y0N54Mk_JV.xJ0meVF%2F1%2FEP1m7BY0SZ9jNQ%2FLaIl3jaoHHsxH%2FPIhQ; Path=/; HttpOnly 
< Date: Thu, 31 Aug 2023 06:11:54 GMT                                                                                              
< ETag: W/"146-Gpco5rr4Zu8qMK21l8EJcRbPeqI"                                                                                        
<                                                                                                                                  
{                                                                                                                                  
  "args": {                                                                                                                        
    "user_key": ""                                                                                                                 
  },                                                                                                                               
  "headers": {                                                                                                                     
    "x-forwarded-proto": "https",                                                                                                  
    "x-forwarded-port": "443",                                                                                                     
    "host": "postman-echo.com",                                                                                                    
    "x-amzn-trace-id": "Root=1-64f02f2a-2403dd5e6c8931eb1878f63e",                                                                 
    "user-agent": "curl/7.61.1",                                                                                                   
    "accept": "*/*"                                                                                                                
  },                                                                                                                               
  "url": "https://postman-echo.com/get?user_key="                                                                                  
* Connection #0 to host localhost left intact

We can see that the the request is going through the proxy

gateway/src/apicast/http_proxy.lua

t/apicast-policy-http-proxy.t

eguzki

HAve you considered tests with HTTP(S)_PROXY env var?

tkan145 · 2023-09-27T13:35:34Z

HAve you considered tests with HTTP(S)_PROXY env var?

Added tests for HTTP(S)_PROXY env var.

To discuss:

For HTTP connections sent through the proxy, the set Proxy-Authorization in the request itself. However when sent over HTTPS, the Proxy-Authorization header must be sent in the CONNECT request as the proxy has no visibility into further tunneled requests.

The camel proxy policy is interesting here as we sent over HTTPS but skip CONNECT, so I wonder what is the correct approach here

Set Proxy-Authorization header in the original request
Don't set the header and update the documentation saying that camel proxy policy is incompatible with proxy authentication

eguzki

Looking super good!

Missing documentation.

At least, I would update the readme in the policies folder.

eguzki · 2023-10-22T22:09:55Z

t/apicast-policy-http-proxy.t

+    access_by_lua_block {
+      local assert = require('luassert')
+      local proxy_auth = ngx.req.get_headers()['Proxy-Authorization']
+      assert.equals(proxy_auth, "Basic Zm9vOmJhcg==")


Not a big deal, but according to rfc7235#section-4.4,

A proxy MAY relay the credentials from the client request to the next proxy if that is the mechanism by which the proxies cooperatively authenticate a given request.

So asserting in the upstream the existence of the Proxy-Authorization maybe not be accurate.

Ideally, the test should check that the header reaches the proxy instead. Can we add assertions at the proxy?

Added documentation for http_proxy policy.

I tried extending /t/fixture/proxy.lua to enable proxy authentication checks for each test but couldn't figure out the best way to do so, I ended up printing the header in the debug log and assert the log, quick and easy.

eguzki · 2023-10-22T22:54:20Z

t/apicast-policy-camel.t

+using proxy: http://foo:bar@127.0.0.1:$TEST_NGINX_HTTP_PROXY_PORT
+
+
+=== TEST 7: using HTTPS proxy for backend with Basic Auth.


I wonder how this works with the camel proxy when upstream is TLS.

You are adding the Proxy-Authorization header to the method _connect_proxy_https. But precisely, for the camel proxies, that method is not being used. Instead, _connect_tls_direct is being used. Correct me if I am wrong.

Does it even make sense to add authentication support on camel proxy when upstream is TLS?

Now I see your comment

The camel proxy policy is interesting here as we sent over HTTPS but skip CONNECT, so I wonder what is the correct approach here Set Proxy-Authorization header in the original request Don't set the header and update the documentation saying that camel proxy policy is incompatible with proxy authentication

Regarding Set Proxy-Authorization header in the original request, IMO it does not make sense to me. The header would not even be read by camel proxy and would end up in the upstream.

Regarding Don't set the header and update the documentation saying that camel proxy policy is incompatible with proxy authentication, we may need to investigate further how camel proxy works and if they define a use case for proxy auth on TLS connections. Maybe here? https://github.com/zregvart/camel-netty-proxy They say it is only an example.

Anyway, the integration tests for the TLS upstream does not make sense as it is for now.

so I talked to few friends from the FuseSource team and no one really knows what's called a camel proxy.

The example you provided is built using Camel netty-http. And checking the source code of netty-http it looks like it doesn't support proxy authentication and CONNECT out of the box.

With that said, I believe the camel proxy's behavior will depend on the implementation.

The whole point of camel proxy is to proxy the request without CONNECT, otherwise they can just use the normal proxy.

I would vote to not support authentication with camel proxy. Let me know what you think and I will push a new patch

It is being supported feature https://access.redhat.com/documentation/en-us/red_hat_3scale_api_management/2.13/html/administering_the_api_gateway/apicast-policies#camel-service_standard-policies

IDK how many customers are using it, tho

What I would do is to add doc (a note somewhere) saying that camel proxying does not support basic client auth.

It's already in the camel proxy Readme file
https://github.com/3scale/APIcast/blob/master/gateway/src/apicast/policy/camel/Readme.md?plain=1#L65

Also fixed the camel proxy integration tests

gateway/src/apicast/policy/http_proxy/Readme.md

eguzki · 2023-10-31T13:51:07Z

gateway/src/apicast/policy/http_proxy/Readme.md


 ## Caveats

 - This policy will disable all load-balancing policies and traffic will be
  always send to the proxy. 
 - In case of HTTP_PROXY, HTTPS_PROXY or ALL_PROXY parameters are defined, this
  policy will overwrite those values. 
- Proxy connection does not support authentication.
+- 3scale currently does not support connecting to an HTTP proxy via TLS. For this reason, the scheme of the HTTPS_PROXY value is restricted to http.


3scale currently does not support connecting to an HTTP proxy via TLS. For this reason, the scheme of the HTTPS_PROXY value is restricted to http.

IMO not very accurate

3scale currently does not support connecting to an HTTP proxy via TLS

Correct.

For this reason, the scheme of the HTTPS_PROXY value is restricted to http.

All proxy URLs should be http based, never https based.

If upstream is http -> then HTTP_PROXY is used by apicast

if upstream is https -> then HTTPS_PROXY is used by apicast

All proxy URLs should be http based, never https based.

That's exactly what I meant when I said For this reason, the HTTPS_PROXY value scheme is restricted to http. Maybe that causes more confusion, maybe we should delete that sentence?

eguzki · 2023-11-06T13:00:29Z

Changes looks good to me.

I cannot approve because there are git conflicts and some integration tests are failing

tkan145 · 2023-11-07T01:15:13Z

Ack, let me rebase and review the test log. It seems like a bunch of unrelated tests also failed

Update tests to check if Proxy-Authorization reaches to the proxy server instead of asserting the header in the upstream block.

Proxy urls should all be http, https not supported

tkan145 · 2023-11-07T07:16:59Z

Conflict resolved but I'm still not sure why the tests are failing, it seems very random to me

eguzki · 2023-11-07T08:23:17Z

Conflict resolved but I'm still not sure why the tests are failing, it seems very random to me

Yeah, there are few flaky tests we should be fixing sooner or later.

After few tries, no more than 5, they should all pass

tkan145 changed the title ~~[THREESCALE-4393] Add support to use Basic Authentication with the HTTP(s) proxy~~ [THREESCALE-4393] Add support to use Basic Authentication with the forward proxy Aug 31, 2023

tkan145 force-pushed the THREESCALE-4393 branch from 9590e66 to a0131de Compare August 31, 2023 06:25

tkan145 marked this pull request as ready for review August 31, 2023 06:26

tkan145 requested a review from a team as a code owner August 31, 2023 06:26

eguzki reviewed Sep 8, 2023

View reviewed changes

gateway/src/apicast/http_proxy.lua Outdated Show resolved Hide resolved

eguzki reviewed Sep 8, 2023

View reviewed changes

t/apicast-policy-http-proxy.t Outdated Show resolved Hide resolved

eguzki reviewed Sep 8, 2023

View reviewed changes

t/apicast-policy-http-proxy.t Outdated Show resolved Hide resolved

eguzki reviewed Sep 8, 2023

View reviewed changes

eguzki requested changes Oct 22, 2023

View reviewed changes

eguzki reviewed Oct 22, 2023

View reviewed changes

eguzki reviewed Oct 23, 2023

View reviewed changes

gateway/src/apicast/policy/http_proxy/Readme.md Outdated Show resolved Hide resolved

eguzki reviewed Oct 31, 2023

View reviewed changes

An Tran added 8 commits November 7, 2023 16:50

Add support to use Basic Authentication with the forward proxy

3dc087e

Update CHANGELOG.md

6fcf36b

Correct the http header var

caa1db9

Minor refactor to avoid Proxy-Authorization leak

b6081f7

Update http_proxy documentation

adf7c93

Update integration tests when sending http request through proxy

836babc

Update tests to check if Proxy-Authorization reaches to the proxy server instead of asserting the header in the upstream block.

Update proxy url in the documentation

260dca5

Proxy urls should all be http, https not supported

Do not send Proxy-Authorization when using Camel proxy policy

4769da4

tkan145 force-pushed the THREESCALE-4393 branch from 96282ef to 4769da4 Compare November 7, 2023 06:56

eguzki approved these changes Nov 7, 2023

View reviewed changes

tkan145 merged commit eb92543 into 3scale:master Nov 8, 2023
11 of 12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[THREESCALE-4393] Add support to use Basic Authentication with the forward proxy #1409

[THREESCALE-4393] Add support to use Basic Authentication with the forward proxy #1409

tkan145 commented Aug 31, 2023 •

edited

Loading

eguzki left a comment

tkan145 commented Sep 27, 2023

eguzki left a comment

eguzki Oct 22, 2023

eguzki Oct 22, 2023

tkan145 Oct 23, 2023

eguzki Oct 22, 2023

eguzki Oct 22, 2023

eguzki Oct 22, 2023

tkan145 Oct 23, 2023

tkan145 Oct 31, 2023

eguzki Oct 31, 2023

tkan145 Nov 2, 2023

tkan145 Nov 2, 2023

eguzki Oct 31, 2023

tkan145 Nov 2, 2023

eguzki commented Nov 6, 2023

tkan145 commented Nov 7, 2023

tkan145 commented Nov 7, 2023

eguzki commented Nov 7, 2023

		using proxy: http://foo:bar@127.0.0.1:$TEST_NGINX_HTTP_PROXY_PORT


		=== TEST 7: using HTTPS proxy for backend with Basic Auth.

[THREESCALE-4393] Add support to use Basic Authentication with the forward proxy #1409

[THREESCALE-4393] Add support to use Basic Authentication with the forward proxy #1409

Conversation

tkan145 commented Aug 31, 2023 • edited Loading

Verification steps

APIcast <> forward HTTP proxy with Basic Auth <> HTTP upstream

APIcast <> forward HTTP proxy with Basic Auth <> HTTPS upstream

eguzki left a comment

Choose a reason for hiding this comment

tkan145 commented Sep 27, 2023

To discuss:

eguzki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eguzki commented Nov 6, 2023

tkan145 commented Nov 7, 2023

tkan145 commented Nov 7, 2023

eguzki commented Nov 7, 2023

tkan145 commented Aug 31, 2023 •

edited

Loading