Removing Accept-Encoding header

[timrwilliams]

I'm running goproxy as a simple HTTP & HTTPS proxy, just relaying requests and responses as is without modifying or inspecting the content.

I've noticed that you're stripping out the Accept-Encoding header on all requests presumably to allow us to inspect the response without having to decode it. Is it feasible to leave this header intact if we only want to relay the response back to the client?

Essentially I want to allow a client to specify a header of "Accept-Encoding: gzip, deflate" and pass this through to the end server.

[elazarl]

I do not do that, it is the Go http server that does, since it want to
decode the gzip'd stream itself.

If you want to be a fully transparent proxy, why do you need goproxy at all?

I'm not saying this is not a good idea, I'm just trying to understand the
use case.

On Mon, May 18, 2015 at 4:19 PM, Tim Williams notifications@github.com
wrote:

I'm running goproxy as a simple HTTP & HTTPS proxy, just relaying requests
and responses as is without modifying or inspecting the content.

I've noticed that you're stripping out the Accept-Encoding header on all
requests presumably to allow us to inspect the response without having to
decode it. Is it feasible to leave this header intact if we only want to
relay the response back to the client?

Essentially I want to allow a client to specify a header of
"Accept-Encoding: gzip, deflate" and pass this through to the end server.

—
Reply to this email directly or view it on GitHub
#102.

[timrwilliams]

I'm collecting stats on traffic as it passes through the proxy so we inspect the headers on the way in and out but not the body content itself.

I saw the header deletion here and assumed that you'd made a design decision to remove it:
https://github.com/elazarl/goproxy/blob/master/proxy.go#L79

[timrwilliams]

I've looked in to it a bit further and understand what you mean now, I didn't realise transport.go was just the standard golang library.

I've commented out the header deletion line in proxy.go in my local copy of your repo and its working fine for my use case. All your test cases pass too.

Looking at a few other proxy implementations and it seems a common approach is to offer a boolean configuration option that will remove or preserve this header.

I'm happy using my modified fork if you don't think my use case justifies a change to core code.

[elazarl]

I'm really not sure what to do.

The problem is, not removing these header might cause problems. For example
client might accept an uncommon encoding, not supported by Go. When the
user would send it, he wouldn't be able to read the body.

Maybe we can keep the original headrs, and add a hook that would allow you
to add them back before sending them to the server.

What do you think?

On Mon, May 18, 2015 at 6:53 PM, Tim Williams notifications@github.com
wrote:

I've looked in to it a bit further and understand what you mean now, I
didn't realise transport.go was just the standard golang library.

I've commented out the header deletion line in proxy.go in my local copy
of your repo and its working fine for my use case. All your test cases pass
too.

Looking at a few other proxy implementations and it seems a common
approach is to offer a boolean configuration option that will remove or
preserve this header.

I'm happy using my modified fork if you don't think my use case justifies
a change to core code.

—
Reply to this email directly or view it on GitHub
#102 (comment).

[timrwilliams]

I don't think removing and re-adding the Accept-Encoding header would work as it changes how a client and server encode the response so can't be added on at the end.

To be honest I don't know enough of the inner workings of the Go transport layer to know what edge cases could cause my approach to go wrong.

I send a decent amount of real world traffic from different applications to different servers through my proxy so what I could do is put this change out there on one of my instances and see how it behaves in the wild. We can mark this issue as pending or closed and come back to it in the future if its an issue for other people or if I see any adverse results.

I really appreciate you looking in to this by the way.

[rakoo]

For example client might accept an uncommon encoding, not supported by Go. When the user would send it, he wouldn't be able to read the body.

If the client sends an uncommon Accept-Encoding, is he not supposed to be able to read it ?

I'd like to implement a SDCH proxy that automatically compress target pages. In order to do that I need to know that the client sent an Accept-Encoding of sdch, which I can't in the current state because goproxy removes the header. I'm going to do it in a OnResponse since this is where I have access to the target's response and the original request. It would be super duper if I could access the original request's header there, so I can modify the response if needed.

(Thanks for goproxy, it's really great !)

[bls]

Hi, have been using goproxy for a while as a generic MITM proxy, and this causes issues for me for AWS API clients behind the proxy.

The AWS v4 signature scheme includes header signing; some AWS clients send an "Accept-Encoding" of "identity". This gets removed by the proxy and replaced with a value of "gzip". This breaks the signature and causes client requests to fail.

[elazarl]

Blair, we have to fix it then. I'll look into it today.

…

On Tue, Jan 8, 2019, 07:07 Blair Strang ***@***.*** wrote: Hi, have been using goproxy for a while as a generic MITM proxy, and this causes issues for me for AWS API clients behind the proxy. The AWS v4 signature scheme includes header signing; some AWS clients send an "Accept-Encoding" of "identity". This gets removed by the proxy and replaced with a value of "gzip". This breaks the signature scheme and causes client requests to fail. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#102 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAP4otzMK7U5ioSv8Ljpw08E4il3NWCIks5vBCd0gaJpZM4Ee1fo> .

[bls]

Hi! I have submitted a PR for this although it may not be what you want since I can see you were thinking of a more general solution.

On the other hand, the Accept-Encoding handling seems to be the only real place the proxy is meaningfully "non transparent" with respect to requests.

With KeepAcceptEncoding enabled and DisableCompression enabled on the proxy's client transport, I can run a complex "terraform" command through the proxy (hundreds of AWS resources of many different types) with no signature verification problems.

Thanks for goproxy!

[ErikPelli]

Added the KeepAcceptEncoding option in #585