compression: add generic decompressor filter #11172
Conversation
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
…atted Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
Signed-off-by: Jose Nino <jnino@lyft.com>
| // A decompressor library to use for decompression. Currently only | ||
| // :ref:`envoy.compression.gzip.compressor<envoy_api_msg_extensions.compression.gzip.decompressor.v3.Gzip>` | ||
| // is included in Envoy. | ||
| config.core.v3.TypedExtensionConfig decompressor_library = 1 |
There was a problem hiding this comment.
That was my concern too: that might surprise if some options are configured separately and some are not. I'd probably prefer to configure two separate decompressor filters configured differently for different directions. But now with bufferization moved out it looks much better. I like it.
Perhaps the message's hierarchy could be flatter with request_direction_config.common_config.advertise_accept_encoding changed to just request_advertise_accept_encoding. Though I don't have strong opinion.
source/extensions/filters/http/decompressor/decompressor_filter.cc
Outdated
Show resolved
Hide resolved
source/extensions/filters/http/decompressor/decompressor_filter.cc
Outdated
Show resolved
Hide resolved
source/extensions/filters/http/decompressor/decompressor_filter.cc
Outdated
Show resolved
Hide resolved
test/extensions/filters/http/decompressor/decompressor_filter_test.cc
Outdated
Show resolved
Hide resolved
test/extensions/filters/http/decompressor/decompressor_filter_test.cc
Outdated
Show resolved
Hide resolved
test/extensions/filters/http/decompressor/decompressor_filter_test.cc
Outdated
Show resolved
Hide resolved
|
@rojkov, @rebello95 updated if you can take a look |
source/extensions/filters/http/decompressor/decompressor_filter.h
Outdated
Show resolved
Hide resolved
Signed-off-by: Jose Nino <jnino@lyft.com>
|
@mattklein123 this is now ready for you! Thanks |
| RequestDirectionConfig request_direction_config = 2; | ||
|
|
||
| ResponseDirectionConfig response_direction_config = 3; |
There was a problem hiding this comment.
Can you add docs here to explain what this config does and what happens if it is not set? Presumably if empty it doesn't decompress in the given direction?
There was a problem hiding this comment.
Yep, bears repeating here what I wrote in the runtime field.
The default is actually to decompress if empty. Do you prefer to not decompress if empty?
There was a problem hiding this comment.
I think it's fine either way I would just clearly document it as it wasn't clear to me on first read through.
| // A decompressor library to use for decompression. Currently only | ||
| // :ref:`envoy.compression.gzip.compressor<envoy_api_msg_extensions.compression.gzip.decompressor.v3.Gzip>` | ||
| // is included in Envoy. | ||
| config.core.v3.TypedExtensionConfig decompressor_library = 1 |
There was a problem hiding this comment.
FWIW it seemed odd to me that we don't allow this to be configured differently in each direction? Can you at least clarify this applies to both the request and response configuration depending on whether they are configured or not?
|
|
||
| When decompression is *applied*: | ||
|
|
||
| - The *content-length* is removed from headers. |
There was a problem hiding this comment.
Is this going to break some uses cases? Is it a future work item that we might want to buffer the decompressed data and rewrite content-length?
There was a problem hiding this comment.
Ah, sorry @mattklein123 , this context got dropped because github didn't allow me to move my original PR over.
We discussed here: https://github.com/junr03/envoy/pull/1#discussion_r411669576
And then I had this idea: #11172 (comment)
tl;dr: I originally implemented buffering here. But then decided to remove (and depend on the buffer filter) to keep this filter simpler. wdyt?
There was a problem hiding this comment.
Does the buffer filter reset content length? Anyway I think that's fine but I might make that clear in the documentation that the user might want to couple this with the buffer filter.
There was a problem hiding this comment.
Does the buffer filter reset content length?
Yep, it sets it if not present, which made it ideal for this arrangement.
Yeah I can add a section in the docs about it 👌
There was a problem hiding this comment.
Ultimately the issue is that the setup would still be a bit kludgy because of filter ordering, and the fact that the buffer filter can't be disabled independently for requests and responses. For Lyft's use case it'll be fine because we don't need to preserve content-length on responses.
So at the end of the day it might make sense to have this filter pay the price and do buffering itself. My proposal is to leave as is (with the new docs I added) to get this PR in, and then I can open an issue with a link to the commit that had buffering, it would just need to be updated a bit, and integration tests written (which shouldn't be hard at all). I can get to it as a low-pri.
There was a problem hiding this comment.
I think it's fine to document the limitation and we can revisit later.
| @@ -9,11 +9,15 @@ namespace Decompressor { | |||
| namespace { | |||
| const uint32_t DefaultWindowBits = 12; | |||
| const uint32_t DefaultChunkSize = 4096; | |||
| // When summed to window bits, this tells zlib library to decompress gzip data per: | |||
There was a problem hiding this comment.
nit: s/summed/logical ORd or something like that (even though sum is technically correct I think this is more clear)
Signed-off-by: Jose Nino <jnino@lyft.com>
Commit Message: add generic decompressor filter
Additional Description: this PR takes the design of the compressor filter, and uses the decompressor extension point to build a generic decompressor http filter. This filter can be used to decompress compressed payloads both on the request and response direction.
Risk Level: low - low as it is an extension, med - for users as this is a brand new filter.
Testing: unit tests, integration tests
Docs Changes: added docs
Release Notes: added release notes