Add compression support to socket_listener

[m4ce]

Feature Request

It would be useful to have compression support in socket_listener plugin. We send metrics over UDP and the traffic could be highly optimised if we could compress them before shipping them out over the network.

Proposal:

The http plugin allows to compress the body before sending the request. Something similar for the socket_listener would be great. One could have an option called compression which could be set to the compression type (e.g. gzip).

[danielnelson]

I like this idea and I think it makes sense for similar options on most plugins that use the data_format. There is some prior art, in the amqp_consumer and amqp consumer as well as some of the HTTP plugins such as influxdb, http, etc we have a content_encoding option. Let's try to standardize around this naming and implement with the types in internal/content_coding.go:

  ## Content encoding for message payloads, can be set to "gzip" to or
  ## "identity" to apply no encoding.
  # content_encoding = "identity"

[nwneisen]

I'd like to take a look at this.

It looks like there are ~14 input plugins that use data_format besides what was mentioned. If you are interested in adding compression to them as well then I could open another issue and link PRs back to it to track the changes.

[danielnelson]

I'm unsure if we will want it for all parser inputs, it seems like some of them it may not work well with. For example, with the tail input it seems like it wouldn't work since we don't have the full file.

Let's try it with socket_listener only now, we can see how it goes and think about how it might work with other plugins.