awss3export to support gzip

[rahnarsson]

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

No

Describe the solution you'd like

When using AWS S3 exporter it would be nice to have ability to select compression for stored objects i.e. gzip

Describe alternatives you've considered

Current workflow for storing gzipped data is to use AWS Kinesis Exporter and then through Firehose store data to s3 gzipped. But having ability to straightly upload gzipped files to S3 would drop the need to use separate Kinesis stream for just gzipping the data.

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/awss3: @atoulme @pdelewski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Sounds like a good idea to me!

For solution details, most of the changes could be done with an added config option and then changes to this method:

opentelemetry-collector-contrib/exporter/awss3exporter/s3_writer.go

Line 76 in 0a79aa8

func (s3writer *s3Writer) writeBuffer(_ context.Context, buf []byte, config *Config, metadata string, format string) error {

The AWS s3 uploader's Upload method that we're already using takes another optional argument ContentEncoding. We can specify gzip here when the config option is enabled to properly upload data.

I believe using golang's compress/gzip module, we could create a NewWriter on a new buffer, then call Write on the passed in variable buf, and lastly call Close to ensure data is written to the new buffer before uploading the newly compressed buffer contents.

We could just have an if statement for the new config option and keep existing functionality or use the gzip logic. I believe we'd want gzip disabled by default so this isn't a breaking change.

[atoulme]

What's the rotation strategy? Do we accumulate data into gzip and update in place files? Do we upload in bigger chunks? Please help understand how this would work.

[rahnarsson]

My original idea was just that if the exporter now uploads raw json files and now with the new config option there would be possibility to upload that same json but just with compressed format.

All the other logic like rotation etc. would work just like now with json writer.

[bortojac]

@atoulme @pdelewski Is this work still being considered? Would love to use this if so.

[atoulme]

It's still being considered. Contributions welcome!