feat: Add decompress support for COPY INTO and streaming loading

[Xuanwo]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

In this PR, we will add decompress read support for COPY INTO and streaming leading! 🚀

This means the following query and requests will be functional:

• copy from a location/internal stage/external stage:

copy into ontime200 from '@s1' FILES = ('ontime_200.csv.gz') FILE_FORMAT = (type = 'CSV' field_delimiter = ',' compression = 'gzip'  record_delimiter = '\n' skip_header = 1);

• load via HTTP request:

curl -H "insert_sql:insert into ontime_streaming_load format Csv" -H "skip_header:1" -H "compression:zstd" -F  "upload=@/tmp/ontime_200.csv.zst" -u root: -XPUT "http://localhost:8000/v1/streaming_load"

This PR adds the basic support along with stateful tests.

We supports the following compression algorithm: gzip, bz2, zstd, brotli, deflate

There are still many works to do:

• More compression algorithm support (auto, lzo, snappy, ...)
• Performance (we will do a benchmark on the decompress load in the future)
• Code refactor (stage async source and multipart load with format trait are different logic)

But I think this PR is a good start to getting this feature works!

Changelog

• New Feature

Related Issues

Fixes #5380

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		May 29, 2022 at 3:34PM (UTC)

[mergify]

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

[sundy-li]

Impressive ！

[BohuTANG]

💯

[sundy-li]

/LGTM

[BohuTANG]

Hi @Xuanwo
It would be nice if we add the doc to the formattypeoptions option:
https://databend.rs/doc/reference/sql/dml/dml-copy#formattypeoptions

[wubx]

nice