feat: improve the COPY filting copied files performance

[BohuTANG]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Reuse the StageFile info when filtering the copied files.

List 300 files and do force copy:

--rows: 1000000, size: 100*43MB
mysql> COPY into bendlog.log from @log pattern='.*[.]csv$' FILE_FORMAT = (TYPE = 'CSV'  FIELD_DELIMITER = '\t' RECORD_DELIMITER = '\n' SKIP_HEADER=1 COMPRESSION=AUTO) force=true;
Query OK, 0 rows affected (17.80 sec)

--rows: 1000000, size: 100*23MB
mysql> COPY into bendlog.log from @log pattern='.*[.]csv.gz$' FILE_FORMAT = (TYPE = 'CSV'  FIELD_DELIMITER = '\t' RECORD_DELIMITER = '\n' SKIP_HEADER=1 COMPRESSION=AUTO) force=true;
Query OK, 0 rows affected (15.68 sec)

Fixes #8574

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		Nov 1, 2022 at 2:31PM (UTC)

[Xuanwo]

We can eliminate the extra stat here by reusing the StageFileInfo we already got during list.

[BohuTANG]

We can eliminate the extra stat here by reusing the StageFileInfo we already got during list.

That's cool, and if you have an idea, I plan to continue this parallel after your PR.
From my test, this parallel will hang, I will address them tomorrow :/

[Xuanwo]

That's cool, and if you have an idea, I plan to continue this parallel after your PR.

I will implement this tomorrow~

[BohuTANG]

Test again. This PR seems no help(:).
This PR:

mysql> COPY into bendlog.log from @log pattern='.*[.]csv.gz$' FILE_FORMAT = (TYPE = 'CSV'  FIELD_DELIMITER = '\t' RECORD_DELIMITER = '\n' SKIP_HEADER=1 COMPRESSION=auto) force=true;
Query OK, 0 rows affected (17.36 sec)

main:

mysql> COPY into bendlog.log from @log pattern='.*[.]csv.gz$' FILE_FORMAT = (TYPE = 'CSV'  FIELD_DELIMITER = '\t' RECORD_DELIMITER = '\n' SKIP_HEADER=1 COMPRESSION=auto) force=true;
Query OK, 0 rows affected (16.57 sec)

[BohuTANG]

That's cool, and if you have an idea, I plan to continue this parallel after your PR.

I will implement this tomorrow~

I have got the point, let me have try

[BohuTANG]

@Xuanwo Done

[Xuanwo]

Great!

[yufan022]

A marked improvement！

before this PR:

COPY INTO import   FROM 's3://xx/10000-100/'   credentials=(aws_key_id='xx' aws_secret_key='xx')   pattern ='.*[.]tsv'   file_format = (type = 'tsv')   force=true;
Query OK, 0 rows affected (1 min 30.55 sec)

after this PR:

COPY INTO import   FROM 's3://xx/10000-100/'   credentials=(aws_key_id='xx' aws_secret_key='xx')   pattern ='.*[.]tsv'   file_format = (type = 'tsv')   force=true;
Query OK, 0 rows affected (12.19 sec)