bug: COPY INTO CPU load takes a long time to rise

[yufan022]

Search before asking

• I had searched in the issues and found no similar issues.

Version

v0.8.89-nightly

What's Wrong?

COPY INTO import FROM 's3://xx/xx/100000-100/' credentials=(aws_key_id='xx' aws_secret_key='xx') pattern ='.*[.]gz' file_format = (type = 'tsv' compression
= GZIP) force=true;
Query OK, 0 rows affected (2 min 22.85 sec)

The s3 path /100000-100/* only has 100 files.

At the beginning of the command execution, I observed a very low cpu load:

It took about 60 seconds for the CPU load to rise:

i checked info logs during the period of low CPU load.

got alot of this

2022-11-01T06:47:01.463339Z  INFO opendal::services::s3::backend: backend build started: Builder
2022-11-01T06:47:01.463371Z  INFO opendal::services::s3::backend: backend use root
2022-11-01T06:47:02.067443Z  INFO opendal::services::s3::backend: backend build finished: Builder
2022-11-01T06:47:01.463339Z  INFO opendal::services::s3::backend: backend build started: Builder
2022-11-01T06:47:01.463371Z  INFO opendal::services::s3::backend: backend use root
2022-11-01T06:47:02.067443Z  INFO opendal::services::s3::backend: backend build finished: Builder
2022-11-01T06:47:01.463339Z  INFO opendal::services::s3::backend: backend build started: Builder
2022-11-01T06:47:01.463371Z  INFO opendal::services::s3::backend: backend use root
2022-11-01T06:47:02.067443Z  INFO opendal::services::s3::backend: backend build finished: Builder
...

How to Reproduce?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[BohuTANG]

Ping @Xuanwo for the first-glance troubleshooting. Many opendal::services::s3::backend logs is by design?

[Xuanwo]

Ping @Xuanwo for the first-glance troubleshooting. Many opendal::services::s3::backend logs is by design?

Yes.

I'm guessing they are plenty of read (which is in debug) logs here. However, it's still worth to reuse the same operator to reduce a bit of cost.

[yufan022]

Ping @Xuanwo for the first-glance troubleshooting. Many opendal::services::s3::backend logs is by design?

Yes.

I'm guessing they are plenty of read (which is in debug) logs here. However, it's still worth to reuse the same operator to reduce a bit of cost.

Is it concurrent? The log will last for one minute, the speed is not fast, and there is not much network traffic during this period.

[Xuanwo]

Is it concurrent? The log will last for one minute, the speed is not fast, and there is not much network traffic during this period.

Most of the time is consumed on parsing CSV content. We will improve it at #8486.

[Xuanwo]

I found the root cause of this issue.

https://github.com/datafuselabs/databend/blob/2dfd578d234fe4ea1f65c432b60bca37ec8b4095/src/query/service/src/interpreters/interpreter_copy_v2.rs#L101-L181

This call requires all files to be stat which is not needed. I will improve this.

[BohuTANG]

I will try to fix in #8586, cc @Xuanwo