feat: copy files order by last modified time asc

[BohuTANG]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Copy files in asc order of file last modification time.

Closes #issue

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		Nov 3, 2022 at 1:07PM (UTC)

[Xuanwo]

Why we need this change? After stream copy implemented, we can't do sort like this anymore.

[BohuTANG]

Why we need this change? After stream copy implemented, we can't do sort like this anymore.

If we have many files need to copy, let the oldest files fisrt, we can keep the insert order same as the file modified order.

[Xuanwo]

Doesn't make sense to me. But worth a try.

[PsiACE]

It feels possible to add a switch for it if necessary.

[Xuanwo]

It feels possible to add a switch for it if necessary.

We can't sort the files if there are 10W files, we will copy them in stream which will in the order returned by list (in the near future).

[PsiACE]

We can't sort the files if there are 10W files, we will copy them in stream which will in the order returned by list (in the near future).

So I suggest leaving it up to the user to decide whether to use this change

set order_by_last_modified_time = 1

[BohuTANG]

1. Each file size is larger, for example, 100GB
2. We have a lot of these files like 1), and it's being produced all the time
3. The order is important, let's the oldest copy first.