[NSE-1075] Support dynamic merge file partition

[jackylee-ch]

What changes were proposed in this pull request?

Support merge file partition dynamically.

How was this patch tested?

unit tests.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/native-sql-engine/issues

Then could you also rename commit message and pull request title in the following format?

[NSE-${ISSUES_ID}] ${detailed message}

See also:

• Other pull requests

[github-actions]

#1075

[PHILO-HE]

Thanks for your PR!
A small suggestion, I note two new configs are introduced in this PR. Could you put these two configs into GazellePluginConfig and leave some comments for users? It will make the configs more clear and we can also uniformly set a default value for them at GazellePluginConfig.
When your patch is merged into upstream and gazelle's supported spark version is upgraded, we can make a cleanup then.

[jackylee-ch]

Thanks for your PR! A small suggestion, I note two new configs are introduced in this PR. Could you put these two configs into GazellePluginConfig and leave some comments for users? It will make the configs more clear and we can also uniformly set a default value for them at GazellePluginConfig. When your patch is merged into upstream and gazelle's supported spark version is upgraded, we can make a cleanup then.

Thanks for your advice. The GazellePluginConfig cannot be imported here. I have add the config to ArrowSQLConf.

[PHILO-HE]

OK. Let's keep using ArrowSQLConf in this patch. GazellePluginConfig can be available to be used here by moving it into a separate module where common classes are kept, which can be refactored in other PR.
Will go through internal tests.

[PHILO-HE]

All internal tests passed. @zhouyuan, please have a check further.