is there any way to coalesce shuffle partition except AQE on rapids [QST]

[chenrui17]

What is your question?
I found that value of spark.sql.shuffle.partitions partition seriously affects performance of sql. and in some cases , in order to pass all sql like TPC-DS , I need set a larger shuffle partition , but when I set it larger , the performance is poorest ,so I set it to suitable to most queries , it behaves better , but other query will failes if i set same shuffle.partitions.
now I want to set spark.sql.shuffle.partitions larger and with AQE on at the same time , and expect it coalesce partitions when shuffle read in most cases, and no degradation of performance .
i have 2 questions,

1. Can AQE (coalesce shuffle partition)solve this problem completely？if so , it means that i need achieve dpp on ae together ,Although this feature has not been implemented in the community spark ,like before [FEA] AQE and DPP work together in rapids #863 we discussed thought it (now this work is I am doing now). if not ,what can i do ?
2. is there any way to solve this problem ?

[andygrove]

Unfortunately, I don't think there is a good answer to your question. It wouldn't be efficient to try and use the same initial number of shuffle partitions for all queries and it really does need to be tuned for different circumstances currently (both in community Spark and with the RAPIDS plugin).

However, AQE definitely does provide benefits when the initial number of shuffle partitions is high because it will coalesce them when it sees that there are many small partitions, making the rest of the query run more efficiently.

[sameerz]

Please reopen in case there are follow up questions.