significant degraded performance of select * from large tables in main compared to datafusion v41.0.0

[haohuaijin]

Describe the bug

after try latest main, I find the plan performance of select * from large tables is degraded, like below, you can see that physical_select_all_from_1000 has a 5x performance degradation

group                                         main                                   v41
-----                                         ----                                   ---
logical_aggregate_with_join                   1.53  1271.0±21.98µs        ? ?/sec    1.00   828.2±25.11µs        ? ?/sec
logical_plan_tpcds_all                        1.21    156.4±2.05ms        ? ?/sec    1.00    129.1±7.42ms        ? ?/sec
logical_plan_tpch_all                         1.11     14.2±2.75ms        ? ?/sec    1.00     12.8±0.31ms        ? ?/sec
logical_select_all_from_1000                  1.00      4.0±0.24ms        ? ?/sec    4.40     17.8±0.19ms        ? ?/sec
logical_select_one_from_700                   1.01   620.0±13.54µs        ? ?/sec    1.00   613.3±36.38µs        ? ?/sec
logical_trivial_join_high_numbered_columns    1.18   674.9±14.66µs        ? ?/sec    1.00   573.8±13.26µs        ? ?/sec
logical_trivial_join_low_numbered_columns     1.11   616.7±19.91µs        ? ?/sec    1.00   556.3±19.04µs        ? ?/sec
physical_plan_tpcds_all                       1.26  1112.4±27.08ms        ? ?/sec    1.00   884.2±26.51ms        ? ?/sec
physical_plan_tpch_all                        1.10     62.8±1.28ms        ? ?/sec    1.00     57.3±2.48ms        ? ?/sec
physical_plan_tpch_q1                         1.12      2.3±0.03ms        ? ?/sec    1.00      2.0±0.06ms        ? ?/sec
physical_plan_tpch_q10                        1.12      3.3±0.17ms        ? ?/sec    1.00      2.9±0.35ms        ? ?/sec
physical_plan_tpch_q11                        1.08      2.6±0.10ms        ? ?/sec    1.00      2.4±0.04ms        ? ?/sec
physical_plan_tpch_q12                        1.08      2.0±0.03ms        ? ?/sec    1.00  1860.8±34.67µs        ? ?/sec
physical_plan_tpch_q13                        1.08  1498.7±39.12µs        ? ?/sec    1.00  1389.9±28.17µs        ? ?/sec
physical_plan_tpch_q14                        1.16  1851.9±91.01µs        ? ?/sec    1.00  1594.1±70.39µs        ? ?/sec
physical_plan_tpch_q16                        1.05      2.5±0.06ms        ? ?/sec    1.00      2.4±0.32ms        ? ?/sec
physical_plan_tpch_q17                        1.06      2.2±0.04ms        ? ?/sec    1.00      2.1±0.04ms        ? ?/sec
physical_plan_tpch_q18                        1.00      2.7±0.04ms        ? ?/sec    1.03      2.8±0.47ms        ? ?/sec
physical_plan_tpch_q19                        1.03      3.9±0.05ms        ? ?/sec    1.00      3.8±0.06ms        ? ?/sec
physical_plan_tpch_q2                         1.14      5.9±0.41ms        ? ?/sec    1.00      5.2±0.21ms        ? ?/sec
physical_plan_tpch_q20                        1.06      3.0±0.05ms        ? ?/sec    1.00      2.8±0.05ms        ? ?/sec
physical_plan_tpch_q21                        1.08      4.3±0.07ms        ? ?/sec    1.00      4.0±0.13ms        ? ?/sec
physical_plan_tpch_q22                        1.07      2.2±0.03ms        ? ?/sec    1.00      2.0±0.04ms        ? ?/sec
physical_plan_tpch_q3                         1.10      2.2±0.12ms        ? ?/sec    1.00      2.0±0.15ms        ? ?/sec
physical_plan_tpch_q4                         1.12  1679.2±47.29µs        ? ?/sec    1.00  1501.2±96.77µs        ? ?/sec
physical_plan_tpch_q5                         1.06      3.1±0.05ms        ? ?/sec    1.00      2.9±0.04ms        ? ?/sec
physical_plan_tpch_q6                         1.05  1000.2±22.01µs        ? ?/sec    1.00   956.7±22.22µs        ? ?/sec
physical_plan_tpch_q7                         1.16      4.3±0.05ms        ? ?/sec    1.00      3.7±0.05ms        ? ?/sec
physical_plan_tpch_q8                         1.19      5.4±0.15ms        ? ?/sec    1.00      4.6±0.06ms        ? ?/sec
physical_plan_tpch_q9                         1.26      4.4±0.09ms        ? ?/sec    1.00      3.5±0.05ms        ? ?/sec
physical_select_all_from_1000                 5.21    220.3±5.03ms        ? ?/sec    1.00     42.3±0.38ms        ? ?/sec
physical_select_one_from_700                  1.00      2.3±0.03ms        ? ?/sec    1.00      2.3±0.19ms        ? ?/sec

To Reproduce

v41's commit id is b10b820, and main commit id is 4a227c5

git checkout main 
cargo bench --bench sql_planner -- --save-baseline main
git checkout v41
cargo bench --bench sql_planner -- --save-baseline v41
critcmp main v41

Expected behavior

No response

Additional context

No response