Skip to content

Commit

Permalink
opt: prefer sorting fewer columns
Browse files Browse the repository at this point in the history
Currently, if we have to sort results and project a new column, there
is no cost difference between the two orders and we happen to prefer
the sort on top. It is preferable to sort before adding new columns to
avoid storing the extra value in memory or on disk.

This change improves the sort costing by adding a cost proportional to
the total number of values.

Fixes cockroachdb#32952.
  • Loading branch information
RaduBerinde committed Feb 11, 2021
1 parent 673a257 commit 3e7e0a5
Show file tree
Hide file tree
Showing 30 changed files with 769 additions and 782 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -171,11 +171,11 @@ vectorized: true
• filter
│ filter: (b @> '[[1, 2]]') OR (b @> '[[3, 4]]')
└── • sort
order: +a
└── • index join
table: json_tab@primary
└── • index join
table: json_tab@primary
└── • sort
order: +a
└── • inverted filter
│ inverted column: b_inverted_key
Expand All @@ -186,7 +186,7 @@ vectorized: true
table: json_tab@json_inv
spans: 4 spans
·
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJyUk9Fv2j4Qx99_f8XpXkp_85Q4oarkp7RrqlExYAnSNkE0GXLrMlE7s03FhPjfpyS0W7oSwUMi350_d1_f6bZof65QYPx5MrwajKB3M0in6cfhOaTxMH43BQm3yfgD_LBafXVyAZ_ex0kMvQVE87XvhwRnsxlnEGTZ2TmMkxeRkEF_H7mJE7j-AhIZKp3TSD6QRTFDjhnD0uglWatN5drWFwb5BoXPsFDl2lXujOFSG0KxRVe4FaHAqVysKCGZk_F8ZJiTk8WqTvskOKoPhXpEhmkplRXgcd-7Mqb--NuW5U0MfSs2scpBqhwuQbvvZCwyHK-dgIizKMBsx1Cv3R9Z1sl7QsF37HjpA_VIxlF-W6wcGTIeb-t_iseb0oBWEHEBttIP1knjxBwv53Pf96tf-MqJ_1_9g8ZGIJWfgvA5Qt2B8J8OHHx-cMrz73Sh9oMLDgyuNMWDNL-O6314SvFUG0fGC9uFI_7mYPr-KemfR9pvF2j8AnpR8Mr2CCHu0vHoutmi9pX9Gj1fOWIaF6coTsiWWllq6T2U2d9lDCm_p2ZXrV6bJU2MXtZlGnNcc7UjJ-uaKG-MgWpClcC_Yd4JB91w0AmH3XDYCfe74X4nfPECznb__Q4AAP__3TS2RA==
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJyUk9Fv2j4Qx99_f8XpXkp_85Q4oarkp7RrqlExYAnSNkE0GXLrMlE7s03FhPjfpyS0W7oSwUMi350_5-_5fFu0P1coMP48GV4NRtC7GaTT9OPwHNJ4GL-bgoTbZPwBflitvjq5gE_v4ySG3gKi-dr3Q4Kz2YwzCLLs7BzGyYtIyKC_j9zECVx_AYkMlc5pJB_Iopghx4xhafSSrNWmcm3rDYN8g8JnWKhy7Sp3xnCpDaHYoivcilDgVC5WlJDMyXg-MszJyWJVp30SHNWLQj0iw7SUygrwuO9dGVN__G3L8iaGvhWbWOUgVQ6XoN13MhYZjtdOQMRZFGC2Y6jX7o8s6-Q9oeA7drz0gXok4yi_LVaODBmPt_U_xeNNaUAriLgAW-kH66RxYo6X87nv-9UvfGXF_6_-QWMjkMpPQfgcob6B8J8bOFh-cEr5qTaOjBe0i474m4Ppw1PS3-lC7d9FeOBdlKZ4kObXca3tn3L4c0v77aMbv4BeFLwyPUKIu3Q8um6mqL1lP0bPW47oxsUpihOypVaWWnoPZfZ3GUPK76mZVavXZkkTo5f1MY05rrnakZN1TZQ3xkA1oUrg3zDvhINuOOiEw2447IT73XC_E754AWe7_34HAAD__4w7tkQ=

# Combine predicates with OR.
query T
Expand Down Expand Up @@ -221,11 +221,11 @@ vectorized: true
• filter
│ filter: (b @> '[3]') OR (b @> '[[1, 2]]')
└── • sort
order: +a
└── • index join
table: json_tab@primary
└── • index join
table: json_tab@primary
└── • sort
order: +a
└── • inverted filter
│ inverted column: b_inverted_key
Expand All @@ -236,7 +236,7 @@ vectorized: true
table: json_tab@json_inv
spans: 3 spans
·
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJyUU02P0zAQvfMrRnPZD4wSJ12QfMoumxVdlbYklQC1EXKbYQnq2sF2V0VV_ztKQgsBJdCLNV9v3huPZof22xoFxh-mo-vhGM5vh-ksfTe6gDQexa9ncAl3yeQtfLVafXJyCe_fxEkM50uIFhvfDwnO5mF2dgGTpB2ccwZB1mRu4wRuPoJEhkrnNJaPZFHMkWPGsDR6RdZqU4V2dcEw36LwGRaq3LgqnDFcaUModugKtyYUOJPLNSUkczKejwxzcrJY120PWqPaKNQTMkxLqawAj_vetTFe-OJoeVNDn4ttrHKQKocr0O4LGYsMJxsnIOIsCjDbM9Qb90uOdfKBUPA9-3_JQ_VExlF-V6wdGTIeb-s-5ONtaUAriLgAW-kG66RxYoGvFgvf96snPFr8snpfNj4CqfzfhXyBUE8b_DVt56jBKaPe60L9XE7QsZzSFI_SfG_9M4vCTv7wFP5UG0fGC9vcEX_e2X5wSvvjBgdtgiYu4DwK2vchhLhPJ-Ob5k5a2cOhHEs6JV6dIjEhW2plqSWwq7O_zxhS_kDNDVq9MSuaGr2qaRp3UuPqQE7WNVneOEPVpCqBv4N5LzjoBwe94LAfHPaCB_3gQS_46g9wtn_2IwAA___7DKoE
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJyUk99v0zAQx9_5K073sh8YJU46kPyUjWWiU2lLUglQGyG3OUZQZwfbnYqq_u8oCS0ElEBfLN-Pz933dLod2m9rFBh_mI6uh2M4vx2ms_Td6ALSeBS_nsEl3CWTt_DVavXJySW8fxMnMZwvIVpsfD8kOJuH2dkFTJK2c84ZBFkTuY0TuPkIEhkqndNYPpJFMUeOGcPS6BVZq03l2tUJw3yLwmdYqHLjKnfGcKUNodihK9yaUOBMLteUkMzJeD4yzMnJYl2XPWiN6k-hnpBhWkplBXjc966N8cIXx583NfS52MYqB6lyuALtvpCxyHCycQIizqIAsz1DvXG_5FgnHwgF37P_lzxUT2Qc5XfF2pEh4_G27kM83pYGtIKIC7CVbrBOGicW-Gqx8H2_esLjj19W78vGRiCV_zuRLxDqaYO_pu0cNThl1FQbR8YL2gNG_Hln-fCU8ve6UD93H3bsvjTFozTfW2tkUdjZf3BK_-MGB-3ujV_AeRS070MIcZ9OxjfNnbSih0M5pnRKvDpFYkK21MpSS2BXZX-fMaT8gZobtHpjVjQ1elW3acxJzdWOnKxrorwxhqoJVQJ_h3kvHPTDQS8c9sNhLzzohwe98NUfcLZ_9iMAAP__ygWqBA==

# More complex combination.
query T
Expand Down
13 changes: 6 additions & 7 deletions pkg/sql/opt/exec/execbuilder/testdata/distsql_agg
Original file line number Diff line number Diff line change
Expand Up @@ -508,14 +508,13 @@ EXPLAIN (DISTSQL) SELECT c, d, sum(a+c::INT) + avg(b+d) FROM data GROUP BY c, d
distribution: full
vectorized: true
·
• render
• sort
│ order: +c,+d
└── • group
│ group by: c, d
│ ordered: +c,+d
└── • render
└── • sort
order: +c,+d
└── • group
group by: c, d
└── • render
Expand All @@ -524,7 +523,7 @@ vectorized: true
table: data@primary
spans: FULL SCAN
·
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJzUl9tr60YQxt_7VyzzZKM18l6Ui56UnpMWg4-c2g70UExQrMUJJJa7UkpDyP9edEliyc2OzEZgv0mbfPpGM_P7kF8g_fsBfLj882p8MQpJ7_toNp_9Me6T2eX48tucLCmJKUmfHnsRccjS90fh_KxPHBL9s-rdEofEffLbdPKDxFEWkd-nk-sr8uvPQkYm0--X07c7oLBOYhVGjyoF_y9gQIEDBQEUJFDwYEFho5OlStNE5__yUghG8b_gDyncrzdPWX68oLBMtAL_BbL77EGBD_Po9kFNVRQr7Q6BQqyy6P6hsMnLCjb6_jHSz0BhtonWqU8GLiPROiaMJNmd0kBhqtax0j4JmBOI8i0pCbgTSEoCQUkgYfFKIXnKPspIs2ilwGevtH2ps0RnSrtevcpAODSQzqcWfB-Li9VKq1WUJdplw6YPDfJmT3SstIp98nZwEf68CSfzm_B6PO4For9zJPOj2fWPXsDer3h-9W1yHc6L63rxH_XcPpO7KL1rVMJowGHx-vGO4tN3_HhUUtbdfJRDA-6UDzM1o9n0ooTtZlQH9Tdnu83g7y0Q71dvDboZ5e3w-rWdylfpbY_cwPt0znKfOYfJINm4vDHiXVPHsLteJ02vKqv3-7MSTmolsPakszaku2zg8q5YR4qtWD-xYR2x2F5vdqyssw5Ybzb90FlH5lwRxb6Odd4eNN4KND5wRVegIcVWoJ3agIZYbO8WP1bQeAegNZt-6KAhc65A418HmmgPmmgFmhi4sivQkGIr0M5sQEMstndLHCtoogPQmk0_dNCQOVegia8DTbYHTbYCTQ5cryvQkGIr0M5tQEMstndLHitosgPQmk0_dNCQOVegyW5-Jv6P4VSlm2Sdqla__ob5NFS8UuUE0-RJL9WVTpaFTXk7KXTFQazSrPwrK29G6_JPeYHbYmYUc7OYN8VsWyxqYraf-NxGzKSV2sqbI97C2HBpbrg0ij2zs2ce9YnZ-sSoPjWLT43iM7P4zGbLzGJk0mYxtmWI2sob27JzcyYMkVAwRwqyZ8ycKQwJFbbDV10uEPkOYPusC6JGZoaosYXB5Hbu2Mowc7gwD-m7OV6wnTHHC0PyhZkDhiEJw6wiBlFjU7MLGUxu547ujDlnOJIz3CpnOPLtgn28mHOGIznDrXIGUWNfIHY5g8nt3LGd4eac4UjO8P1yZvH6y38BAAD__9PhRac=
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJzUl9FvqkgUxt_3r5icJw1jcGbAWp6me293Y-LFrtpkbzamoTKxTVpwB9xs0_R_vwFqq1TnYMBE3wD78Z055_y-lFdI_n0CD67_vhleDXzS-j6YTCd_Ddtkcj28_jYlc0pCSpLVcysgFpl73sCf9tvEIsF_i9Y9sUjYJn-MRz9IGKQB-XM8ur0hv__MZWQ0_n49Xt8BhSgOlR88qwS8f4ABBQ4UBFBwgIILMwpLHc9VksQ6-5PXXDAI_wevS-ExWq7S7PGMwjzWCrxXSB_TJwUeTIP7JzVWQai03QUKoUqDx6fcJitLLvXjc6BfgMJkGUSJRzo2I0EUEkbi9EFpoDBWUai0RySzpChOSYnklnQokYIS6cDsjUK8Sj_LSNJgocBjb7R6qVeLhVaLII217W5XKgWVWSOu_J93_mh6598Ohy0p2l8eOdmjye2PlmQfVzy7-ja69af59Xapn-73L-QhSB5KxoxKDrO3zxPxvSf6fNUqinWotAq3Xpa_xXBm1t3lXT4h-3po_nFU8XG1bsTdIDu2294aYza99ehs6e6dnjhken7ciZc2K03uw1QU65J77zV0DjGcxDpV2ubltnGLSmEBhdEq9bKTZtvD9nq6FQa6a5wgmUUlt2DnXItucLe0AbtL6G2VwKoDzaoAbbOOzY-FNFLsxnr3zgRp1iTS7LSRRqb3jnSvOaQRwzXS5bbVQppX54lX4ol3bHEsnpBiN3br4kx44k3yxE-bJ2R67zxdNMcTYrjmqdy2WjyJ6jyJSjyJju0ciyek2I3d6p8JT6JJnsRp84RM752nfnM8IYZrnsptq8WTU50npxJPTsd2j8UTUuzGbl2eCU9Okzw5p80TMr13ni6b4wkxXPNUbltjn3A7PMcqWcZRoip9mXWzgatwoYrtSOKVnqsbHc9zm-J2lOvyB6FK0uJXVtwMouKnrMBNMSuL2aaYb4nZYeJ-HTETtdS1vDnizY0NF-aGC6PYMYsdo9g1l-0axbxntu4Z1Rdm8UWdLTOLkUmbxdiWIepa3tiW9Y0NvzQ3_NKcCV0kFMyRguwZ-0LXtjlHzL_gdVAkmdVYLpjVaCgh8nru2MIwc7QwJFuYOVyYi8jN8YLtjDleGJIvrFbAIGpsavUiBpPXc0d3xpwyDIkZZs4ZjuQMr5Uz3JwzHMkZXitnEDUyNUSN7Qwmr-eO_vtjzhmO5Aw35wxHcoYfljOzt99-BQAA__8WaCXH

# There should be no "by hash" routers if there is a single stream.
query T
Expand Down
16 changes: 8 additions & 8 deletions pkg/sql/opt/exec/execbuilder/testdata/distsql_numtables
Original file line number Diff line number Diff line change
Expand Up @@ -51,17 +51,17 @@ EXPLAIN (DISTSQL) SELECT 5, 2+y, * FROM NumToStr WHERE y <= 10 ORDER BY str
distribution: full
vectorized: true
·
• sort
│ order: +str
• render
└── • render
└── • sort
│ order: +str
└── • scan
missing stats
table: numtostr@primary
spans: [ - /10]
·
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJyUkFFL60AQhd_vrxjmvtxrR5KNFmRBiNqIgdrWJKCieYjJUAJpNu5uwFLy36UJYitU9PGcyfnysRs0rxVKDB4W04twBv8mYZzEd9P_EAfT4CqBMYEHI1gTHMF1NL-Ful1ZZayG-5sgCmANz63rnuTnIFyYR5MggstHMFYjYa0KnmUrNiifUGBK2GiVszFKb6tN_0FYvKF0Ccu6ae22TglzpRnlBm1pK0aJSfZSccRZwdpxkbBgm5VVj_3Q8RtdrjK9RsK4yWoj4dgRrvMXCSOuC9YSxlLKcJacEfhi5O0EAt_DtCNUrf2UMDZbMkrR0c9FY6Uta0fsO_qno4N47zf4iE2jasN7-ENkt0sJuVjy8NZGtTrnhVZ5_5shzvtdXxRs7HAVQwjr4bQV3B2Lb8fel3Ha_XkPAAD___kwwOA=
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJyUkFFL-0AQxN__n2LZ_4valeQiBTkQojZioLY1CahoHmKylECai3cXsJR8d2mC2AoVfZzZm5kft0HzVqHE4HExvQxncDQJ4yS-nx5DHEyD6wTGBB6MYE1wAjfR_A7qdmWVsRoeboMogDW8tK57ll-AcGEeTYIIrp7AWI2EtSp4lq3YoHxGgSlho1XOxii9tTb9g7B4R-kSlnXT2q2dEuZKM8oN2tJWjBKT7LXiiLOCteMiYcE2K6u-9hPHb3S5yvQaCeMmq42EU0e4zn9MO0LV2q9yY7MloxQd_R4gVtqydsT-tu-NkDDiumAtYSylDGfJOYEvRt6OIPC9gxjeXzAiNo2qDe9hHGp2u5SQiyUPf21Uq3NeaJX3M4Oc97neKNjY4SoGEdbDaQu4GxY_hr1v4bT79xEAAP__JNjA3g==

# Query which requires a full table scan.
query T
Expand All @@ -70,10 +70,10 @@ EXPLAIN (DISTSQL) SELECT 5, 2 + y, * FROM NumToStr WHERE y % 1000 = 0 ORDER BY s
distribution: full
vectorized: true
·
• sort
│ order: +str
• render
└── • render
└── • sort
│ order: +str
└── • filter
│ filter: (y % 1000) = 0
Expand All @@ -83,7 +83,7 @@ vectorized: true
table: numtostr@primary
spans: FULL SCAN
·
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJzElV-LGjEUxd_7KcKFwm43MpP54-pAwbY7SwWr21FoS_Fh6lwWQSfTJEJF_O5lZtRdRZNUC76ZTM49x_wOZAXy9wwiiL8_9T50--TmoTscDb_2bskw7sWfRiSkxCN3ZEnJO_KYDL6QfDFXXCpBvn2Ok5jcLMlbwlzXvSXviUsGyUOckI8_iFQCKOQ8w346RwnRT2BAwQMKPlAIgEIIYwqF4BOUkovyyKoSdLM_ELkUpnmxUOX2mMKEC4RoBWqqZggRjNJfM0wwzVA4LlDIUKXTWWWzDdgpxHSeiiVQGBZpLiPScDzXdWG8psAX6mW4VOkzQsTW1D7A43SmUKBwwn33ej8iNx22uZgoirr9Uau6n81voJBgnpXnws0WJR12571aUNLxTkb1_iXqkAuFwmEH99QJ7k7O90_OfxnLRYYCs2NDj4To8wYvHLZ_W6fsgz17Zl8FZluFqgkNJ7DvgyHFrg_N6_fBEHXbB3ZuHzx7IJ41kKAC0rQHYkixA3J_fSCGqFsg3rlAfHsgvjWQZgWkZQ_EkGIHpHV9IIaoWyD-uUACeyCBNZASRcOShSHAjkX7-iwMUbcsgv_xeh2Zn6AseC7R6mFyy6cNs2esn0LJF2KCT4JPKpt6Oah01UaGUtVfWb3o5vWnMuBrMdOKPb3Y04r9PTE7FPv62E29daBVh3pxqBUbnJuX_Ol7rbild25pxW29uH1JbGbomKlk-pYxQ83YRT1jhqIFBnN905ihakzftcPs4_WbvwEAAP__8vsheA==
Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJy8lWGL2jAcxt_vU4Q_DO5mpE3aehoYuO16THB6q8I2hi86Gw5Bmy6JMBG_-2ir3ilnkiHdu2t7j8-v_T2QLajfS2AQf38cfhiM0M39YDKdfB3eokk8jD9NUYQRRS20wegdekjGX1C-XmmhtETfPsdJjG426C0ivu_fovfIR-PkPk7Qxx9IaQkYcpHxUbriCthPIICBAoYAMISAIYIZhkKKOVdKyPJftlVgkP0B5mNY5MVal7dnGOZCcmBb0Au95MBgmv5a8oSnGZeeDxgyrtPFsqo5APYLuVilcgMYJkWaK4baHvV9H2Y7DGKtn39c6fSJAyM77A7wsFhqLrn0otP2-j5DN32y_zCMscFo2q2-z_7viwj0XxAmQmouPXL2_n3aAgwJz7OSI9pXYtQnLfriAqM-vcgRXOR4rhcy45JnZ-VhC2a7V2BHoi0Kj5x-rUv14Uk9cZ8CcZ1CtYS2F7rvwUJx3EOnuT1YEA57IE3vgboLoc5CwkpIx12IheIo5K45IRaEgxDatJDAXUjgLKRTCem6C7FQHIV0mxNiQTgICZoWEroLCZ2FlCraji4sAEcXveZcWBAOLsL_eXq9wpFwVYhccaeDyS-PNp498fooVGIt5_xRinlVU1-Oq1x1I-NK109JfTHI60cl4MswMYapOUyN4eAkTM7DgRm7Y64OjenIHI6MYUtz55qXvjOGu-bmrjHcM4d712ATy8ZsIzOvjFhmRq7aGbEMLbSUm5dGLFMj5q2ds892b_4GAAD__4SNIW4=

# Query with a restricted span + filter.
query T
Expand Down
8 changes: 4 additions & 4 deletions pkg/sql/opt/exec/execbuilder/testdata/explain
Original file line number Diff line number Diff line change
Expand Up @@ -1097,7 +1097,7 @@ EXPLAIN (OPT,VERBOSE) SELECT * FROM tc WHERE a = 10 ORDER BY b
sort
├── columns: a:1 b:2
├── stats: [rows=10, distinct(1)=1, null(1)=0]
├── cost: 76.5943856
├── cost: 76.7943856
├── fd: ()-->(1)
├── ordering: +2 opt(1) [actual: +2]
├── prune: (2)
Expand All @@ -1123,7 +1123,7 @@ EXPLAIN (OPT,TYPES) SELECT * FROM tc WHERE a = 10 ORDER BY b
sort
├── columns: a:1(int!null) b:2(int)
├── stats: [rows=10, distinct(1)=1, null(1)=0]
├── cost: 76.5943856
├── cost: 76.7943856
├── fd: ()-->(1)
├── ordering: +2 opt(1) [actual: +2]
├── prune: (2)
Expand Down Expand Up @@ -1228,7 +1228,7 @@ sort
├── columns: a:1 b:2 [hidden: column6:6]
├── immutable
├── stats: [rows=333.333333]
├── cost: 1183.26548
├── cost: 1193.26548
├── fd: (1,2)-->(6)
├── ordering: +6
├── prune: (1,2,6)
Expand Down Expand Up @@ -1265,7 +1265,7 @@ sort
├── columns: a:1(int) b:2(int) [hidden: column6:6(int)]
├── immutable
├── stats: [rows=333.333333]
├── cost: 1183.26548
├── cost: 1193.26548
├── fd: (1,2)-->(6)
├── ordering: +6
├── prune: (1,2,6)
Expand Down
16 changes: 8 additions & 8 deletions pkg/sql/opt/exec/execbuilder/testdata/inverted_filter_json_array
Original file line number Diff line number Diff line change
Expand Up @@ -102,11 +102,11 @@ vectorized: true
• filter
│ filter: (b @> '[[1, 2]]') OR (b @> '[[3, 4]]')
└── • sort
order: +a
└── • index join
table: json_tab@primary
└── • index join
table: json_tab@primary
└── • sort
order: +a
└── • inverted filter
│ inverted column: b_inverted_key
Expand Down Expand Up @@ -146,11 +146,11 @@ vectorized: true
• filter
│ filter: (b @> '[3]') OR (b @> '[[1, 2]]')
└── • sort
order: +a
└── • index join
table: json_tab@primary
└── • index join
table: json_tab@primary
└── • sort
order: +a
└── • inverted filter
│ inverted column: b_inverted_key
Expand Down
16 changes: 8 additions & 8 deletions pkg/sql/opt/exec/execbuilder/testdata/inverted_index
Original file line number Diff line number Diff line change
Expand Up @@ -419,11 +419,11 @@ EXPLAIN SELECT * from d where b @> '{"a": []}' ORDER BY a;
distribution: local
vectorized: true
·
sort
order: +a
index join
table: d@primary
└── • index join
table: d@primary
└── • sort
order: +a
└── • inverted filter
│ inverted column: b_inverted_key
Expand All @@ -440,11 +440,11 @@ EXPLAIN SELECT * from d where b @> '{"a": {}}' ORDER BY a;
distribution: local
vectorized: true
·
sort
order: +a
index join
table: d@primary
└── • index join
table: d@primary
└── • sort
order: +a
└── • inverted filter
│ inverted column: b_inverted_key
Expand Down
Loading

0 comments on commit 3e7e0a5

Please sign in to comment.