exec: Handle NULLS in TopK sorter #38654

rohany · 2019-07-03T15:51:06Z

This commit fixes NULLs in the TopK sorter by avoiding use
of the vec copy method, which has a bug. Instead, we add
a set method to the vec comparator, and use the templatized
comparator to perform the sets that the TopK sorter needs.

To facilitate this, we add an UnsetNull method to the Nulls
object. However, use of this method results in HasNull()
maybe returning true even if the vector doesn't have nulls.
This behavior already occurs when selection vectors are used.
Based on discussions with @solongordon and @asubiotto, this behavior
is OK, and future PR's will attempt to make this behavior better, and address
the bugs within the Vec Copy method.

cockroach-teamcity · 2019-07-03T15:51:12Z

This change is

solongordon

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @asubiotto, @rohany, and @solongordon)

pkg/sql/exec/vec_comparators.go, line 26 at r1 (raw file):

	// set sets the value of the vector at dstVecIdx at index dstIdx to the value
	// at the vector at srcVecIdx at index srcIdx.
	set(srcVecIdx, dstVecIdx int, srcIdx, dstIdx uint16)

Nit: How about srcValIdx and dstValIdx to match the argument names in compare?

pkg/sql/exec/vec_comparators_tmpl.go, line 82 at r1 (raw file):

func (c *_TYPEVecComparator) set(srcVecIdx, dstVecIdx int, srcIdx, dstIdx uint16) {
	if c.nulls[srcVecIdx].HasNulls() && c.nulls[srcVecIdx].NullAt(srcIdx) {

I think it's possible for nulls[srcVecIdx] or nulls[dstVecIdx] to be nil if they don't have nulls. Could you test this and handle those cases if necessary?

pkg/sql/exec/coldata/nulls.go, line 156 at r1 (raw file):

// UnsetNull64 unsets the ith values of the column.
func (n *Nulls) UnsetNull64(i uint64) {
	n.nulls[i/8] |= ^flippedBitMask[i%8]

^flippedBitMask is just bitMask.

rohany

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @asubiotto and @solongordon)

pkg/sql/exec/vec_comparators_tmpl.go, line 82 at r1 (raw file):

Previously, solongordon (Solon) wrote…

I think it's possible for nulls[srcVecIdx] or nulls[dstVecIdx] to be nil if they don't have nulls. Could you test this and handle those cases if necessary?

Done.

pkg/sql/exec/coldata/nulls.go, line 156 at r1 (raw file):

Previously, solongordon (Solon) wrote…

^flippedBitMask is just bitMask.

Done.

yuzefovich

Nice work!

[nit]: I think a release note is not necessary for this fix (and a missing period if you decide to keep it).

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @asubiotto and @solongordon)

This commit fixes NULLs in the TopK sorter by avoiding use of the vec copy method, which has a bug. Instead, we add a set method to the vec comparator, and use the templatized comparator to perform the sets that the TopK sorter needs. To facilitate this, we add an UnsetNull method to the Nulls object. However, use of this method results in HasNull() maybe returning true even if the vector doesn't have nulls. This behavior already occurs when selection vectors are used. Release note: none

rohany · 2019-07-03T17:16:46Z

alright, I removed the release note. Thanks for the look!

solongordon

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @asubiotto)

rohany · 2019-07-08T14:17:58Z

bors r+

craig · 2019-07-08T15:08:21Z

Build failed (retrying...)

GitHub CI (Cockroach)

craig · 2019-07-08T15:21:50Z

Build failed (retrying...)

GitHub CI (Cockroach)

rohany · 2019-07-08T15:43:23Z

bors r+

craig · 2019-07-08T15:43:24Z

Already running a review

@solongordon

37199: storage: propagate errors from contentionQueue, catch stalls in roachtest r=nvanbenschoten a=nvanbenschoten Informs #36089. The PR is split into a series of commits. The first fixes part of a bug that was causing #36089 to fail (thanks to #36748) and the second improves the test to have a more obvious failure condition for this class of bug in the future. The third, fifth, and sixth clean up code. Finally, the fourth fixes another bug that could cause issues with #36089. Before the first commit, requests could get stuck repeatedly attempting to push a transaction only to repeatedly find that they themselves were already aborted. The error would not propagate up to the transaction coordinator and the request would get stuck. This commit fixes this behavior by correctly propagating errors observed by the `contentionQueue`. The second commit bumps the TxnLivenessThreshold for clusters running `kv/contention/nodes=4` to 10 minutes. This is sufficiently large such that if at any point a transaction is abandoned then all other transactions will begin waiting for it, throughput will drop to 0 for 10 straight minutes, and the test will fail to achieve its minimum QPS requirement. The fourth commit instructs pushers in the `txnwait.Queue` to inform all other pushers that are waiting for the same transaction when it observes an ABORTED transaction. I never saw this cause issues with #36089, but it seems very possible that it could given frequent tscache rotations. 38397: sql: deflake TestLogic//crdb_internal/max_retry_counter r=knz a=knz Fixes #38062. Release note: None 38654: exec: Handle NULLS in TopK sorter r=rohany a=rohany This commit fixes NULLs in the TopK sorter by avoiding use of the vec copy method, which has a bug. Instead, we add a set method to the vec comparator, and use the templatized comparator to perform the sets that the TopK sorter needs. To facilitate this, we add an UnsetNull method to the Nulls object. However, use of this method results in HasNull() maybe returning true even if the vector doesn't have nulls. This behavior already occurs when selection vectors are used. Based on discussions with @solongordon and @asubiotto, this behavior is OK, and future PR's will attempt to make this behavior better, and address the bugs within the Vec Copy method. 38725: cli/dump: more clearly inform the user upon tables with no visible columns r=knz a=knz Informs #37768. Informs #28948. This is coming up quite often on support, lately again on gitter and forum https://forum.cockroachlabs.com/t/error-while-dumping-core-backup/2901/3. This PR aims to lessen the burden on support and propose a clear "next action" for the user. Before: ``` kena@kenax ~/cockroach % ./cockroach dump --insecure defaultdb CREATE TABLE t (, FAMILY "primary" (rowid) ); Error: pq: at or near "from": syntax error Failed running "dump" ``` After: ``` kena@kenax ~/cockroach % ./cockroach dump --insecure defaultdb CREATE TABLE t (, FAMILY "primary" (rowid) ); Error: table "defaultdb.public.t" has no visible columns HINT: To proceed with the dump, either omit this table from the list of tables to dump, drop the table, or add some visible columns. -- See: #37768 Failed running "dump" ``` Release note (cli change): `cockroach dump` will now more clearly refer to issue #37768 when it encounters a table with no visible columns, which (currently) cannot be dumped successfully. Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com> Co-authored-by: Raphael 'kena' Poss <knz@cockroachlabs.com> Co-authored-by: Rohan Yadav <rohany@alumni.cmu.edu>

craig · 2019-07-08T16:25:39Z

Build succeeded

GitHub CI (Cockroach)

38669: exec: Update NULL handling in Vec Copy r=rohany a=rohany The copy method in Vec previously unset ALL NULL values in the destination vector, and then proceeded to set the appropriate NULL values from the source vector, which is incorrect. To address this problem, this commit adds a UnsetNullRange operation to the Nulls object. The operation unsets all null values within an input range, but is unable to smartly/efficiently identify when the vector no longer contains any null values. The best way to address this problem can be left to future PR's and discussion. Depends on #38654. Until that is landed, look at the latest commit to review. Co-authored-by: Rohan Yadav <rohany@alumni.cmu.edu>

rohany requested review from solongordon, asubiotto and a team July 3, 2019 15:51

solongordon suggested changes Jul 3, 2019

View reviewed changes

rohany force-pushed the topknulls branch from 2665a78 to 88801c1 Compare July 3, 2019 16:38

rohany commented Jul 3, 2019

View reviewed changes

yuzefovich reviewed Jul 3, 2019

View reviewed changes

rohany force-pushed the topknulls branch from 88801c1 to 3c0aa0b Compare July 3, 2019 17:16

rohany requested a review from a team July 3, 2019 17:42

rohany mentioned this pull request Jul 3, 2019

exec: Update NULL handling in Vec Copy #38669

Merged

solongordon approved these changes Jul 8, 2019

View reviewed changes

craig bot merged commit 3c0aa0b into cockroachdb:master Jul 8, 2019

rohany mentioned this pull request Jul 8, 2019

exec: handle nulls in top K sorter #38618

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

exec: Handle NULLS in TopK sorter #38654

exec: Handle NULLS in TopK sorter #38654

rohany commented Jul 3, 2019 •

edited

Loading

cockroach-teamcity commented Jul 3, 2019

solongordon left a comment

rohany left a comment

yuzefovich left a comment

rohany commented Jul 3, 2019

solongordon left a comment

rohany commented Jul 8, 2019

craig bot commented Jul 8, 2019

craig bot commented Jul 8, 2019

rohany commented Jul 8, 2019

craig bot commented Jul 8, 2019

craig bot commented Jul 8, 2019

exec: Handle NULLS in TopK sorter #38654

exec: Handle NULLS in TopK sorter #38654

Conversation

rohany commented Jul 3, 2019 • edited Loading

cockroach-teamcity commented Jul 3, 2019

solongordon left a comment

Choose a reason for hiding this comment

rohany left a comment

Choose a reason for hiding this comment

yuzefovich left a comment

Choose a reason for hiding this comment

rohany commented Jul 3, 2019

solongordon left a comment

Choose a reason for hiding this comment

rohany commented Jul 8, 2019

craig bot commented Jul 8, 2019

Build failed (retrying...)

craig bot commented Jul 8, 2019

Build failed (retrying...)

rohany commented Jul 8, 2019

craig bot commented Jul 8, 2019

craig bot commented Jul 8, 2019

Build succeeded

rohany commented Jul 3, 2019 •

edited

Loading