planner: support vector index in planner #56204

winoros · 2024-09-21T16:18:54Z

What problem does this PR solve?

Issue Number: ref #54245, close #56510

Problem Summary:

What changed and how does it work?

Support to choose vector index by CBO and USE INDEX hint.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

codecov · 2024-09-21T16:47:00Z

Codecov Report

Attention: Patch coverage is 77.88462% with 46 lines in your changes missing coverage. Please review.

Project coverage is 76.0249%. Comparing base (5e8bb8e) to head (8b9840f).
Report is 1 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #56204        +/-   ##
================================================
+ Coverage   73.3611%   76.0249%   +2.6637%     
================================================
  Files          1626       1649        +23     
  Lines        449213     457362      +8149     
================================================
+ Hits         329548     347709     +18161     
+ Misses        99482      87974     -11508     
- Partials      20183      21679      +1496

Flag	Coverage Δ
integration	`51.7887% <15.3846%> (?)`
unit	`72.4581% <77.8846%> (-0.0068%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9478% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.1288% <ø> (+17.5649%)`	⬆️

pkg/planner/core/task.go

AilinKid

other parts lgtm

AilinKid · 2024-09-29T08:00:22Z

pkg/planner/core/plan_cost_ver2.go

+	TiFlashMem:      costusage.CostVer2Factor{Name: "tiflash_mem_factor", Value: 0.05},
+	TiDBDisk:        costusage.CostVer2Factor{Name: "tidb_disk_factor", Value: 200.00},
+	TiDBRequest:     costusage.CostVer2Factor{Name: "tidb_request_factor", Value: 6000000.00},
+	ANNIndexStart:   costusage.CostVer2Factor{Name: "ann_index_start_factor", Value: 0.000144},


how does this value be decided？

According to benchmark of comparing:

TiFlash Ann Top K

TiFlash Read K rows

AilinKid · 2024-09-29T08:08:42Z

pkg/planner/core/explain.go

+			}
+		}
+		buffer.WriteString(", limit:")
+		buffer.WriteString(fmt.Sprint(p.AnnIndexExtra.TopK))


strconv.FormatUint()

AilinKid · 2024-09-29T08:24:13Z

pkg/planner/core/casetest/index/index_test.go

+		"Warning 1105 No predicate column has been collected yet for table test.t, so only indexes and the columns composing the indexes will be analyzed",
+		"Warning 1105 The version 2 would collect all statistics not only the selected indexes",
+		"Warning 1105 analyzing vector index is not supported, skip idx",
+		"Warning 1105 analyzing vector index is not supported, skip idx2"))


analyze table t index idx
we specify idx only, while idx2 is checked and throw warnings, it that normal?

AilinKid · 2024-09-29T08:33:27Z

pkg/expression/vs_helper.go

+// ExtractVectorSearch extracts a VectorSearchExpr from an expression.
+// NOTE: not all VectorSearch functions are supported by the index. The caller
+// needs to check the distance function name.
+func ExtractVectorSearch(expr Expression) (*VectorSearchExpr, error) {


why not let VectorSearchExpr be an another expression Expression implementor

then we could do this in getFunction verify args

and the handling in pkg/planner/core/task.go xx31 will be more clean

AilinKid · 2024-09-29T08:38:25Z

pkg/planner/core/task.go

+		return false
+	}
+	ts.AnnIndexExtra.RefVecF32 = vs.Vec.SerializeTo(nil)
+	ts.AnnIndexExtra.TopK = uint32(p.Count)


better add some comments for this function

we could only see: that it checks some elements in topN's by item, then fills some fields in ts. what is it used for? another projection injection logic is not seen here?

pkg/planner/core/find_best_task.go

pkg/planner/core/casetest/vectorsearch/vector_index_test.go

Signed-off-by: Wish <breezewish@outlook.com>

tidbcloud/tidb-cse#940 tidbcloud/tidb-cse#920 Signed-off-by: Wish <breezewish@outlook.com>

…s/tidb into support-planner-cbo * 'support-planner-cbo' of ssh://ssh.github.com:443/winoros/tidb: address comments

Signed-off-by: Wish <breezewish@outlook.com>

winoros · 2024-10-09T15:08:19Z

go.mod

@@ -307,7 +307,7 @@ require (
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 // indirect
 	google.golang.org/protobuf v1.34.2
 	gopkg.in/inf.v0 v0.9.1 // indirect
-	gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect


need to revert it.

tiprow · 2024-10-09T15:58:55Z

@breezewish: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test fast_test_tiprow_ddlargsv1
/test tidb_parser_test

Use /test all to run the following jobs that were automatically triggered:

fast_test_tiprow
tidb_parser_test

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

breezewish · 2024-10-09T16:18:29Z

/test pull-br-integration-test

tiprow · 2024-10-09T16:18:35Z

@breezewish: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test fast_test_tiprow_ddlargsv1
/test tidb_parser_test

Use /test all to run the following jobs that were automatically triggered:

fast_test_tiprow
tidb_parser_test

In response to this:

/test pull-br-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2024-10-10T02:39:37Z

[LGTM Timeline notifier]

Timeline:

2024-10-09 08:26:15.555359972 +0000 UTC m=+1034530.975572984: ☑️ agreed by AilinKid.
2024-10-10 02:39:36.975351743 +0000 UTC m=+1100132.395564769: ☑️ agreed by XuHuaiyu.

tangenta

lgtm for ddl part

ti-chi-bot · 2024-10-10T03:40:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AilinKid, tangenta, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [AilinKid,XuHuaiyu,tangenta]
~~pkg/ddl/OWNERS~~ [tangenta]
~~pkg/expression/OWNERS~~ [XuHuaiyu]
~~pkg/meta/OWNERS~~ [tangenta]
~~pkg/planner/OWNERS~~ [AilinKid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

breezewish · 2024-10-10T05:31:24Z

/test unit-test

tiprow · 2024-10-10T05:31:30Z

@breezewish: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test fast_test_tiprow_ddlargsv1
/test tidb_parser_test

Use /test all to run the following jobs that were automatically triggered:

fast_test_tiprow
tidb_parser_test

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

AilinKid · 2024-10-10T04:02:20Z

pkg/planner/core/exhaust_physical_plans.go

+		if len(lt.ByItems) != 1 {
+			return ret
+		}
+		vs := expression.ExtractVectorHelper(lt.ByItems[0].Expr)


how about multiple order by items.

It is no longer a vector search.

pkg/planner/core/casetest/vectorsearch/vector_index_test.go

…into support-planner-cbo

winoros · 2024-10-10T14:16:22Z

/retest

ti-chi-bot · 2024-10-10T14:31:00Z

In response to a cherrypick label: new pull request created to branch release-8.4: #56546.

ref #54245, close #56510

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 21, 2024

winoros force-pushed the support-planner-cbo branch from a475b88 to 2bdb3b6 Compare September 21, 2024 16:31

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 21, 2024

winoros force-pushed the support-planner-cbo branch from 2bdb3b6 to 35bdfee Compare September 22, 2024 13:33

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 22, 2024

JaySon-Huang reviewed Sep 25, 2024

View reviewed changes

pkg/planner/core/task.go Outdated Show resolved Hide resolved

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 26, 2024

zimulala requested a review from AilinKid September 27, 2024 06:25

AilinKid reviewed Sep 29, 2024

View reviewed changes

This comment was marked as resolved.

Sign in to view

zimulala reviewed Oct 8, 2024

View reviewed changes

pkg/planner/core/find_best_task.go Outdated Show resolved Hide resolved

pkg/planner/core/find_best_task.go Show resolved Hide resolved

pkg/planner/core/casetest/vectorsearch/vector_index_test.go Outdated Show resolved Hide resolved

breezewish force-pushed the support-planner-cbo branch from f5f94db to afa9ff7 Compare October 8, 2024 07:35

breezewish changed the base branch from feature/vector-index to master October 8, 2024 07:35

ti-chi-bot bot added the component/dumpling This is related to Dumpling of TiDB. label Oct 8, 2024

planner: support vector index by CBO

add4469

winoros force-pushed the support-planner-cbo branch from b62ad3d to add4469 Compare October 8, 2024 08:33

winoros removed the component/dumpling This is related to Dumpling of TiDB. label Oct 8, 2024

breezewish and others added 3 commits October 8, 2024 16:46

Pass column_id to TiFlash

e35fd47

Signed-off-by: Wish <breezewish@outlook.com>

Cherry-pick

2d015d8

tidbcloud/tidb-cse#940 tidbcloud/tidb-cse#920 Signed-off-by: Wish <breezewish@outlook.com>

address comments

bf3a8e7

winoros force-pushed the support-planner-cbo branch from 2d015d8 to bf3a8e7 Compare October 8, 2024 09:51

breezewish added 3 commits October 8, 2024 20:37

Merge branch 'support-planner-cbo' of ssh://ssh.github.com:443/winoro…

784fd88

…s/tidb into support-planner-cbo * 'support-planner-cbo' of ssh://ssh.github.com:443/winoros/tidb: address comments

Fix lint

a0b1755

Signed-off-by: Wish <breezewish@outlook.com>

Fix bazel

c638b26

Signed-off-by: Wish <breezewish@outlook.com>

winoros added 2 commits October 9, 2024 22:42

fix tests

bce942f

go mod tidy & fix test

e1f54ae

winoros commented Oct 9, 2024

View reviewed changes

go mod tidy & don't do sync load

0f761e6

XuHuaiyu approved these changes Oct 10, 2024

View reviewed changes

ti-chi-bot bot added the lgtm label Oct 10, 2024

XuHuaiyu approved these changes Oct 10, 2024

View reviewed changes

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Oct 10, 2024

tangenta approved these changes Oct 10, 2024

View reviewed changes

ti-chi-bot bot added the approved label Oct 10, 2024

AilinKid reviewed Oct 10, 2024

View reviewed changes

zimulala added the needs-cherry-pick-release-8.4 Should cherry pick this PR to release-8.4 branch. label Oct 10, 2024

winoros added 2 commits October 10, 2024 15:27

fix wrong test

f60bfc7

Merge branch 'master' into support-planner-cbo

8692688

winoros changed the title ~~planner: support vector index by CBO~~ planner: support vector index in planner Oct 10, 2024

winoros added 2 commits October 10, 2024 21:30

fix static check

95feba8

Merge branch 'support-planner-cbo' of https://github.com/winoros/tidb …

8b9840f

…into support-planner-cbo

ti-chi-bot bot merged commit 7aefb08 into pingcap:master Oct 10, 2024
25 checks passed

ti-chi-bot mentioned this pull request Oct 10, 2024

planner: support vector index in planner (#56204) #56546

Merged

13 tasks

winoros deleted the support-planner-cbo branch October 10, 2024 14:35

ti-chi-bot bot pushed a commit that referenced this pull request Oct 11, 2024

planner: support vector index in planner (#56204) (#56546)

8516d55

ref #54245, close #56510

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: support vector index in planner #56204

planner: support vector index in planner #56204

winoros commented Sep 21, 2024 •

edited by zimulala

Loading

codecov bot commented Sep 21, 2024 •

edited

Loading

AilinKid left a comment

AilinKid Sep 29, 2024

breezewish Oct 7, 2024

AilinKid Sep 29, 2024

AilinKid Sep 29, 2024

AilinKid Sep 29, 2024

AilinKid Sep 29, 2024

AilinKid Sep 29, 2024

This comment was marked as resolved.

winoros Oct 9, 2024

tiprow bot commented Oct 9, 2024

breezewish commented Oct 9, 2024

tiprow bot commented Oct 9, 2024

ti-chi-bot bot commented Oct 10, 2024

tangenta left a comment

ti-chi-bot bot commented Oct 10, 2024

breezewish commented Oct 10, 2024

tiprow bot commented Oct 10, 2024

AilinKid Oct 10, 2024

breezewish Oct 10, 2024

winoros commented Oct 10, 2024

ti-chi-bot commented Oct 10, 2024

planner: support vector index in planner #56204

planner: support vector index in planner #56204

Conversation

winoros commented Sep 21, 2024 • edited by zimulala Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

codecov bot commented Sep 21, 2024 • edited Loading

Codecov Report

AilinKid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

tiprow bot commented Oct 9, 2024

breezewish commented Oct 9, 2024

tiprow bot commented Oct 9, 2024

ti-chi-bot bot commented Oct 10, 2024

[LGTM Timeline notifier]

tangenta left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Oct 10, 2024

breezewish commented Oct 10, 2024

tiprow bot commented Oct 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

winoros commented Oct 10, 2024

ti-chi-bot commented Oct 10, 2024

winoros commented Sep 21, 2024 •

edited by zimulala

Loading

codecov bot commented Sep 21, 2024 •

edited

Loading