Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

infoschema: adapt TableByName for query of memory table #54918

Merged
merged 15 commits into from
Jul 31, 2024

Conversation

tangenta
Copy link
Contributor

@tangenta tangenta commented Jul 25, 2024

What problem does this PR solve?

Issue Number: ref #50305

Problem Summary:

Read memory tables from information_schema is slow when tidb_schema_cache_size is non-zero.

What changed and how does it work?

  • Add TableSchemaSelector to help return the table/schema names specified in predicate.
  • Adapt TableByName for memory table information_schema.tables.
  • Adapt FindTableByPartitionID for information_schema.partitions.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    Before this PR:
    -- first load
    mysql> select * from  information_schema.tables where table_schema = 'test' and table_name = 't6613';
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | ENGINE | VERSION | ROW_FORMAT | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH | MAX_DATA_LENGTH | INDEX_LENGTH | DATA_FREE | AUTO_INCREMENT | CREATE_TIME         | UPDATE_TIME | CHECK_TIME | TABLE_COLLATION | CHECKSUM | CREATE_OPTIONS | TABLE_COMMENT | TIDB_TABLE_ID | TIDB_ROW_ID_SHARDING_INFO | TIDB_PK_TYPE | TIDB_PLACEMENT_POLICY_NAME |
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    | def           | test         | t6613      | BASE TABLE | InnoDB |      10 | Compact    |          2 |              8 |          16 |               0 |           16 |         0 |              5 | 2024-07-25 21:10:37 | NULL        | NULL       | utf8mb4_bin     |     NULL |                |               |          8063 | NOT_SHARDED               | NONCLUSTERED | NULL                       |
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    1 row in set (22.68 sec)
    
    
    -- subsequent load
    mysql> select * from  information_schema.tables where table_schema = 'test' and table_name = 't6613';
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | ENGINE | VERSION | ROW_FORMAT | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH | MAX_DATA_LENGTH | INDEX_LENGTH | DATA_FREE | AUTO_INCREMENT | CREATE_TIME         | UPDATE_TIME | CHECK_TIME | TABLE_COLLATION | CHECKSUM | CREATE_OPTIONS | TABLE_COMMENT | TIDB_TABLE_ID | TIDB_ROW_ID_SHARDING_INFO | TIDB_PK_TYPE | TIDB_PLACEMENT_POLICY_NAME |
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    | def           | test         | t6613      | BASE TABLE | InnoDB |      10 | Compact    |          2 |              8 |          16 |               0 |           16 |         0 |              5 | 2024-07-25 21:10:37 | NULL        | NULL       | utf8mb4_bin     |     NULL |                |               |          8063 | NOT_SHARDED               | NONCLUSTERED | NULL                       |
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    1 row in set (2.51 sec)
    
    After this PR:
    mysql> select * from  information_schema.tables where table_schema = 'test' and table_name = 't6613';
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | ENGINE | VERSION | ROW_FORMAT | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH | MAX_DATA_LENGTH | INDEX_LENGTH | DATA_FREE | AUTO_INCREMENT | CREATE_TIME         | UPDATE_TIME | CHECK_TIME | TABLE_COLLATION | CHECKSUM | CREATE_OPTIONS | TABLE_COMMENT | TIDB_TABLE_ID | TIDB_ROW_ID_SHARDING_INFO | TIDB_PK_TYPE | TIDB_PLACEMENT_POLICY_NAME |
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    | def           | test         | t6613      | BASE TABLE | InnoDB |      10 | Compact    |          2 |              8 |          16 |               0 |           16 |         0 |              1 | 2024-07-25 21:10:37 | NULL        | NULL       | utf8mb4_bin     |     NULL |                |               |          8063 | NOT_SHARDED               | NONCLUSTERED | NULL                       |
    +---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
    1 row in set (0.00 sec)
    
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 25, 2024
Copy link

tiprow bot commented Jul 25, 2024

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copy link

codecov bot commented Jul 25, 2024

Codecov Report

Attention: Patch coverage is 89.70588% with 28 lines in your changes missing coverage. Please review.

Project coverage is 74.6847%. Comparing base (199bf90) to head (f292a3c).
Report is 9 commits behind head on master.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #54918        +/-   ##
================================================
+ Coverage   72.6158%   74.6847%   +2.0688%     
================================================
  Files          1565       1565                
  Lines        440229     442705      +2476     
================================================
+ Hits         319676     330633     +10957     
+ Misses       100711      91766      -8945     
- Partials      19842      20306       +464     
Flag Coverage Δ
integration 49.4718% <85.6617%> (?)
unit 71.7194% <87.1323%> (+0.0557%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.9567% <ø> (ø)
parser ∅ <ø> (∅)
br 52.5020% <ø> (+6.6531%) ⬆️

@tiancaiamao
Copy link
Contributor

@tangenta

[2024/07/25 14:54:35.357 +00:00] [INFO] [schema_version.go:381] ["wait latest schema version changed(get the metadata lock if tidb_enable_metadata_lock is true)"] [category=ddl] [ver=52] ["take time"=2.815636ms] [job="ID:108, Type:create table, State:done, SchemaState:public, SchemaID:2, TableID:104, RowCount:0, ArgLen:2, start time: 2024-07-25 14:54:35.351 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/07/25 14:54:35.359 +00:00] [INFO] [ddl_worker.go:940] ["finish DDL job"] [worker="worker 22, tp general"] [category=ddl] [jobID=108] [conn=3] [job="ID:108, Type:create table, State:synced, SchemaState:public, SchemaID:2, TableID:104, RowCount:0, ArgLen:0, start time: 2024-07-25 14:54:35.351 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/07/25 14:54:35.361 +00:00] [INFO] [ddl.go:1271] ["DDL job is finished"] [category=ddl] [jobID=108]
    testkit.go:274: 
        	Error Trace:	pkg/testkit/testkit.go:274
        	            				pkg/testkit/testkit.go:199
        	            				pkg/testkit/testkit.go:174
        	            				pkg/planner/core/memtable_predicate_extractor_test.go:1679
        	Error:      	Received unexpected error:
        	            	[schema:1146]Table 'INFORMATION_SCHEMA.p' doesn't exist
        	            	github.com/pingcap/errors.AddStack
        	            		external/com_github_pingcap_errors/errors.go:178
        	            	github.com/pingcap/errors.Trace
        	            		external/com_github_pingcap_errors/juju_adaptor.go:15
        	            	github.com/pingcap/tidb/pkg/executor.getAllTableInfos
        	            		pkg/executor/infoschema_reader.go:588
        	            	github.com/pingcap/tidb/pkg/executor.(*memtableRetriever).setDataFromTables
        	            		pkg/executor/infoschema_reader.go:631
        	            	github.com/pingcap/tidb/pkg/executor.(*memtableRetriever).retrieve
        	            		pkg/executor/infoschema_reader.go:133
        	            	github.com/pingcap/tidb/pkg/executor.(*MemTableReaderExec).Next
        	            		pkg/executor/memtable_reader.go:120
        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next
        	            		pkg/executor/internal/exec/executor.go:451
        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next
        	            		pkg/executor/adapter.go:1249
        	            	github.com/pingcap/tidb/pkg/executor.(*recordSet).Next
        	            		pkg/executor/adapter.go:169
        	            	github.com/pingcap/tidb/pkg/session.GetRows4Test
        	            		pkg/session/tidb.go:351
        	            	github.com/pingcap/tidb/pkg/session.ResultSetToStringSlice
        	            		pkg/session/tidb.go:369
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ResultSetToResultWithCtx
        	            		pkg/testkit/testkit.go:273
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQueryWithContext
        	            		pkg/testkit/testkit.go:199
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQuery
        	            		pkg/testkit/testkit.go:174
        	            	github.com/pingcap/tidb/pkg/planner/core_test.TestTikvRegionStatusExtractor
        	            		pkg/planner/core/memtable_predicate_extractor_test.go:1679
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1595
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1650
        	Test:       	TestTikvRegionStatusExtractor
        	Messages:   	sql:select * from information_schema.tables where table_name = 'p', args:[]
[2024/07/25 14:54:35.362 +00:00] [INFO] [mock.go:101] ["owner manager retire owner"] [ownerKey=/tidb/ddl/fg/owner] [ID=536f52c9-7647-4fea-bf30-15b953339b6d]
[2024/07/25 14:54:35.362 +00:00] [INFO] [job_table.go:268] ["scheduleLoop quit due to context canceled"] [category=ddl]

@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 29, 2024
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 30, 2024
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 30, 2024
@@ -115,6 +115,7 @@ Projection_4 8000.00 root Column#5, Column#10
└─MemTableScan_6 10000.00 root table:TABLES table_schema:["infoschema__infoschema"]
select engine, DATA_LENGTH from information_schema.tables where lower(table_name) = 't5' and upper(table_schema) = 'INFOSCHEMA__INFOSCHEMA';
engine DATA_LENGTH
InnoDB 8
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a bug in previous impl.

# TestPartitionsColumn
create table pt1(a int primary key, b int) partition by hash(a) partitions 4;
create table pt2(a int primary key, b int) partition by hash(a) partitions 4;
-- sorted_result
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with these logic. Can these test make sure the execution does not query full data in TiKV?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. It only make sure the result is correct. Let me add a manual test about information_schema.partitions.

Copy link
Contributor

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

pkg/planner/core/memtable_predicate_extractor.go Outdated Show resolved Hide resolved
pkg/executor/infoschema_reader.go Outdated Show resolved Hide resolved
pkg/executor/infoschema_reader.go Outdated Show resolved Hide resolved
pkg/executor/infoschema_reader.go Outdated Show resolved Hide resolved
pkg/executor/infoschema_reader.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 31, 2024
Copy link

ti-chi-bot bot commented Jul 31, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-07-30 07:01:20.314276687 +0000 UTC m=+252796.594324758: ☑️ agreed by tiancaiamao.
  • 2024-07-31 06:22:12.211505647 +0000 UTC m=+336848.491553700: ☑️ agreed by lance6716.

Copy link

tiprow bot commented Jul 31, 2024

@tangenta: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
fast_test_tiprow f175483 link true /test fast_test_tiprow

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copy link

ti-chi-bot bot commented Jul 31, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AilinKid, lance6716, tiancaiamao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Jul 31, 2024
@tangenta
Copy link
Contributor Author

/retest

Copy link

tiprow bot commented Jul 31, 2024

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot merged commit 05853dd into pingcap:master Jul 31, 2024
21 checks passed
hawkingrei pushed a commit to hawkingrei/tidb that referenced this pull request Aug 1, 2024
joechenrh added a commit to joechenrh/tidb that referenced this pull request Aug 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants