ttl: support to split TTL tasks for a table without binary primary key #55663

lcwangchao · 2024-08-26T09:26:31Z

What problem does this PR solve?

Issue Number: close #55660

Problem Summary:

When running a TTL job, the job will be split into several tasks to delete rows of concurrency. However, if a table's primary key is a string type without a binary flag, the task cannot be split. For example:

CREATE TABLE t(id varchar(32) CHARACTER SET UTF8MB4, t timestamp) TTL=`t` + INTERVAL 1 HOUR;

When table t starts a TTL job, it only has one task even if it is a large table with many regions.

The reason is that we split tasks according to the region edges in the current implementation. However, the region is a conception in TiKV which will not ensure it will have a legal encoding with the table record. If use the raw bytes decoded from the region edge as the SQL condition, it will report an error.

For example, for the above table t, the below SQL will report an error:

> select * from t where id < x'ff';
(1105, "Cannot convert string '\\xFF' from binary to utf8mb4")

To avoid this, we did not do split works for this type of table.

But in practice, using a string column as the primary key is very normal, such as to store uuid, phone number, etc... Though we can use binary charset to walk around it to make TTL split tasks, it is some what strange and not easy to use.

What changed and how does it work?

We support to split TTL tasks for tables with string columns without binary flag. In this PR, we only parse the visible ASCII prefix of each region edge bytes. For example, if a table has some regions:

region1: recordPrefix -> recordPrefix + encodedBytes('abc\xffde')
region2: recordPrefix + encodedBytes('abc\xffde') -> recordPrefix + encodedBytes('abd\xffxyz') 
region2: encodedBytes('abc\xde') -> end

3 tasks in TTL job will be created with primary ranges:

id < 'abc'
id >= 'abc' AND id < 'abd'
id > 'abd'

Though it still has some limitations such as it still cannot work well when most keys does not contain ASCII prefix or share the same ASCII prefix, it can fit most scenes.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

tested in a real cluster:

> show create table t1;
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                      |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| t1    | CREATE TABLE `t1` (                                                                                                                                               |
|       |   `id` varchar(64) NOT NULL,                                                                                                                                      |
|       |   `t` timestamp NULL DEFAULT NULL,                                                                                                                                |
|       |   PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */                                                                                                           |
|       | ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin /*T![ttl] TTL=`t` + INTERVAL 0 DAY */ /*T![ttl] TTL_ENABLE='ON' */ /*T![ttl] TTL_JOB_INTERVAL='1m' */ |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set
Time: 0.019s

insert several rows, we can see the TTL tasks are splitter to several tasks:

> select scan_id, state, status from mysql.tidb_ttl_task;
+---------+-------------------------------------------------------------------------------+----------+
| scan_id | state                                                                         | status   |
+---------+-------------------------------------------------------------------------------+----------+
| 0       | {"total_rows":821229,"success_rows":821229,"error_rows":0,"scan_task_err":""} | finished |
| 1       | {"total_rows":524288,"success_rows":524288,"error_rows":0,"scan_task_err":""} | finished |
| 2       | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 3       | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 4       | {"total_rows":454692,"success_rows":454692,"error_rows":0,"scan_task_err":""} | finished |
| 5       | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 6       | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 7       | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 8       | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 9       | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 10      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 11      | {"total_rows":315500,"success_rows":315500,"error_rows":0,"scan_task_err":""} | finished |
| 12      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 13      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 14      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 15      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 16      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 17      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 18      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 19      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 20      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 21      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 22      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 23      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 24      | {"total_rows":296942,"success_rows":296942,"error_rows":0,"scan_task_err":""} | finished |
| 25      | {"total_rows":37117,"success_rows":37117,"error_rows":0,"scan_task_err":""}   | finished |
+---------+-------------------------------------------------------------------------------+----------+

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

ttll: support to split job to several tasks for `utf8` and `utf8mb4` keys.

tiprow · 2024-08-26T09:26:52Z

Hi @lcwangchao. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-08-26T09:40:12Z

Codecov Report

Attention: Patch coverage is 91.89189% with 3 lines in your changes missing coverage. Please review.

Project coverage is 74.7450%. Comparing base (4eeeef8) to head (47444dd).
Report is 3 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #55663        +/-   ##
================================================
+ Coverage   72.9214%   74.7450%   +1.8235%     
================================================
  Files          1581       1584         +3     
  Lines        442336     442645       +309     
================================================
+ Hits         322558     330855      +8297     
+ Misses        99970      91525      -8445     
- Partials      19808      20265       +457

Flag	Coverage Δ
integration	`48.9646% <0.0000%> (?)`
unit	`71.8602% <91.8918%> (-0.1409%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9567% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`51.5820% <ø> (+6.3029%)`	⬆️

YangKeao · 2024-08-26T09:35:51Z

pkg/ttl/cache/table.go

-		if mysql.HasBinaryFlag(ft.GetFlag()) {
-			return t.splitCommonHandleRanges(ctx, tikvStore, splitCnt, false, false)
-		}
+		return t.splitCommonHandleRanges(ctx, tikvStore, splitCnt, false, false, mysql.HasBinaryFlag(ft.GetFlag()))


Should it only accept utf8mb4_bin (or other binary collations)?

For utf8mb4_unicode_ci (or other UCA collations), there is no direct connection between the character and the weight. It's meaningless to get the ASCII prefix.

For example, the ASCII string a has weight 0x0E33.

Done, PTAL again

bb7133

LGTM

bb7133 · 2024-08-26T18:18:19Z

Do we need to update the doc, too?

lcwangchao · 2024-08-27T01:31:39Z

Do we need to update the doc, too?

Yes, I'll update the doc after this PR merged

YangKeao

LGTM

ti-chi-bot · 2024-08-27T03:00:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bb7133, YangKeao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [YangKeao,bb7133]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-08-27T03:00:42Z

[LGTM Timeline notifier]

Timeline:

2024-08-26 18:08:31.674798384 +0000 UTC m=+807306.809248498: ☑️ agreed by bb7133.
2024-08-27 03:00:41.872953214 +0000 UTC m=+839237.007403338: ☑️ agreed by YangKeao.

YangKeao · 2024-08-27T03:20:01Z

/hold

utf8mb4_0900_bin is also acceptable.

pkg/ttl/cache/table.go

YangKeao · 2024-08-27T03:29:48Z

/unhold

lcwangchao · 2024-08-27T04:38:31Z

/retest

tiprow · 2024-08-27T04:38:53Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ttl: support split TTL tasks for non-binary key column

7efcb83

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 26, 2024

lcwangchao requested review from bb7133 and YangKeao August 26, 2024 09:33

YangKeao reviewed Aug 26, 2024

View reviewed changes

lcwangchao force-pushed the ttl_string_split branch from 9617653 to 8037809 Compare August 26, 2024 12:19

bb7133 approved these changes Aug 26, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 26, 2024

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Aug 27, 2024

YangKeao approved these changes Aug 27, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 27, 2024

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 27, 2024

YangKeao reviewed Aug 27, 2024

View reviewed changes

pkg/ttl/cache/table.go Show resolved Hide resolved

lcwangchao force-pushed the ttl_string_split branch from 8037809 to c363ec2 Compare August 27, 2024 03:28

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 27, 2024

address commnet

47444dd

lcwangchao force-pushed the ttl_string_split branch from c363ec2 to 47444dd Compare August 27, 2024 03:33

ti-chi-bot bot merged commit 3d42e34 into pingcap:master Aug 27, 2024
21 checks passed

lcwangchao deleted the ttl_string_split branch August 27, 2024 05:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ttl: support to split TTL tasks for a table without binary primary key #55663

ttl: support to split TTL tasks for a table without binary primary key #55663

lcwangchao commented Aug 26, 2024 •

edited

Loading

tiprow bot commented Aug 26, 2024

codecov bot commented Aug 26, 2024 •

edited

Loading

YangKeao Aug 26, 2024

YangKeao Aug 26, 2024 •

edited

Loading

lcwangchao Aug 26, 2024

bb7133 left a comment

bb7133 commented Aug 26, 2024

lcwangchao commented Aug 27, 2024

YangKeao left a comment

ti-chi-bot bot commented Aug 27, 2024

ti-chi-bot bot commented Aug 27, 2024

YangKeao commented Aug 27, 2024

YangKeao commented Aug 27, 2024

lcwangchao commented Aug 27, 2024

tiprow bot commented Aug 27, 2024

ttl: support to split TTL tasks for a table without binary primary key #55663

ttl: support to split TTL tasks for a table without binary primary key #55663

Conversation

lcwangchao commented Aug 26, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Aug 26, 2024

codecov bot commented Aug 26, 2024 • edited Loading

Codecov Report

YangKeao Aug 26, 2024

Choose a reason for hiding this comment

YangKeao Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

lcwangchao Aug 26, 2024

Choose a reason for hiding this comment

bb7133 left a comment

Choose a reason for hiding this comment

bb7133 commented Aug 26, 2024

lcwangchao commented Aug 27, 2024

YangKeao left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Aug 27, 2024

ti-chi-bot bot commented Aug 27, 2024

[LGTM Timeline notifier]

YangKeao commented Aug 27, 2024

YangKeao commented Aug 27, 2024

lcwangchao commented Aug 27, 2024

tiprow bot commented Aug 27, 2024

lcwangchao commented Aug 26, 2024 •

edited

Loading

codecov bot commented Aug 26, 2024 •

edited

Loading

YangKeao Aug 26, 2024 •

edited

Loading