runaway: Mitigation tidb_runaway_queries flooding #55664

HuSharp · 2024-08-26T09:28:53Z

What problem does this PR solve?

Issue Number: ref #54434

Problem Summary: There could be a large number of runaway queries that are identified in a short period.
Currently, the TiDB logs each occurrence which flood the log table. Since it's likely the most queries are identical, we shall introduce a way to compress the logs.

What changed and how does it work?

changed

Include sql_digest of this query in this table. If users add manual rules with sql_digest, they can easily identify the hits by filtering sql_digest.
There's a potential risk that tabletidb_runaway_queries could be flooded if massive queries are identified as runaway queries in a short period. For example, SQL injections. Since this table is informational, we can afford to lose details.
1. modify flushes the records to KV at an interval of 10 seconds
2. Provide a record map with hash (resouce_name, sql_digest, plan_digest, match_type). This map will be flushed to tidb_runaway_queries table every 10s. (This is because flush inserts all the records, so the map is added to minimize the number of inserts)
3. Add an UNIQUE KEY runaway_task(resource_group_name, sql_digest, plan_digest), time represents the timestamp of the first occurrence. repeat represents the times that the message repeats since the last flush.

result

mysql> ALTER RESOURCE GROUP default QUERY_LIMIT=(EXEC_ELAPSED='1s', ACTION=KILL,  WATCH=EXACT DURATION='10m');
Query OK, 0 rows affected (0.22 sec)

// now is empty
mysql> select * from mysql.tidb_runaway_queries limit 3 \G;
Empty set (0.00 sec)

mysql> select sleep(2) from t;
ERROR 8253 (HY000): Query execution was interrupted, identified as runaway query

mysql> select * from mysql.tidb_runaway_queries limit 3 \G;
*************************** 1. row ***************************
resource_group_name: default
         start_time: 2024-09-05 11:59:57
            repeats: 1
         match_type: identify
             action: kill
         sample_sql: select sleep(2) from t
         sql_digest: 4adbc838b86c573265d4b39a3979d0a362b5f0336c91c26930c83ab187701a55
        plan_digest: 5d094f78efbce44b2923733b74e1d09233cb446318293492901c5e5d92e27dbc
        tidb_server: 127.0.0.1:4000
1 row in set (0.00 sec)

And then execute select sleep(3) from t; which has same UNIQUE KEY runaway_task(resource_group_name, sql_digest, plan_digest) as select sleep(3) from t;

mysql> select sleep(3) from t;
ERROR 8253 (HY000): Query execution was interrupted, identified as runaway query

// We can find time and repeats have been updated
mysql> select * from mysql.tidb_runaway_queries limit 3 \G;
*************************** 1. row ***************************
resource_group_name: default
         start_time: 2024-09-05 11:59:57
            repeats: 2
         match_type: identify
             action: kill
         sample_sql: select sleep(2) from t
         sql_digest: 4adbc838b86c573265d4b39a3979d0a362b5f0336c91c26930c83ab187701a55
        plan_digest: 5d094f78efbce44b2923733b74e1d09233cb446318293492901c5e5d92e27dbc
        tidb_server: 127.0.0.1:4000

And then execute select sleep(2) from t; which will be regarded as watch type rather than kill.

mysql> select sleep(2) from t;
ERROR 8253 (HY000): Query execution was interrupted, identified as runaway query

// We can find time and repeats have been updated
mysql> select * from mysql.tidb_runaway_queries limit 3 \G;
*************************** 1. row ***************************
resource_group_name: default
         start_time: 2024-09-05 11:59:57
            repeats: 2
         match_type: identify
             action: kill
         sample_sql: select sleep(2) from t
         sql_digest: 4adbc838b86c573265d4b39a3979d0a362b5f0336c91c26930c83ab187701a55
        plan_digest: 5d094f78efbce44b2923733b74e1d09233cb446318293492901c5e5d92e27dbc
        tidb_server: 127.0.0.1:4000
*************************** 2. row ***************************
resource_group_name: default
         start_time: 2024-09-05 12:00:13
            repeats: 1
         match_type: watch
             action: kill
         sample_sql: select sleep(2) from t
         sql_digest: 4adbc838b86c573265d4b39a3979d0a362b5f0336c91c26930c83ab187701a55
        plan_digest: 5d094f78efbce44b2923733b74e1d09233cb446318293492901c5e5d92e27dbc
        tidb_server: 127.0.0.1:4000
2 rows in set (0.00 sec)

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

runaway: Mitigation  tidb_runaway_queries flooding

tiprow · 2024-08-26T09:29:14Z

Hi @HuSharp. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-08-26T09:45:49Z

Codecov Report

Attention: Patch coverage is 94.33962% with 3 lines in your changes missing coverage. Please review.

Project coverage is 56.8043%. Comparing base (fef43c5) to head (35d31aa).
Report is 58 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #55664         +/-   ##
=================================================
- Coverage   72.8403%   56.8043%   -16.0360%     
=================================================
  Files          1599       1731        +132     
  Lines        444670     635469     +190799     
=================================================
+ Hits         323899     360974      +37075     
- Misses       100835     250512     +149677     
- Partials      19936      23983       +4047

Flag	Coverage Δ
integration	`37.3851% <5.6603%> (?)`
unit	`72.0964% <94.3396%> (+0.1343%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9567% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`52.7356% <ø> (+6.9623%)`	⬆️

nolouch

can u add some tests?

HuSharp · 2024-09-04T05:27:16Z

/retest-required

tiprow · 2024-09-04T05:27:42Z

@HuSharp: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pkg/resourcegroup/runaway/checker.go

HuSharp · 2024-09-04T06:40:04Z

can u add some tests?

added map-related and flood-related tests :)

HuSharp · 2024-09-04T06:57:30Z

/retest-required

tiprow · 2024-09-04T06:57:50Z

@HuSharp: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

HuSharp · 2024-09-04T06:59:50Z

/retest-required

tiprow · 2024-09-04T07:00:46Z

@HuSharp: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

HuSharp · 2024-09-04T07:05:18Z

/retest-required

tiprow · 2024-09-04T07:05:39Z

@HuSharp: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nolouch

rest lgtm

nolouch · 2024-09-04T08:36:45Z

pkg/session/bootstrap_test.go

@@ -270,6 +270,13 @@ func revertVersionAndVariables(t *testing.T, se sessiontypes.Session, ver int) {
 		// for version <= version195, tidb_enable_dist_task should be disabled before upgrade
 		MustExec(t, se, "update mysql.global_variables set variable_value='off' where variable_name='tidb_enable_dist_task'")
 	}
+	if ver < version212 {


only for tests? if in real environment, do we have a way or need a way to revert?

Yes, for testing purposes only. We will not encounter calling unsetStoreBootstrapped and supporting fallback/revert in the real world. PTAL, thx~ @tangenta

HuSharp · 2024-09-04T08:54:23Z

/retest-required

tiprow · 2024-09-04T08:54:44Z

@HuSharp: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

HuSharp · 2024-09-04T13:33:04Z

/retest-required

tiprow · 2024-09-04T13:33:29Z

@HuSharp: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: husharp <ihusharp@gmail.com>

pkg/session/bootstrap.go

Signed-off-by: husharp <ihusharp@gmail.com>

HuSharp · 2024-09-12T02:01:31Z

/test unit-test

tiprow · 2024-09-12T02:01:54Z

@HuSharp: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

glorv · 2024-09-12T02:05:27Z

/ok-to-test
/retest

HuSharp · 2024-09-12T02:22:31Z

/retest-required

Signed-off-by: husharp <ihusharp@gmail.com>

HuSharp · 2024-09-12T04:59:25Z

/retest-required

lcwangchao

LGTM for session part

yudongusa

LGTM

ti-chi-bot · 2024-09-13T02:14:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: glorv, JmPotato, lcwangchao, nolouch, tangenta, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [glorv,lcwangchao,tangenta]
~~pkg/session/OWNERS~~ [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

HuSharp · 2024-09-13T02:35:51Z

/retest-required

ti-chi-bot · 2024-09-13T02:48:30Z

@HuSharp: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-lightning-integration-test	`f7ca2d1`	link	true	`/test pull-lightning-integration-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

HuSharp · 2024-09-13T02:49:26Z

/retest-required

tiprow · 2024-09-13T03:01:50Z

@HuSharp: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`35d31aa`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot bot added do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 26, 2024

nolouch reviewed Sep 3, 2024

View reviewed changes

HuSharp force-pushed the add_union_key_runaway branch from f7ca2d1 to acc9e35 Compare September 4, 2024 01:10

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 4, 2024

HuSharp force-pushed the add_union_key_runaway branch 2 times, most recently from 2523bdb to d8fbe0d Compare September 4, 2024 03:53

JmPotato reviewed Sep 4, 2024

View reviewed changes

pkg/resourcegroup/runaway/checker.go Outdated Show resolved Hide resolved

HuSharp requested review from JmPotato and nolouch September 4, 2024 06:40

ti-chi-bot bot removed the do-not-merge/needs-tests-checked label Sep 4, 2024

nolouch reviewed Sep 4, 2024

View reviewed changes

HuSharp force-pushed the add_union_key_runaway branch from 7c6a2ed to 6b76e24 Compare September 5, 2024 03:17

HuSharp requested a review from glorv September 11, 2024 05:44

HuSharp mentioned this pull request Sep 11, 2024

resource_control: add repeats to tidb_runaway_queries pingcap/docs-cn#18611

Merged

7 tasks

remove unnecessary flush

0457a34

Signed-off-by: husharp <ihusharp@gmail.com>

glorv approved these changes Sep 11, 2024

View reviewed changes

lcwangchao reviewed Sep 11, 2024

View reviewed changes

pkg/session/bootstrap.go Outdated Show resolved Hide resolved

HuSharp force-pushed the add_union_key_runaway branch from f2a1f00 to 91e3cbb Compare September 11, 2024 09:19

change rebuild index operation to rename column

b36e3e2

Signed-off-by: husharp <ihusharp@gmail.com>

HuSharp force-pushed the add_union_key_runaway branch from 91e3cbb to b36e3e2 Compare September 11, 2024 09:27

add upgrade version test

1607836

Signed-off-by: husharp <ihusharp@gmail.com>

HuSharp force-pushed the add_union_key_runaway branch from 61ced36 to 1607836 Compare September 11, 2024 10:06

remove domain close in test

94ebfaf

Signed-off-by: husharp <ihusharp@gmail.com>

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Sep 12, 2024

HuSharp added 2 commits September 12, 2024 11:31

ensure version upragde

f89d02f

Signed-off-by: husharp <ihusharp@gmail.com>

accelerate flush in test

35d31aa

Signed-off-by: husharp <ihusharp@gmail.com>

lcwangchao approved these changes Sep 12, 2024

View reviewed changes

yudongusa approved these changes Sep 13, 2024

View reviewed changes

ti-chi-bot bot added the approved label Sep 13, 2024

ti-chi-bot bot merged commit bb3456b into pingcap:master Sep 13, 2024
22 of 23 checks passed

HuSharp deleted the add_union_key_runaway branch September 13, 2024 04:12

lilin90 mentioned this pull request Sep 14, 2024

resource_control: add repeats to tidb_runaway_queries pingcap/docs#18909

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runaway: Mitigation tidb_runaway_queries flooding #55664

runaway: Mitigation tidb_runaway_queries flooding #55664

HuSharp commented Aug 26, 2024 •

edited

Loading

tiprow bot commented Aug 26, 2024

codecov bot commented Aug 26, 2024 •

edited

Loading

nolouch left a comment •

edited

Loading

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 4, 2024

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

nolouch left a comment

nolouch Sep 4, 2024

HuSharp Sep 4, 2024 •

edited

Loading

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 12, 2024

tiprow bot commented Sep 12, 2024

glorv commented Sep 12, 2024

HuSharp commented Sep 12, 2024

HuSharp commented Sep 12, 2024

lcwangchao left a comment

yudongusa left a comment

ti-chi-bot bot commented Sep 13, 2024

HuSharp commented Sep 13, 2024

ti-chi-bot bot commented Sep 13, 2024 •

edited

Loading

HuSharp commented Sep 13, 2024

tiprow bot commented Sep 13, 2024

runaway: Mitigation tidb_runaway_queries flooding #55664

runaway: Mitigation tidb_runaway_queries flooding #55664

Conversation

HuSharp commented Aug 26, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

changed

result

Check List

Release note

tiprow bot commented Aug 26, 2024

codecov bot commented Aug 26, 2024 • edited Loading

Codecov Report

nolouch left a comment • edited Loading

Choose a reason for hiding this comment

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 4, 2024

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

nolouch left a comment

Choose a reason for hiding this comment

nolouch Sep 4, 2024

Choose a reason for hiding this comment

HuSharp Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

HuSharp commented Sep 12, 2024

tiprow bot commented Sep 12, 2024

glorv commented Sep 12, 2024

HuSharp commented Sep 12, 2024

HuSharp commented Sep 12, 2024

lcwangchao left a comment

Choose a reason for hiding this comment

yudongusa left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Sep 13, 2024

HuSharp commented Sep 13, 2024

ti-chi-bot bot commented Sep 13, 2024 • edited Loading

HuSharp commented Sep 13, 2024

tiprow bot commented Sep 13, 2024

HuSharp commented Aug 26, 2024 •

edited

Loading

codecov bot commented Aug 26, 2024 •

edited

Loading

nolouch left a comment •

edited

Loading

HuSharp Sep 4, 2024 •

edited

Loading

ti-chi-bot bot commented Sep 13, 2024 •

edited

Loading