*: add a PERFORMANCE_SCHEMA.slow_query table #8981

tiancaiamao · 2019-01-08T08:23:40Z

What problem does this PR solve?

Create a virtual table TiNiuB.slow_query

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| INFORMATION_SCHEMA |
| PERFORMANCE_SCHEMA |
| TiNiuB             |
| mysql              |
| test               |
+--------------------+
5 rows in set (0.01 sec)

mysql> use tiniub;
Database changed
mysql> select * from slow_query order by duration limit 2;
+-----------------+----------------------------+----------+---------+------+---------+----------------+----------------+--------+-----------+-----------+----------+
| SQL             | START                      | DURATION | DETAILS | SUCC | CONN_ID | TRANSACTION_TS | USER           | DB     | TABLE_IDS | INDEX_IDS | INTERNAL |
+-----------------+----------------------------+----------+---------+------+---------+----------------+----------------+--------+-----------+-----------+----------+
| select sleep(1) | 2019-01-08 16:14:26.540384 | 00:00:01 |         |    1 |       1 |              0 | root@127.0.0.1 | tiniub |           |           |        0 |
| select sleep(2) | 2019-01-08 16:14:31.577712 | 00:00:02 |         |    1 |       1 |              0 | root@127.0.0.1 | tiniub |           |           |        0 |
+-----------------+----------------------------+----------+---------+------+---------+----------------+----------------+--------+-----------+-----------+----------+
2 rows in set (0.00 sec)

Compare to the admin show slow method, this virtual table if more flexible.

mysql> admin show slow recent 2;
+-----------------+----------------------------+----------+---------+------+---------+----------------+----------------+--------+-----------+-----------+----------+
| SQL             | START                      | DURATION | DETAILS | SUCC | CONN_ID | TRANSACTION_TS | USER           | DB     | TABLE_IDS | INDEX_IDS | INTERNAL |
+-----------------+----------------------------+----------+---------+------+---------+----------------+----------------+--------+-----------+-----------+----------+
| select sleep(3) | 2019-01-08 16:14:35.718031 | 00:00:03 |         |    1 |       1 |              0 | root@127.0.0.1 | tiniub |           |           |        1 |
| select sleep(2) | 2019-01-08 16:14:31.577712 | 00:00:02 |         |    1 |       1 |              0 | root@127.0.0.1 | tiniub |           |           |        1 |
+-----------------+----------------------------+----------+---------+------+---------+----------------+----------------+--------+-----------+-----------+----------+
2 rows in set (0.00 sec)

Borrow ideas from https://github.com/TiNiuB/tiquery
This could serve as a plugin demo later @lysu

What is changed and how it works?

slow_query table is similar to perfschema tables, except that it's not standard, so a plugin maybe more applicable.

Note that tiniub.Init() is not called in this PR, so the TiNiuB database is not created by default.

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

tiancaiamao · 2019-01-08T08:25:56Z

PTAL @lysu @jackysp

codecov-io · 2019-01-08T08:48:00Z

Codecov Report

Merging #8981 into master will decrease coverage by 0.05%.
The diff coverage is 37.5%.

@@            Coverage Diff             @@
##           master    #8981      +/-   ##
==========================================
- Coverage    67.6%   67.54%   -0.06%     
==========================================
  Files         363      365       +2     
  Lines       75309    75426     +117     
==========================================
+ Hits        50909    50944      +35     
- Misses      19911    19986      +75     
- Partials     4489     4496       +7

Impacted Files	Coverage Δ
domain/topn_slow_query.go	`95.69% <0%> (-2.11%)`	⬇️
infoschema/tiniub/tables.go	`24.41% <24.41%> (ø)`
domain/domain.go	`35.7% <60%> (-0.02%)`	⬇️
infoschema/infoschema.go	`78.91% <66.66%> (-0.67%)`	⬇️
infoschema/tiniub/init.go	`80.95% <80.95%> (ø)`
ddl/delete_range.go	`74.28% <0%> (-4.58%)`	⬇️
expression/schema.go	`94.11% <0%> (-0.85%)`	⬇️
executor/join.go	`78.29% <0%> (+0.4%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0bc0f7d...a7ac149. Read the comment docs.

morgo · 2019-01-08T20:09:27Z

May I suggest changing the duration to ms? On some systems 100ms is a slow query.

jackysp · 2019-01-09T01:50:30Z

infoschema/tiniub/init.go

+		}
+		dbInfo := &model.DBInfo{
+			ID:      autoid.GenLocalSchemaID(),
+			Name:    model.NewCIStr("TiNiuB"),


Will the server fail to start if there is already a database which name is also 'TiNiuB'?
Maybe we should give some suggestions to the user when this happened in @lysu 's PR: 1. disable this plugin and start again, 2. rename the existed database.
btw: it is better to use an English name, not a Chinese Pinyin as the database name.

re, it's better give tiniub a English name? or can we direct put new table into PERFORMANCE_SCHEMA or INFOMATION_SCHEMA?

In mysql, we can write a information_schema plugin to add a table to information_schema, https://dev.mysql.com/doc/refman/5.7/en/writing-information-schema-plugins.html

Or should I just put it into INFOMATION_SCHEMA and you turn it into plugin later ? @lysu @jackysp

lysu · 2019-01-10T03:06:56Z

infoschema/infoschema.go

+	if dbName == "information_schema" {
+		return true
+	}
+	for _, driver := range drivers {


when do we call tiniub.Init()? and do we have race-condition to access this drivers? for current plugin is start time is ok...

but if drivers can be dynamic change it's better to take care race.

tiniub.Init is not yet used in this PR.
Your concern is right, if TiNiuB is make into a plugin and load multiple times.

lysu · 2019-01-10T03:08:16Z

infoschema/tiniub/init.go

+		}
+		dbInfo := &model.DBInfo{
+			ID:      autoid.GenLocalSchemaID(),
+			Name:    model.NewCIStr("TiNiuB"),


re, it's better give tiniub a English name? or can we direct put new table into PERFORMANCE_SCHEMA or INFOMATION_SCHEMA?

In mysql, we can write a information_schema plugin to add a table to information_schema, https://dev.mysql.com/doc/refman/5.7/en/writing-information-schema-plugins.html

lysu · 2019-01-10T03:15:12Z

infoschema/tiniub/tables.go

+// IterRecords rewrites the IterRecords method of slowQueryTable.
+func (s slowQueryTable) IterRecords(ctx sessionctx.Context, startKey kv.Key, cols []*table.Column, fn table.RecordIterFunc) error {
+	dom := domain.GetDomain(ctx)
+	result := dom.ShowSlowQuery(&ast.ShowSlow{Tp: 42})


42 is a great number maybe added in ShowSlowType? 😆

A hack here because define a new ShowSlowType need to update parser...
I think I should update parser and avoid hack

lysu · 2019-01-10T03:33:13Z

infoschema/tiniub/tables.go

+		row = append(row, types.NewDatum(item.DB))
+		row = append(row, types.NewDatum(item.TableIDs))
+		row = append(row, types.NewDatum(item.IndexIDs))
+		row = append(row, types.NewDatum(item.Internal))


I found it's maybe useful to combine pingcap/parser#32 sql digest into here, then we can agg a group similar sql's time in furture.

winoros · 2019-01-10T07:44:01Z

I just wonder that whether this pr will be merged into master?
This means that one plugin may need to change TiDB's own code?

lysu · 2019-01-10T14:06:36Z

@tiancaiamao

I had tried to move tables part to https://github.com/lysu/slow_log_table

and integration it in #9006 (https://github.com/lysu/tidb/commit/a8b1f3f0ed51121ed795a4758743b3c7fee75d9b)

It seems work at first glance - - but needs more refining.

also found we can solve "can not work with go module problem" with https://github.com/lysu/slow_log_table/blob/master/go.mod#L8

morgo · 2019-01-10T15:21:26Z

I have some feedback with naming (sorry, not all of it is easily actionable):

The performance_schema name for queries is "statements". Since it is possible for future aggregations of this (by user, by account, by host) I think we should call it something like statements_slow_global.
The column user should be called account to match pfs terminology.
Using MySQL guidelines, this would technically not be a pfs table but a sys table. pfs is the raw structured event data (although it can have an aggregation). If you filter it or order it for a DBA perspective, that is sys. So for example in MySQL 5.7, there is the statement_analysis view:

mysql> select * from sys.statement_analysis LIMIT 1\G
*************************** 1. row ***************************
            query: SELECT `cat` . `name` AS `CATA ... s_database` ( `sch` . `name` )
               db: performance_schema
        full_scan: *
       exec_count: 1
        err_count: 0
       warn_count: 0
    total_latency: 6.53 ms
      max_latency: 6.53 ms
      avg_latency: 6.53 ms
     lock_latency: 6.10 ms
        rows_sent: 5
    rows_sent_avg: 5
    rows_examined: 26
rows_examined_avg: 26
    rows_affected: 0
rows_affected_avg: 0
       tmp_tables: 1
  tmp_disk_tables: 0
      rows_sorted: 5
sort_merge_passes: 0
           digest: d76cf3ff2efdc088db44d6c7e099dd54911416168582241c1aea8a2188e05e66
       first_seen: 2019-01-10 08:12:28.923857
        last_seen: 2019-01-10 08:12:28.923857
1 row in set (0.00 sec)

Note: If you haven't seen sys before, for every view there is an x$ version as well, so I can find the unformatted values in bytes and pico seconds as well:

mysql> select * from x$statement_analysis LIMIT 1\G
*************************** 1. row ***************************
            query: SELECT `sys` . `format_statement` ( `performance_schema` . `events_statements_summary_by_digest` . `DIGEST_TEXT` ) AS `query` , `performance_schema` . `events_statements_summary_by_digest` . `SCHEMA_NAME` AS `db` , IF ( ( ( `performance_schema` . `events_statements_summary_by_digest` . `SUM_NO_GOOD_INDEX_USED` > ? ) OR ( `performance_schema` . `events_statements_summary_by_digest` . `SUM_NO_INDEX_USED` > ? ) ) , ?, ... ) AS `full_scan` , `performance_schema` . `events_statements_summary_by_digest` . `COUNT_STAR` AS `exec_count` , `performance_schema` . `events_statements_summary_by_digest` . `SUM_ERRORS` AS `err_count` , `performance_schema` . `events_statements_summary_by_digest` . `SUM_WARNINGS` AS `warn_count` , `sys` . `format_time` ( `performance_schema` . `events_statements_summary_by_digest` . `SUM_TIMER_WAIT` ) AS `total_latency` , `sys` . `format_time` ( `performance_schema` . `events_statements_summary_by_digest` . `MAX_TIMER_WAIT` ) AS `max_latency` , `sys` . `format_time` (
               db: sys
        full_scan: *
       exec_count: 3
        err_count: 0
       warn_count: 0
    total_latency: 7053198000
      max_latency: 4463873000
      avg_latency: 2351066000
     lock_latency: 1648000000
        rows_sent: 11
    rows_sent_avg: 4
    rows_examined: 41
rows_examined_avg: 14
    rows_affected: 0
rows_affected_avg: 0
       tmp_tables: 0
  tmp_disk_tables: 0
      rows_sorted: 11
sort_merge_passes: 0
           digest: 4762b230873d55f6aff0188d5acbda058b9eb045b5153225eb1dde8b12fb4267
       first_seen: 2019-01-10 08:19:53.128113
        last_seen: 2019-01-10 08:20:03.124817
1 row in set (0.00 sec)

lysu · 2019-01-13T10:39:47Z

LGTM

lysu · 2019-01-13T10:44:50Z

@tiancaiamao maybe need remove tiniub/tables.go and it will be package as plugin

tiancaiamao · 2019-01-14T06:23:46Z

Close it since @lysu done a great work that make it pluggable #9006

tiancaiamao · 2019-01-14T06:37:30Z

Hi @morgo @winoros
We'll make a plugin first, and if the plugin works well, it can serve as canonical plugin example.
Due to the time constraints (plugin feature demo on the devcon), we can fulfill @lysu 's work later https://github.com/lysu/slow_log_table/.

tiancaiamao added 2 commits January 8, 2019 16:04

infoschema,domain: add a TiNiuB.slow_query table as plugin example

e8ccee7

Merge branch 'master' into tiniub

da927c5

tiancaiamao added the component/plugin label Jan 8, 2019

make golint happy

1a67dd8

tiancaiamao mentioned this pull request Jan 8, 2019

infoschema: move perfschema to infoschema directory, tiny refactor #8982

Merged

lysu self-assigned this Jan 8, 2019

jackysp reviewed Jan 9, 2019

View reviewed changes

lysu reviewed Jan 10, 2019

View reviewed changes

tiancaiamao added the status/DNM label Jan 10, 2019

Merge branch 'master' into tiniub

bbb6a22

address comment

345e067

tiancaiamao force-pushed the tiniub branch from a7ac149 to 345e067 Compare January 10, 2019 10:41

tiancaiamao changed the title ~~infoschema,domain: add a TiNiuB.slow_query table as plugin example~~ infoschema,domain: add a PERFORMANCE_SCHEMA.slow_query table Jan 10, 2019

tiancaiamao changed the title ~~infoschema,domain: add a PERFORMANCE_SCHEMA.slow_query table~~ *: add a PERFORMANCE_SCHEMA.slow_query table Jan 10, 2019

Merge branch 'master' into tiniub

c89b817

lysu mentioned this pull request Jan 14, 2019

*: integrate plugin framework with TiDB #9006

Merged

tiancaiamao closed this Jan 14, 2019

tiancaiamao deleted the tiniub branch January 14, 2019 06:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add a PERFORMANCE_SCHEMA.slow_query table #8981

*: add a PERFORMANCE_SCHEMA.slow_query table #8981

tiancaiamao commented Jan 8, 2019

tiancaiamao commented Jan 8, 2019

codecov-io commented Jan 8, 2019 •

edited

Loading

morgo commented Jan 8, 2019

jackysp Jan 9, 2019 •

edited

Loading

lysu Jan 10, 2019

tiancaiamao Jan 10, 2019

lysu Jan 10, 2019

tiancaiamao Jan 10, 2019

lysu Jan 10, 2019

lysu Jan 10, 2019

tiancaiamao Jan 10, 2019

lysu Jan 10, 2019 •

edited

Loading

winoros commented Jan 10, 2019

lysu commented Jan 10, 2019 •

edited

Loading

morgo commented Jan 10, 2019 •

edited

Loading

lysu commented Jan 13, 2019

lysu commented Jan 13, 2019

tiancaiamao commented Jan 14, 2019

tiancaiamao commented Jan 14, 2019

*: add a PERFORMANCE_SCHEMA.slow_query table #8981

*: add a PERFORMANCE_SCHEMA.slow_query table #8981

Conversation

tiancaiamao commented Jan 8, 2019

What problem does this PR solve?

What is changed and how it works?

Check List

tiancaiamao commented Jan 8, 2019

codecov-io commented Jan 8, 2019 • edited Loading

Codecov Report

morgo commented Jan 8, 2019

jackysp Jan 9, 2019 • edited Loading

Choose a reason for hiding this comment

lysu Jan 10, 2019

Choose a reason for hiding this comment

tiancaiamao Jan 10, 2019

Choose a reason for hiding this comment

lysu Jan 10, 2019

Choose a reason for hiding this comment

tiancaiamao Jan 10, 2019

Choose a reason for hiding this comment

lysu Jan 10, 2019

Choose a reason for hiding this comment

lysu Jan 10, 2019

Choose a reason for hiding this comment

tiancaiamao Jan 10, 2019

Choose a reason for hiding this comment

lysu Jan 10, 2019 • edited Loading

Choose a reason for hiding this comment

winoros commented Jan 10, 2019

lysu commented Jan 10, 2019 • edited Loading

morgo commented Jan 10, 2019 • edited Loading

lysu commented Jan 13, 2019

lysu commented Jan 13, 2019

tiancaiamao commented Jan 14, 2019

tiancaiamao commented Jan 14, 2019

codecov-io commented Jan 8, 2019 •

edited

Loading

jackysp Jan 9, 2019 •

edited

Loading

lysu Jan 10, 2019 •

edited

Loading

lysu commented Jan 10, 2019 •

edited

Loading

morgo commented Jan 10, 2019 •

edited

Loading