Fix mysql slowlog template to support mysql 5.7.22 #9647
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Time: 2018-08-07T16:27:47.169604+08:00 | ||
| # User@Host: root[root] @ [218.76.8.37] Id: 7234 | ||
| # Query_time: 15.000223 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0 | ||
| SET timestamp=1533630467; | ||
| select sleep(15); | ||
| # Time: 2018-08-07T16:27:47.169604+08:00 | ||
| # User@Host: debian-sys-maint[debian-sys-maint] @ localhost [] | ||
| # Query_time: 0.000153 Lock_time: 0.000061 Rows_sent: 1 Rows_examined: 5 | ||
| SET timestamp=1533630467; | ||
| SELECT count(*) FROM mysql.user WHERE user='root' and password=''; | ||
| # Time: 2018-08-07T16:27:47.169604+08:00 | ||
| # User@Host: apphost[apphost] @ apphost [1.1.1.1] Id: 10997316 | ||
| # Query_time: 4.071491 Lock_time: 0.000212 Rows_sent: 1000 Rows_examined: 1489615 | ||
| SET timestamp=1533630467; | ||
| SELECT mcu.mcu_guid, mcu.cus_guid, mcu.mcu_url, mcu.mcu_crawlelements, mcu.mcu_order, GROUP_CONCAT(mca.mca_guid SEPARATOR ";") as mca_guid | ||
| FROM kat_mailcustomerurl mcu, kat_customer cus, kat_mailcampaign mca | ||
| WHERE cus.cus_guid = mcu.cus_guid | ||
| AND cus.pro_code = 'CYB' | ||
| AND cus.cus_offline = 0 | ||
| AND mca.cus_guid = cus.cus_guid | ||
| AND (mcu.mcu_date IS NULL OR mcu.mcu_date < CURDATE()) | ||
| AND mcu.mcu_crawlelements IS NOT NULL | ||
| GROUP BY mcu.mcu_guid | ||
| ORDER BY mcu.mcu_order ASC | ||
| LIMIT 1000; | ||
| # Time: 2018-08-07T16:27:47.169604+08:00 | ||
| # User@Host: apphost[apphost] @ apphost [1.1.1.1] Id: 10999834 | ||
| # Query_time: 10.346539 Lock_time: 0.000036 Rows_sent: 0 Rows_examined: 4751313 | ||
| SET timestamp=1533630467; | ||
| call load_stats(1, '2017-04-28 00:00:00'); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| [ | ||
| { | ||
| "@timestamp": "2018-08-07T08:27:47.000Z", | ||
|
There was a problem hiding this comment. I worried at first that taking the unix timestamp instead of the title one we would loose the timezone diff. But seems like the timestamp is in UTC which is good. There was a problem hiding this comment. @ycombinator I remember you saw something different for ES for example? There was a problem hiding this comment. @ruflin I used to start test environment under beats/testing/environments using There was a problem hiding this comment.
The ES (and LS) logs contain ISO8601 timestamps without the timezone. In MySQL the value of the There was a problem hiding this comment. Thanks for the details @ycombinator |
||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "slowlog", | ||
| "event.module": "mysql", | ||
| "input.type": "log", | ||
| "log.flags": [ | ||
| "multiline" | ||
| ], | ||
| "log.offset": 41, | ||
| "mysql.slowlog.id": "7234", | ||
| "mysql.slowlog.ip": "218.76.8.37", | ||
| "mysql.slowlog.lock_time.sec": "0.000000", | ||
| "mysql.slowlog.query": "select sleep(15);", | ||
| "mysql.slowlog.query_time.sec": "15.000223", | ||
| "mysql.slowlog.rows_examined": "0", | ||
| "mysql.slowlog.rows_sent": "1", | ||
| "mysql.slowlog.timestamp": "1533630467", | ||
| "mysql.slowlog.user": "root" | ||
| }, | ||
| { | ||
| "@timestamp": "2018-08-07T08:27:47.000Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "slowlog", | ||
| "event.module": "mysql", | ||
| "input.type": "log", | ||
| "log.flags": [ | ||
| "multiline" | ||
| ], | ||
| "log.offset": 254, | ||
| "mysql.slowlog.host": "localhost", | ||
| "mysql.slowlog.lock_time.sec": "0.000061", | ||
| "mysql.slowlog.query": "SELECT count(*) FROM mysql.user WHERE user='root' and password='';", | ||
| "mysql.slowlog.query_time.sec": "0.000153", | ||
| "mysql.slowlog.rows_examined": "5", | ||
| "mysql.slowlog.rows_sent": "1", | ||
| "mysql.slowlog.timestamp": "1533630467", | ||
| "mysql.slowlog.user": "debian-sys-maint" | ||
| }, | ||
| { | ||
| "@timestamp": "2018-08-07T08:27:47.000Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "slowlog", | ||
| "event.module": "mysql", | ||
| "input.type": "log", | ||
| "log.flags": [ | ||
| "multiline" | ||
| ], | ||
| "log.offset": 526, | ||
| "mysql.slowlog.host": "apphost", | ||
| "mysql.slowlog.id": "10997316", | ||
| "mysql.slowlog.ip": "1.1.1.1", | ||
| "mysql.slowlog.lock_time.sec": "0.000212", | ||
| "mysql.slowlog.query": "SELECT mcu.mcu_guid, mcu.cus_guid, mcu.mcu_url, mcu.mcu_crawlelements, mcu.mcu_order, GROUP_CONCAT(mca.mca_guid SEPARATOR \";\") as mca_guid\n FROM kat_mailcustomerurl mcu, kat_customer cus, kat_mailcampaign mca\n WHERE cus.cus_guid = mcu.cus_guid\n AND cus.pro_code = 'CYB'\n AND cus.cus_offline = 0\n AND mca.cus_guid = cus.cus_guid\n AND (mcu.mcu_date IS NULL OR mcu.mcu_date < CURDATE())\n AND mcu.mcu_crawlelements IS NOT NULL\n GROUP BY mcu.mcu_guid\n ORDER BY mcu.mcu_order ASC\n LIMIT 1000;", | ||
| "mysql.slowlog.query_time.sec": "4.071491", | ||
| "mysql.slowlog.rows_examined": "1489615", | ||
| "mysql.slowlog.rows_sent": "1000", | ||
| "mysql.slowlog.timestamp": "1533630467", | ||
| "mysql.slowlog.user": "apphost" | ||
| }, | ||
| { | ||
| "@timestamp": "2018-08-07T08:27:47.000Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "slowlog", | ||
| "event.module": "mysql", | ||
| "input.type": "log", | ||
| "log.flags": [ | ||
| "multiline" | ||
| ], | ||
| "log.offset": 1438, | ||
| "mysql.slowlog.host": "apphost", | ||
| "mysql.slowlog.id": "10999834", | ||
| "mysql.slowlog.ip": "1.1.1.1", | ||
| "mysql.slowlog.lock_time.sec": "0.000036", | ||
| "mysql.slowlog.query": "call load_stats(1, '2017-04-28 00:00:00');", | ||
| "mysql.slowlog.query_time.sec": "10.346539", | ||
| "mysql.slowlog.rows_examined": "4751313", | ||
| "mysql.slowlog.rows_sent": "0", | ||
| "mysql.slowlog.timestamp": "1533630467", | ||
| "mysql.slowlog.user": "apphost" | ||
| } | ||
| ] | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe I'm misunderstanding this, but should we be using this new
Time:header for@timestampinstead of ignoring it?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think there are some magic here that I don't understand. I'm imagining this exclude_lines is only ignoring it when first time parsing
# User@Host. After that somehow thisTimestill get parsed into@timestampsince it is captured inmysql/slowlog/test/mysql-5.7.22.log-expected.json.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did not investigate but I'm also a bit confused on why this keeps working. Perhaps the exlcude does not actually work here? Have a look at the pipeline.
Also it would be good to have in the test file below a few more lines to make sure it also works with multiple entries.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andrewvc @ruflin I looked into this more and found out
@timestampfield is generated frommysql.slowlog.timestampwith formatUNIX. That's why adding# Timeinto theexclude_linesand that didn't change the output.