[Filebeat][New Module] Mysql Enterprise Audit log #22273

P1llus · 2020-10-29T16:50:10Z

What does this PR do?

This creates a module that focuses on logs and files created by MySQL. Currently this only has support for Enterprise Audit logs but can be extended in the future.

Why is it important?

Adding support for more OOTB solutions for filebeat.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Field references: https://dev.mysql.com/doc/refman/8.0/en/audit-log-file-formats.html

elasticmachine · 2020-10-29T16:50:15Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-10-29T16:50:15Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2020-10-29T17:42:33Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #22273 updated
Start Time: 2020-12-08T02:25:59.854+0000
Duration: 48 min 54 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5094
Skipped	566
Total	5660

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	5094
Skipped	566
Total	5660

elasticmachine · 2020-10-29T17:51:44Z

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	4504
Skipped	561
Total	5065

leehinman

looking good.

x-pack/filebeat/module/mysql/audit/_meta/fields.yml

x-pack/filebeat/module/mysql/audit/test/mysql_audit_test.log-expected.json

x-pack/filebeat/module/mysql/audit/ingest/pipeline.yml

filebeat/docs/fields.asciidoc

x-pack/filebeat/modules.d/mysql.yml.disabled

P1llus · 2020-11-10T21:30:18Z

After discussion whether we want to merge this with the current OSS module we decided to rename the module into mysqlenterprise.

Once this is created into a package, both the OSS and the X-Pack module will merge into one.

P1llus · 2020-11-10T21:45:30Z

Seems to be getting there now. @sayden anything you would want to add to this as well?

x-pack/filebeat/module/mysqlenterprise/audit/test/mysql_audit_test.log-expected.json

andrewstucki · 2020-11-11T21:47:25Z

x-pack/filebeat/module/mysqlenterprise/audit/test/mysql_audit_test.log-expected.json

+        "input.type": "log",
+        "log.offset": 0,
+        "mysqlenterprise.audit.class": "audit",
+        "mysqlenterprise.audit.connection_id": 0,


do we know what it means if you have a connection_id field at startup? Just wondering if we should try and figure out an ECS mapping for it or not

I don't think there is any. At first I tried to find a "session ID" field we could use in ECS; but it seems that the session ID is only unique to the current active connections, so it will reset and use the same connection_id for another connection later according to its docs:

https://dev.mysql.com/doc/refman/8.0/en/information-functions.html#function_connection-id

Do you know any other field that might match? I tried to look in client, user and network but could not find anything that matches well

andrewstucki · 2020-11-11T21:48:20Z

x-pack/filebeat/module/mysqlenterprise/audit/test/mysql_audit_test.log-expected.json

+        "mysqlenterprise.audit.class": "connection",
+        "mysqlenterprise.audit.connection_data.connection_attributes._client_name": "libmysql",
+        "mysqlenterprise.audit.connection_data.connection_attributes._client_version": "8.0.22",
+        "mysqlenterprise.audit.connection_data.connection_attributes._os": "Linux",


should these map to host.os fields?

These should be in the client fields, connection_data is for client data and startup_data is for the host. However there was no *.os fields for client in the ECS definition :(

x-pack/filebeat/module/mysqlenterprise/audit/test/mysql_audit_test.log-expected.json

…ilebeat_mysql_audit

P1llus · 2020-11-16T12:33:56Z

@andrewstucki Added a better test log example for processes, added most of the different fields and left comments on the rest.
Also added some extra process fields in case we wanted to use them, please let me know what you think.

adriansr

All in all it's looking good, but I have a few little suggestions here in there.

adriansr · 2020-11-26T11:18:18Z

filebeat/docs/modules/mysqlenterprise.asciidoc

+
+This is a module for different types of MySQL logs. Currently focusing on data from the MySQL Enterprise Audit Plugin in JSON format.
+
+To configure the the Enterprise Audit Plugin to output in JSON format please follow the directions in the https://dev.mysql.com/doc/refman/8.0/en/audit-log-file-formats.html[MySQL Documentation]


Nit: Missing dot at the end.

Suggested change

To configure the the Enterprise Audit Plugin to output in JSON format please follow the directions in the https://dev.mysql.com/doc/refman/8.0/en/audit-log-file-formats.html[MySQL Documentation]

To configure the the Enterprise Audit Plugin to output in JSON format please follow the directions in the https://dev.mysql.com/doc/refman/8.0/en/audit-log-file-formats.html[MySQL Documentation.]

adriansr · 2020-11-26T11:20:12Z

filebeat/docs/modules/mysqlenterprise.asciidoc

+| login.ip                   | client.ip                      |
+| startup_data.os_version    | host.os.full                   |
+| startup_data.args          | process.args                   |
+| onnection_attributes._pid  | process.pid                    |


adriansr · 2020-11-26T11:22:41Z

filebeat/docs/modules/mysqlenterprise.asciidoc

+MySQL Enterprise Audit fields are mapped to ECS in the following way:
+[options="header"]
+|==============================================================
+| Check Point Fields         | ECS Fields                     |


Is "Check Point" a concept in MySQL Audit logs or was this copied over from Check Point docs?

adriansr · 2020-11-26T11:23:41Z

x-pack/filebeat/module/mysqlenterprise/_meta/fields.yml

@@ -0,0 +1,10 @@
+- key: mysqlenterprise
+  title: Mysql Enterprise


Suggested change

title: Mysql Enterprise

title: MySQL Enterprise

adriansr · 2020-11-26T11:23:48Z

x-pack/filebeat/module/mysqlenterprise/_meta/fields.yml

+- key: mysqlenterprise
+  title: Mysql Enterprise
+  description: >
+    Mysql Enterprise Audit module


Same

Suggested change

Mysql Enterprise Audit module

MySQL Enterprise Audit module

adriansr · 2020-11-26T11:32:01Z

x-pack/filebeat/module/mysqlenterprise/audit/ingest/pipeline.yml

+    lang: painless
+    if: ctx?.process?.args != null
+    source: |
+      ctx.process.args_count = ctx.process.args.length - 1;


According to ECS this should be the "length of the process.args array", not the length minus 1.

adriansr · 2020-11-26T11:33:16Z

x-pack/filebeat/module/mysqlenterprise/audit/ingest/pipeline.yml

+    if: ctx?.process?.args != null
+    source: |
+      ctx.process.args_count = ctx.process.args.length - 1;
+      ctx.process.executable = ctx.process.args[0];


Being extra paranoid here, this will cause an exception when process.args is an empty list (non-null, but zero length).

adriansr · 2020-11-26T11:36:21Z

x-pack/filebeat/module/mysqlenterprise/_meta/config.yml

+
+    # Set paths for the log files when file input is used.
+    # Should only be used together with file input
+    # var.paths: /home/user/mysqlauditlogs/audit.*.log


I think this won't work if the user uncomments the line, as vars.paths is expected to be a list not a string.

Suggested change

# var.paths: /home/user/mysqlauditlogs/audit.*.log

# var.paths:

# - /home/user/mysqlauditlogs/audit.*.log

…ilebeat_mysql_audit

changes made

* initial stash of mysql module * initial commit for filebeat mysql module * removing syslog from manifest currently * changing module and field names to mysqlenterprise * fixing some wrong types and adding related fields * adding changelog entry * adding some more process fields and event category/types * fixing merge changelog * adding changes based on PR feedback (cherry picked from commit 03383be)

…dit log (#23021) * [Filebeat][New Module] Mysql Enterprise Audit log (#22273) * initial stash of mysql module * initial commit for filebeat mysql module * removing syslog from manifest currently * changing module and field names to mysqlenterprise * fixing some wrong types and adding related fields * adding changelog entry * adding some more process fields and event category/types * fixing merge changelog * adding changes based on PR feedback (cherry picked from commit 03383be) * regenerate docs Co-authored-by: Marius Iversen <pillus@chasenet.org>

P1llus added 2 commits October 28, 2020 21:04

initial stash of mysql module

358f637

initial commit for filebeat mysql module

15ad791

P1llus added enhancement Filebeat Filebeat Team:Services (Deprecated) Label for the former Integrations-Services team needs_reviewer PR needs to be assigned a reviewer Team:Security-External Integrations labels Oct 29, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 29, 2020

removing syslog from manifest currently

f1b959c

Merge branch 'master' into filebeat_mysql_audit

2fb7efb

leehinman requested changes Oct 30, 2020

View reviewed changes

andrewstucki suggested changes Oct 30, 2020

View reviewed changes

filebeat/docs/fields.asciidoc Show resolved Hide resolved

filebeat/docs/fields.asciidoc Outdated Show resolved Hide resolved

filebeat/docs/fields.asciidoc Outdated Show resolved Hide resolved

filebeat/docs/fields.asciidoc Outdated Show resolved Hide resolved

andrewstucki reviewed Oct 30, 2020

View reviewed changes

x-pack/filebeat/modules.d/mysql.yml.disabled Outdated Show resolved Hide resolved

changing module and field names to mysqlenterprise

31abfbf

fixing some wrong types and adding related fields

aa7b8ff

leehinman approved these changes Nov 10, 2020

View reviewed changes

P1llus added 2 commits November 11, 2020 22:16

adding changelog entry

7836184

Merge branch 'master' into filebeat_mysql_audit

49fa698

andrewstucki reviewed Nov 11, 2020

View reviewed changes

P1llus mentioned this pull request Nov 11, 2020

Migrate MySQL Enterprise Audit log elastic/integrations#397

Closed

11 tasks

P1llus added 3 commits November 16, 2020 13:30

adding some more process fields and event category/types

c8bc685

Merge branch 'filebeat_mysql_audit' of github.com:P1llus/beats into f…

39ab227

…ilebeat_mysql_audit

fixing merge changelog

13d5a54

P1llus added 2 commits November 20, 2020 16:17

Merge branch 'master' into filebeat_mysql_audit

56f38bd

Merge branch 'master' into filebeat_mysql_audit

19806cf

adriansr previously requested changes Nov 26, 2020

View reviewed changes

P1llus added 2 commits December 8, 2020 03:23

adding changes based on PR feedback

300eafb

Merge branch 'filebeat_mysql_audit' of github.com:P1llus/beats into f…

455d535

…ilebeat_mysql_audit

marc-gr approved these changes Dec 9, 2020

View reviewed changes

andrewstucki approved these changes Dec 9, 2020

View reviewed changes

andrewstucki merged commit 03383be into elastic:master Dec 9, 2020

andrewstucki mentioned this pull request Dec 9, 2020

Cherry-pick #22273 to 7.x: [Filebeat][New Module] Mysql Enterprise Audit log #23021

Merged

6 tasks

andrewstucki added the v7.11.0 label Dec 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat][New Module] Mysql Enterprise Audit log #22273

[Filebeat][New Module] Mysql Enterprise Audit log #22273

P1llus commented Oct 29, 2020 •

edited

Loading

elasticmachine commented Oct 29, 2020

elasticmachine commented Oct 29, 2020

elasticmachine commented Oct 29, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

elasticmachine commented Oct 29, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

leehinman left a comment

P1llus commented Nov 10, 2020

P1llus commented Nov 10, 2020

andrewstucki Nov 11, 2020

P1llus Nov 16, 2020

andrewstucki Nov 11, 2020

P1llus Nov 16, 2020

P1llus commented Nov 16, 2020

adriansr left a comment

adriansr Nov 26, 2020

adriansr Nov 26, 2020

adriansr Nov 26, 2020

adriansr Nov 26, 2020

adriansr Nov 26, 2020

adriansr Nov 26, 2020

adriansr Nov 26, 2020 •

edited

Loading

adriansr Nov 26, 2020


		This is a module for different types of MySQL logs. Currently focusing on data from the MySQL Enterprise Audit Plugin in JSON format.

		To configure the the Enterprise Audit Plugin to output in JSON format please follow the directions in the https://dev.mysql.com/doc/refman/8.0/en/audit-log-file-formats.html[MySQL Documentation]

	\| onnection_attributes._pid \| process.pid \|
	\| connection_attributes._pid \| process.pid \|

		@@ -0,0 +1,10 @@
		- key: mysqlenterprise
		title: Mysql Enterprise

	# var.paths: /home/user/mysqlauditlogs/audit.*.log
	# var.paths:
	# - /home/user/mysqlauditlogs/audit.*.log

[Filebeat][New Module] Mysql Enterprise Audit log #22273

[Filebeat][New Module] Mysql Enterprise Audit log #22273

Conversation

P1llus commented Oct 29, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Oct 29, 2020

elasticmachine commented Oct 29, 2020

elasticmachine commented Oct 29, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Oct 29, 2020 • edited by jenkins-beats-ci bot Loading

💚 Flaky test report

Test stats 🧪

leehinman left a comment

Choose a reason for hiding this comment

P1llus commented Nov 10, 2020

P1llus commented Nov 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

P1llus commented Nov 16, 2020

adriansr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr Nov 26, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

P1llus commented Oct 29, 2020 •

edited

Loading

elasticmachine commented Oct 29, 2020 •

edited by jenkins-beats-ci bot

Loading

elasticmachine commented Oct 29, 2020 •

edited by jenkins-beats-ci bot

Loading

adriansr Nov 26, 2020 •

edited

Loading