-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Filebeat] Add postgresql csv fileset #23334
[Filebeat] Add postgresql csv fileset #23334
Commits on Jan 4, 2021
-
[Filebeat] Add postgresql csv fileset
As a configuration option, create another pipeline inside the module to parse CSV log files. Although it's necessary to configure the database to emit such logs, there is some advantages: configured properly, some events like `statement timeout` and `lock timeout` will display the query in the same event, opposed to multiple lines, one with the error message and other with statement.
Configuration menu - View commit details
-
Copy full SHA for 684f912 - Browse repository at this point
Copy the full SHA 684f912View commit details
Commits on Jan 9, 2021
-
Configuration menu - View commit details
-
Copy full SHA for a6b2ea0 - Browse repository at this point
Copy the full SHA a6b2ea0View commit details -
Update filebeat/module/postgresql/csv/_meta/fields.yml
Do not generate duplicate field groups. Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Configuration menu - View commit details
-
Copy full SHA for cacde06 - Browse repository at this point
Copy the full SHA cacde06View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7ec70d8 - Browse repository at this point
Copy the full SHA 7ec70d8View commit details
Commits on Jan 16, 2021
-
filebeat(module/postgres): Fix CSV ingestion / tests
Commit cacde06 changed field names. I commited in github without pay attention to tests. This commit fixes all tests and the pipeline.
Configuration menu - View commit details
-
Copy full SHA for 31334cc - Browse repository at this point
Copy the full SHA 31334ccView commit details -
Refactor the PostgreSQL CSV Log pipeline
* Adding postgresql.log.query_name in grok The query_step was ignored because it's a subset of command_tag * Separate message and query fields Message is the main message, and query is what message is referencing
Configuration menu - View commit details
-
Copy full SHA for 606173e - Browse repository at this point
Copy the full SHA 606173eView commit details
Commits on Jan 24, 2021
-
Fix(filebeat/postgresql): Change core_id to session_line_number
There is no `core_id` inside PostgreSQL logs. Deprecating field and moving it to `session_line_number`.
Configuration menu - View commit details
-
Copy full SHA for d358b32 - Browse repository at this point
Copy the full SHA d358b32View commit details
Commits on Feb 13, 2021
-
[Filebeat] Refactor PostgreSQL log pipeline
This commit will open the path to add 2 pipelines: - The default one, using log files - The CSV pipeline Since both files represent the same information, makes sense to have only one fileset.
Configuration menu - View commit details
-
Copy full SHA for 94de6c1 - Browse repository at this point
Copy the full SHA 94de6c1View commit details -
[Filebeat] PostgreSQL split grok logic
Creates a new grok pattern to match only the timestamp and the next char, the separator. Based on separator, choose the right pipeline.
Configuration menu - View commit details
-
Copy full SHA for 56f2d54 - Browse repository at this point
Copy the full SHA 56f2d54View commit details
Commits on Feb 14, 2021
-
[Filebeat] Merge PostgreSQL's log and csv filesets
Both of then represents the same information, so use the same pipeline to do the job.
Configuration menu - View commit details
-
Copy full SHA for fa308d6 - Browse repository at this point
Copy the full SHA fa308d6View commit details -
[Filebeat] PostgreSQL module adjustments
- remove unused `thread_id` field - issue a `make update`
Configuration menu - View commit details
-
Copy full SHA for 7c90863 - Browse repository at this point
Copy the full SHA 7c90863View commit details -
[Filebeat] PostgreSQL module: drop error_severity
Drop `postgresql.log.error_severity` in favor of `log.level`
Configuration menu - View commit details
-
Copy full SHA for eee16be - Browse repository at this point
Copy the full SHA eee16beView commit details -
[Filebeat] PostgreSQL: append "error" in relevant events
Based on PostgreSQL Error Codes, everything that not starts with `00`, `01` or `02` is an error.
Configuration menu - View commit details
-
Copy full SHA for 17724a0 - Browse repository at this point
Copy the full SHA 17724a0View commit details -
[Filebeat] re-generate test results
Regenerate files to have a consistent field order.
Configuration menu - View commit details
-
Copy full SHA for c95556f - Browse repository at this point
Copy the full SHA c95556fView commit details -
[Filebeat] deprecate postgresql.log.error.code
This field is a long, but postgresql errors can have letters. For now just the deprecation, a naive change in grok will break the parser.
Configuration menu - View commit details
-
Copy full SHA for 6558076 - Browse repository at this point
Copy the full SHA 6558076View commit details
Commits on Feb 15, 2021
-
Update filebeat/module/postgresql/log/_meta/fields.yml
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Configuration menu - View commit details
-
Copy full SHA for 9eed55c - Browse repository at this point
Copy the full SHA 9eed55cView commit details -
Update filebeat/module/postgresql/log/manifest.yml
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Configuration menu - View commit details
-
Copy full SHA for f570e58 - Browse repository at this point
Copy the full SHA f570e58View commit details -
[Filebeat] change field name in postgresql module
Use `raw_message` instead of `message2`.
Configuration menu - View commit details
-
Copy full SHA for 85ab1e3 - Browse repository at this point
Copy the full SHA 85ab1e3View commit details -
[Filebeat] deprecate postgresql.log.error.code
According to PostgreSQL docs, this code is not a number: https://www.postgresql.org/docs/current/errcodes-appendix.html Stop collecting it and move it `sql_state_code`
Configuration menu - View commit details
-
Copy full SHA for b11a3d4 - Browse repository at this point
Copy the full SHA b11a3d4View commit details -
[Filebeat] move postgresql.log.error.code to postgresql.log.sql_state…
…_code After deprecate the field, stop to populate it.
Configuration menu - View commit details
-
Copy full SHA for 7f353b6 - Browse repository at this point
Copy the full SHA 7f353b6View commit details -
[Filebeat] update postresql log fields
Give more details to `details` and `hints` and change `error.code` to an alias.
Configuration menu - View commit details
-
Copy full SHA for ad53939 - Browse repository at this point
Copy the full SHA ad53939View commit details -
[Filebeat] postgresql: do not overlap event.type
The previous logic tag all messages as "info" and then append "error" if it's the case. Put a conditional to not have "event.type": ["info", "error"].
Configuration menu - View commit details
-
Copy full SHA for 978708d - Browse repository at this point
Copy the full SHA 978708dView commit details
Commits on Feb 16, 2021
-
Configuration menu - View commit details
-
Copy full SHA for f131b46 - Browse repository at this point
Copy the full SHA f131b46View commit details -
Update filebeat/module/postgresql/log/_meta/fields.yml
Change part of the description to a comment Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Configuration menu - View commit details
-
Copy full SHA for 3671f00 - Browse repository at this point
Copy the full SHA 3671f00View commit details -
[Filebeat] PostgreSQL module adjustments
- Run update since last `fields.yml` was done only in github - Use `set` instead of `append` in pipeline to not create an array - Capture the `postgresql.log.query_step` in csv pipeline
Configuration menu - View commit details
-
Copy full SHA for 60f064f - Browse repository at this point
Copy the full SHA 60f064fView commit details -
Now PostgreSQL has 3 files in the ingest directory: - pipeline.yml - pipeline-log.yml - pipeline-csv.yml Change the count to 3 to reflect that.
Configuration menu - View commit details
-
Copy full SHA for 1d23ce9 - Browse repository at this point
Copy the full SHA 1d23ce9View commit details
Commits on Feb 17, 2021
-
[Filebeat] fix a typo in PostgreSQL module
postresql.log.sql_state_code -> `postgresql.log.sql_state_code`.
Configuration menu - View commit details
-
Copy full SHA for 5fa37c8 - Browse repository at this point
Copy the full SHA 5fa37c8View commit details -
Update filebeat/module/postgresql/log/ingest/pipeline-csv.yml
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Configuration menu - View commit details
-
Copy full SHA for 991a227 - Browse repository at this point
Copy the full SHA 991a227View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0bfbde6 - Browse repository at this point
Copy the full SHA 0bfbde6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 633a8cd - Browse repository at this point
Copy the full SHA 633a8cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for a518be6 - Browse repository at this point
Copy the full SHA a518be6View commit details