forked from darold/pgbadger
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
1905 lines (1677 loc) · 92.4 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2017-07-27 - v9.2
This release of pgBadger is a maintenance release that adds some new
features.
* Add report of checkpoint distance and estimate.
* Add support of AWS Redshift keywords to SQL code beautifier.
* Add autodetection of log format in remote mode to allow remote
parsing of pgbouncer log file together with PostgreSQL log file.
There's also some bugs fixes and features enhancements.
- Fix reports with histogram that was not showing data upper than
the last range.
- Fix parsing of journalctl without the the log line number pattern
([%l-n]). Thanks to Christian Schmitt for the report.
- Add report of checkpoint distance and estimate. Thanks to jjsantam
for the feature request.
- Append more information on what is done by script to update CSS
and javascript files, tools/updt_embedded_rsc.pl.
- Do not warn when all log files are empty and exit with code 0.
- Fix build_log_line_prefix_regex() that does not include %n as a
lookup in %regex_map. Thanks to ghosthound for the patch.
- Change error level of "FATAL: cannot use CSV" to WARNING. Thanks
to kong1man for the report.
- Fix use of uninitialized value warning. Thanks to Payal for the
report.
- Add permission denied to error normalization
- Update pgbadger to latest commit 5bdc018 of pgFormatter.
- Add support for AWS Redshift keywords. Thanks to cavanaug for the
feature request.
- Fix missing query in temporary file report when the query was
canceled. Thanks to Fabrizio de Royes Mello for the report.
- Normalize query with binded parameters, replaced with a ?.
- Sanity check to avoid end time before start time. Thanks to
Christophe Courtois for the patch.
- Fix a lot of mystyped words and do some grammatical fixes. Use
'pgBadger' where it refers to the program and not the binary file.
Also, use "official" expressions such as PgBouncer, GitHub, and
CSS. POD file was synced with README. Thanks to Euler Taveira for
the patch.
- Menu is broken when --disable-type top_cancelled_info test and
closing list must be inside disable_type test. While in it, ident
disable_lock test. Thanks to Euler Taveira for the patch.
- Fix use of uninitialized value. Thanks to johnkrugger for the
report.
- Remove test to read log file during log format auto-detection when
the file is hosted remotly. Thanks to clomdd for the report.
- Add autodetection of log format in remote mode to allow remote
parsing of pgbouncer log file together with PostgreSQL log file.
- Fix number of sessions wrongly increased after log line validation
Thanks to Achilleas Mantzios for the report.
- Minor reformatting of the pgBadger Description.
- Fix repeated info in documentation. Thanks to cscatolini for the patch.
2017-01-24 - v9.1
This release of pgBadger is a maintenance release that adds some new
features.
* Add report of error class distribution when SQLState is available
in the log_line_prefix (see %e placeholder).
* Update SQL Beautifier to pgFormatter v1.6 code.
* Improve error message normalization.
* Add --normalized-only option to generate a text file containing all
normalized queries found in a log with count.
* Allow %c (session id) to replace %p (pid) as unique session id.
* Add waiting for lock messages to event reports.
* Add --start-monday option to start calendar weeks in Monday
instead of default to Sunday.
There's also some bugs fixes and features enhancements.
- Add report of error class distribution when SQLState is available
in the log line prefix. Thanks to jacks33 for the feature request.
- Fix incremental global index on resize. Thanks to clomdd for the
report.
- Fix command tag log_line_prefix placeholder %i to allow space
character.
- Fix --exclude-line options and removing of obsolete directory
when retention is enabled and --noreport is used.
- Fix typo in "vacuum activity table". Thanks to Nicolas Gollet for
the patch.
- Fix autovacuum report. Thanks to Nicolas Gollet for the patch.
- Fix author of pgbadger's logo - Damien Cazeils and English in
comments. Thanks to Thibaut Madelaine for the patch.
- Add information about pgbouncer log format in the -f option.
Thanks to clomdd for the report.
- Add --normalized-only information in documentation.
- Fix broken report of date-time introduced in previous patch.
- Fix duration/query association when log_duration=on and
log_statement=all. Thanks to Eric Jensen for the report.
- Fix normalization of messages about advisory lock. Thanks to
Thibaut Madelaine for the report.
- Fix report of auto_explain output. Thanks to fch77700 for the
report.
- Fix unwanted log format auto detection with log entry from stdin.
Thanks to Jesus Adolfo Parra for the report.
- Add left open parentheses to the "stop" chars of regex to look
for db client in the prefix to handle the PostgreSQL client
string format that includes source port. Thanks to Jon Nelson
for the patch.
- Fix some spelling errors. Thanks to Jon Nelson for the patch.
- Allow %c (session id) to replace %p (pid) as unique session id.
Thanks to Jerryliuk for the report.
- Allow pgbadger to parse default log_line_prefix that will be
probably used in 10.0: '%m [%p] '
- Fix missing first line with interpreter call.
- Fix missing Avg values in CSV report. Thanks to Yosuke Tomita
for the report.
- Fix error message in autodetect_format() method.
- Add --start-monday option to start calendar weeks in Monday
instead of default to Sunday. Thanks to Joosep Mae for the feature
request.
- Fix --histo-average option. Thanks to Yves Martin for the report.
- Remove plural form of --ssh-option in documentation. Thanks to
mark-a-s for the report.
- Fix --exclude-time filter and rewrite code to skip unwanted line
as well code to update the progress bar. Thanks to Michael
Chesterton for the report.
- Fix support to %r placeholder in prefix instead of %h.
2016-09-02 - v9.0
This major release of pgBadger is a port to bootstrap 3 and a version
upgrade of all resources files (CSS and Javascript). There's also some
bugs fixes and features enhancements.
Backward compatibility with old incremental report might be preserved.
- Sources and licences of resources files are now on a dedicated
subdirectory. A script to update their minified version embedded
in pgbager script has been added. Thanks to Christoph Berg for
the help and feature request.
- Try to detect user/database/host from connection strings if
log_connection is enabled and log_line_prefix doesn't include
them.
Extend the regex to autodetect database name, user name, client
ip address and application name. The regex now are the following:
db => qr/(?:db|database)=([^,]*)/;
user => qr/(?:user|usr)=([^,]*)/;
client => qr/(?:client|remote|ip|host)=([^,]*)/;
appname => qr/(?:app|application)=([^,]*)/;
- Add backward compatibility with older version of pgbadger in
incremental mode by creating a subdirectory for new CSS and
Javascript files. This subdirectory is named with the major
version number of pgbadger.
- Increase the size of the pgbadger logo that appears too small
with the new font size.
- Normalize detailed information in all reports.
- Fix duplicate copy icon in locks report.
- Fix missing chart on histogram of session time. Thanks to
Guillaume Lelarge for the report.
- Add LICENSE file noting the licenses used by the resource
files. Thanks to Christoph Berg for the patch.
- Add patch to jqplot library to fix an infinite loop when trying
to download some charts. Thanks to Julien Tachoires for the help
to solve this issue.
- Script tools/updt_embedded_rsc.pl will apply the patch to resource
file resources/jquery.jqplot.js and doesn't complain if it has
already been applied.
- Remove single last comma at end of pie chart dataset. Thanks to
Julien Tachoires for the report.
- Change display of normalized error
- Remove unused or auto-generated files
- Update all resources files (js+css) and create a directory to
include source of javascript libraries used in pgbadger. There is
also a new script tools/updt_embedded_rsc.pl the can be used to
generate the minified version of those files and embedded them
into pgbadger. This script will also embedded the FontAwesome.otf
open truetype font into the fontawesome.css file.
2016-08-27 - v8.3
This is a maintenance release that fix some minor bugs. This release
also adds replication command messages statistics to the Events
reports.
- Fix auto-detection of stderr format with timestamp as epoch (%n).
- Fix histogram over multiples days to be cumulative per hour, not
an average of the number of event per day.
- Fix parsing of remote file that was failing when the file does
not exists locally. Thanks to clomdd for the report.
- Detect timezones like GMT+3 on CSV logs. Thanks to jacksonfoz
for the patch.
- Add replication command messages statistics to the Events
reports. Thanks to Michael Paquier for the feature request.
This is the last minor version of the 8.x series, next major version
will include an upgrade of boostrap and jquery library which need
some major rewrite.
2016-08-11 version 8.2
This is a maintenance release that fix some minor bug. There is also
some performances improvement up to 20% on huge files and some new
interesting features:
* Multiprocessing can be used with pgbouncer log files.
* pgBouncer and PostgreSQL log files can be used together in
incremental mode.
* With default or same prefix, stderr and syslog file can be
parsed together, csvlog format can always be used.
* Use a modal dialog window to download graphs as png images.
* Add pl/pgSQL function information to queries when available.
Here are the complete list of changes:
- Fix report of database system messages.
- Fix multi line statement concatenation after an error.
- Fix box size for report of queries generating the most
temporary files and the most waiting queries.
- Rewrite code to better handle multi-line queries.
- Fix garbage in examples of event queries with error only mode
(option -w). Thanks to Thomas Reiss for the report.
- Fix getting dataset related to query duration with the use of
auto_explain. Thanks to tom__b for the patch.
- Use a modal dialog window to download graphs as png images.
- Huge rewrite of the incremental mechanism applied to log files
to handle PostgreSQL and pgbouncer logs at the same time.
- Multiprocess can be used with pgbouncer log.
- Add code to remove remaining keyword placeholders tags.
- Fix an other possible case of truncated date in LAST_PARSED file
Thanks to brafaeloliveira for the report.
- Set default scale 1 in pretty_print_number() js function.
- Fix auto-detection of pgbouncer files that contain only stats
lines. Thanks to Glyn Astill for the patch.
- Add date to samples of queries generating most temporary files.
- Do not display warning message of empty log when quiet mode is
enable.
- Fix reading from stdin by disabling pgbouncer format detection.
Thanks to Robert Vargason for the patch.
- Fix case of duplicate normalized error message with "nonstandard
use of ...".
- Fix storage of current temporary file related request.
- Use the mnemonic rather than a signal number in kill calls.
Thanks to Komeda Shinji for the patch.
2016-04-21 version 8.1
This is a maintenance release that fix a major issue introduced with
support to pgbouncer that prevent parsing of compressed PostgreSQL
log files and adds some improvements.
Here are the complete list of changes:
- Fix one case where pid file remain after dying.
- Add requirement of log_error_verbosity = default to documentation.
- Report message "LOG: using stale statistics instead of current
ones because stats collector is not responding" in events view.
- Remove obsolete days when we are in binary mode with --noreport
- Fix wrong report of statements responsible of temporary files.
Thanks to Luan Nicolini Marcondes for the report. This patch also
exclude line with log level LOCATION to be parsed.
- Fix limit on number of sample at report generation and remove
pending LAST_PARSED.tmp file.
- Update load_stat() function and global variables to support
pgbouncer statistics. Update version to 2.0.
- Handle more kind or query types. Thanks to julien Rouhaud for
the patch.
- Fix pgbouncer log parser to handle message: FATAL: the database
system is shutting down
- Fix whitespace placed in between the E and the quote character.
Thanks to clijunky for the report.
- Fix a major issue introduced with support to pgbouncer that
prevent parsing of compressed PostgreSQL log files. Thanks to
Levente Birta for the report.
2016-02-22 version 8.0
This is a major release that adds support to pgbouncer log files.
New pgbouncer reports are:
* Request Throughput
* Bytes I/O Throughput
* Queries Average duration
* Simultaneous sessions
* Histogram of sessions times
* Sessions per database
* Sessions per user
* Sessions per host
* Established connections
* Connections per database
* Connections per user
* Connections per host
* Most used reserved pools
* Most Frequent Errors/Events
pgbouncer log files can be parsed together with PostgreSQL logs.
It also adds a two new command line options:
* --pgbouncer-only to only show pgbouncer related reports.
* --rebuild to be able to rebuild all html reports in incremental
output directory where binary data files are still available.
This release fixes a major bug introduced with journalctl code that
was prevented the use of multiprocess feature.
Here the complete list of other changes:
- Fix progress bar with pgbouncer (only events are increased).
- Sort %SYMBOLE hashtable for remove "!=" / "=" bug. Thanks to
Nicolas Gollet for the patch.
- Fix incorrect numbers on positional parameters in report Queries
generating most temporary files. Thanks to Oskar Wiksten for the
report.
- Update operators list in SQL code beautifier with last update in
pgFormatter. Thanks to Laurenz Albe for the report and the list
of missing operators.
- Cosmetic change to code and add some more debug information.
2016-01-18 version 7.3
This is a maintenance release to fix a major bug that was breaking
the incremental mode in pgBadger. It also adds some more reports and
features.
* Add --timezone=+/-HH to control the timezone used in charts. The
javascript library runs at client side so the timezone used is
the browser timezone so the displayed time in the charts can be
different from the time in the log file.
* Add /tmp/pgbadger.pid file to prevent cron jobs overlaping on
same log files.
* Add command line option --pid-dir to be able to run two pgbadger
at the same time by setting an alternate path to the pid file.
* Report information about "LOG: skipping analyze of ..." into
events reports.
* Report message "LOG: sending cancel to blocking autovacuum" into
events reports. Useful to look for queries generating autovacuum
kill on account of a lock issue.
Here the complete list of changes:
- Automatically remove obsolete pid file when there is no other
pgbadger process running (unix only)
- Update documentation about the --timezone command line option.
- Add --timezone=+/-HH to control the timezone used in charts.
Thanks to CZAirwolf for the report.
- Fix Histogram of session times when there is no data.
- Fix unclosed test file.
- Fix an other case where pgbadger.pid was not removed.
- Always display slides part on connections report even if there
is no data.
- Fix some label on sessions reports
- Add remove of pid file at normal ending.
- Fix wrong size/offset of log files that was breaking incremental
mode. Thanks a lot to CZAirwolf for the report and the help to
find the problem.
- Add command line option --pid-dir to be able to run two pgbadger
at the same time by setting an alternate path to the directory
where the pid file will be written.
- Add /tmp/pgbadger.pid file to prevent cron jobs overlaping on same
log files.
- Report information about "LOG: skipping analyze of ..." into
events reports.
- Report message "LOG: sending cancel to blocking autovacuum" into
events reports. Usefull to know which queries generate autovacuum
kill on account of a lock issue.
- Add more debug information about check log parsing decision.
2016-01-05 version 7.2
This new release fixes some issues especially in temporary files
reports and adds some features.
* Allow pgBadger to parse natively the journalctl command output
* Add new keywords from PG 9.5 for code formating
* Add support to %n log_line_prefix option for Unix epoch (PG 9.6)
There's also some new command line option:
* Adds --journalctl_cmd option to enable this functionality and
set the command. Typically:
--journalctl_cmd "journalctl -u postgresql-9.4"
to parse output of PG 9.4 log
Here is the full list of changes/fixes:
- Fix missing detailed information (date, db, etc.) in Queries
generating the largest temporary files report.
- Fix label of sessions histogram. Thanks to Guillaume Lelarge
for the patch.
- Fix to handle cancelled query that generate more than one
temporary file and more generally aggregate size on queries with
multiple (> 1GB) temporary files.
- Add "Total size" column in Temporary Files Activity table and
fix size increment when a query have multiple 1GB temporary file.
- Fix temporary file query normalization and examples.
- Fix incomplete and wrong queries associated to temporary files
when STATEMENT level log line was missing. Thanks to Mael
Rimbault for the report.
- When -w or --watch-mode is used, message "canceling statement
due to statement timeout" s now reported with other errors.
- Allow dot in dbname and user name. Thanks to David Turvey for
the patch.
- Remove use of unmaintained flotr2 javascript chart library and
use of jqflot instead.
- Fix bad formatting with anonymized values in queries.
- Display 0ms instead of 0s when qery time is under the millisecond.
Thanks to venkatabn for the report.
- Normalize cursor names. Patch from Julien Rouhaud
- Fix unregistered client host name with default pattern. Thanks to
Eric S. Lucinger Ruiz for the report.
- Remove redundant regular expressions.
- Tweaking awkward phrasing, correcting subject-verb agreements,
typos, and misspellings. Patch from Josh Kupershmid.
- Fix potential incorrect creation of subdirectory in incremental mode.
- Allow single white space after duration even if this should not appear.
- Update copyright.
2015-07-11 version 7.1
This new release fixes some issues and adds a new report:
* Distribution of sessions per application
It also adds Json operators to SQL Beautifier.
Here is the full list of changes/fixes:
- Fix unwanted seek on old parsing position when log entry is stdin.
Thanks to Olivier Schiavo for the report.
- Try to fix a potential issue in log start/end date parsing. Thanks
to gityerhubhere for the report.
- Fix broken queries with multiline in bind parameters. Thank to
Nicolas Thauvin for the report.
- Add new report Sessions per application. Thanks to Keith Fiske for
the feature request.
- Add Json Operators to SQL Beautifier. Thanks to Tom Burnett and
Hubert depesz Lubaczewski.
- Makefile.PL: changed manpage section from '1' to '1p', fixes #237.
Thanks to Cyril Bouthors for the patch.
- Update Copyright date-range and installation instructions that was
still refering to version 5. Thanks to Steve Crawford for the report.
- Fix typo in changelog
Note that new official releases must now be downloaded from GitHub and no more
from SourceForge. Download at https://github.com/dalibo/pgbadger/releases
2015-05-08 version 7.0
This major release adds some more useful reports and features.
* New report about events distribution per 5 minutes.
* New per application details (total duration and times executed) for each
query reported in Top Queries reports. The details are visible from a new
button called "App(s) involved".
* Add support to auto_explain extension. EXPLAIN plan will be added together
with top slowest queries when available in log file.
* Add a link to automatically open the explain plan on http://explain.depesz.com/
* New report on queries cumulated durations per user.
* New report about the Number of cancelled queries (graph)
* New report about Queries generating the most cancellation (N)
* New report about Queries most cancelled.
Here is the full list of changes/fixes:
- Update documentation with last reports.
- Fix number of event samples displayed in event reports.
- Add new report about events distribution per x minutes.
- Add app=%a default prefix to documentation.
- Add reports of "App(s) involved" with top queries. Thanks to Antti Koivisto
for the feature request.
- Remove newline between a ) and , in the beautifier.
- Add link to automatically open the explain plan on http://explain.depesz.com/
- Add support to auto_explain, EXPLAIN plan will be added together with top
slowest queries when available in the log file.
- Add a graph on distributed duration per user. Thanks to Korriliam for the
patch.
- Add tree new report: Number of cancelled queries (graph), Queries generating
the most cancellation (N) and Queries most cancelled lists. Thanks to Thomas
Reiss for the feature request.
- Fix case where temporary file statement must be retrieved from the previous
LOG statement and not in the following STATEMENT log entry. Thanks to Mael
Rimbault for the report.
- Add --enable-checksum to show a md5 hash of each reported queries. Thanks
to Thomas Reiss for the feature request.
2015-04-13 version 6.4
This new release fixes a major bugs in SQL beautifier which removed operator
and adds some useful improvement in anonymization of parameters values.
pgBadger will also try to parse the full csvlog when a broken CSV line is
encountered.
- Make anonymization more useful. Thanks to Hubert depesz Lubaczewski
for the patch.
- Fix previous patch for csvlog generated with a PostgreSQL version
before 9.0.
- Try continue CSV parsing after broken CSV line. Thanks to Sergey
Burladyan for the patch.
- Fix bug in SQL beautifier which removed operator. Thanks to Thomas
Reiss for the report.
- Fix loop exit, check terminate quickly and correct comments
indentation. Thanks to Sergey Burladyan for the patch
Please upgrade.
2015-03-27 version 6.3
This new release fixes some bugs and adds some new reports:
* A new per user details (total duration and times executed) for each query
reported in Top Queries reports. The details are visible from a new button
called "User(s) involved".
* Add "Average queries per session" and "Average queries duration per session"
in Sessions tab of the Global statistics.
* Add connection time histogram.
* Use bar graph for Histogram of query times and sessions times.
There's also some cool new features and options:
* Add -L | --logfile-list option to read a list of logfiles from an external
file.
* Add support to log_timezones with + and - signs for timestamp with
milliseconds (%m).
* Add --noreport option to instruct pgbadger to not build any HTML reports
in incremental mode. pgBadger will only create binary files.
* Add auto detection of client=%h or remote=%h from the log so that adding
a prefix is not needed when it respect the default of pgbadger.
* Redefine sessions duration histogram bound to be more accurate.
* Add new option -M | --no-multiline to not collect multi-line statement
and avoid storing and reporting garbage when needed.
* Add --log-duration option to force pgbadger to associate log entries
generated by both log_duration=on and log_statement=all.
The pgbadger_tools script have also been improve with new features:
* Add a new tool to pgbadger_tool to output top queries in CSV format for
follow-up analysis.
* Add --explain-time-consuming and --explain-normalized options to generate
explain statement about top time consuming and top normalized slowest
queries.
Here is the full list of changes/fixes:
- Update flotr2.min.js to latest github code.
- Add per user detail information (total duration and times executed)
for each query reported in "Time consuming queries", "Most frequent
queries" "and Normalized slowest queries". The details are visible
from a new button called "User(s) involved" near the "Examples"
button. Thanks to Guillaume Le Bihan for the patch and tsn77130 for
the feature request.
- pgbadger_tool: add tool to output top queries to CSV format, for
follow-up analysis. Thanks to briklen for the patch.
- Add geometric operators to SQL beautifier. Thanks to Rodolphe
Quiedeville for the report.
- Fix non closing session when a process crash with message:
"terminating connection because of crash of another server process".
Thanks to Mael Rimbault for the report.
- Add -L|--logfile-list command line option to read a list of logfiles
from a file. Thanks to Hubert depesz Lubaczewski for the feature
request.
- Automatically remove %q from prefix. Thanks to mbecroft for report.
- Do not store DEALLOCATE log entries anymore.
- Fix queries histogram where range was not appears in the right order.
Thanks to Grzegorz Garlewicz for the report.
- Fix min yaxis in histogram graph. Thanks to grzeg1 for the patch.
- Add --log-duration command line option to force pgbadger to associate
log entries generated by both log_duration = on and log_statement=all.
Thanks to grzeg1 for the feature request.
- Small typographical corrections. Thanks to Jefferson Queiroz Venerando
and Bill Mitchell the patches.
- Reformat usage output and add explanation of the --noreport command
line option.
- Fix documentation about minimal pattern in custom log format. Thanks
to Julien Rouhaud for the suggestion.
- Add support to log_timezones with + and - signs to timestamp with
milliseconds (%m). Thanks to jacksonfoz for the patch.
pgbadger was not recognize log files with timezones like 'GMT+3'.
- Add --noreport command line option to instruct pgbadger to not build
any reports in incremental mode. pgBadger will only create binary
files. Thanks to hubert Depesz Lubaczewski for the feature request.
- Add time consuming information in tables of Queries per database...
Thanks to Thomas for the feature request.
- Add more details about the CSV parser error. It now prints the line
number and the last parameter that generate the failure. This should
allow to see the malformed log entry.
- Change substitution markup in attempt to fix a new look-behind
assertions error. Thanks to Paolo Cavallini for the report.
- Use bar graph for Histogram of query times and sessions times.
- Fix wrong count of min/max queries per second. Thanks to Guillaume
Lelarge for the report. Add COPY statement to SELECT or INSERT
statements statistics following the copy direction (stdin or stdout).
- Fix Illegal division by zero at line 3832. Thanks to MarcoTrek for
the report.
- Add "Average queries per session" and "Average queries duration per
session" in Sessions tab of the Global stat. Thanks to Guillaume
Lelarge for the feature request.
- Reformat numbers in pie graph tracker. Thanks to jirihlinka for the
report.
- pgbadger_tools: Add --explain-time-consuming and --explain-normalized
to generate explain statement about top time consuming and top
normalized slowest queries. Thanks to Josh Kupershmid fot the feature
request.
- Remove everything than error information from json output when -w |
--watch-mode is enable. Thanks to jason.
- Fix undefined subroutine encode_json when using -x json. Thanks to
jason for the report.
- Add auto detection of client=%h or remote=%h from the log so that
adding a prefix is not needed when it respect the default of pgbadger.
- Redefine sessions duration histogram bound to be more accurate. Thanks
to Guillaume Lelarge for the report.
- Add connection time histogram. Thanks to Guillaume Lelarge for the
feature request.
- Add new option -M | --no-multiline to not collect multi-line statement
to avoid garbage especially on errors that generate a huge report.
- Do not return SUCCESS error code 0 when aborted or something fails.
Thanks to Bruno Almeida for the patch.
2014-10-07 version 6.2
This is a maintenance release to fix a regression in SQL traffic graphs and
fix some other minor issues.
The release also add a new option -D or --dns-resolv to map client ip addresses
to FQDN without having log_hostname enabled on the postgresql's configuration
- Do not display queries in Slowest individual, Time consuming and
Normalized slowest queries reports when there is no duration in
log file. Display NO DATASET instead.
- Fix min/max queries in SQL traffic that was based on duration instead
of query count.
- Fix wrong unit to Synced files in Checkpoints files report. Thanks
to Levente Birta for the report.
- Enable allow_loose_quotes in Text::CSV_XS call to fix CSV parsing
error when fields have quote inside an unquoted field. Thanks to
Josh Berkus for the report.
- Add -D | --dns-resolv command line option to replace ip addresses
by their DNS name. Be warned that this can slow down pgBagder a lot.
Thanks to Jiri Hlinka for the feature request.
2014-09-25 version 6.1
This release fix some issues and adds some new features. It adds a new option
-B or --bar-graph to use bar instead of line in graphs. It will also keep tick
formatting when zooming.
The release also add a new program: pgbadger_tools to demonstrate how to
works with pgBadger binary files to build your own new feature. The first
tools 'explain-slowest' allow printing of top slowest queries as EXPLAIN
statements. There's also additional options to execute automatically the
statements with EXPLAIN ANALYZE and get the execution plan. See help of the
program for more information or the README file in the tools directory.
Some modifications will change certain behavior:
- The -T | --title text value will now be displayed instead of the
pgBadger label right after the logo. It was previously displayed
on mouse over the pgBadger label.
Here is the full list of changes/fixes:
- Change -T | --title position on pgBadger report. Title now override
the pgBadger label. Thanks to Julien Rouhauld for the patch.
- Add --file-per-query and --format-query option to write each slowest
query in separate file named qryXXX.sql and perform minimal formating
of the queries. Thanks to Rodolphe Quiedeville for the patch.
- Remove debug query from explain-slowest tool.
- Fix surge in sessions number report when an exclusion or inclusion
option (dbname, user, appname, etc.) is used. Thanks to suyah for the
report.
- Fix fatal error when remote log file is 0 size. Thanks to Julien
Rouhaud for the report.
- Allow pgbadger_tools --explain-slowest to automatically execute the
EXPLAIN statements an report the plan. See pgbadger_tools --help for
more explanation.
- Add --analyze option to replace EXPLAIN statements by EXPLAIN
(ANALYZE, VERBOSE, BUFFERS).
- Move pgbadger_tools program and README.tools into the tools/
subdirectory with removing the extension. Add more comments and
explanations.
- Fix case where die with interrupt signal is received when using -e
option. Thanks to Lloyd Albin for the report.
- Add a new program pgbadger_tools to demonstrate how to deal with
pgBadger binary files to build your own new feature. The first one
'explain-slowest' allow printing of top slowest queries as EXPLAIN
statements.
- Keep tick formatting when zooming. Thanks to Julien Rouhaud for the
patch.
- Fix automatic detection of rsyslogd logs. Thanks to David Day for
the report.
- Fix issue in calculating min/max/avg in "General Activity" report. It
was build on the sum of queries duration per minutes instead of each
duration. Thanks to Jayadevan M for the report.
- The same issue remains with percentile that are build using the sum of
duration per minutes and doesn't represent the real queries duration.
- This commit also include a modification in convert_time() method to
reports milliseconds.
- Add -B or --bar-graph command line option to use bar instead of line
in graph. Thanks to Bart Dopheide for the suggestion.
- Fix Checkpoint Wal files usage graph title.
2014-08-08 version 6.0
This new major release adds some new features like automatic cleanup of binary
files in incremental mode or maximum number of weeks for reports retention.
It improve the incremental mode with allowing the use of multiprocessing with
multiple log file.
It also adds report of query latency percentile on the general activity table
(percentiles are 90, 95, 99).
There's also a new output format: JSON. This format is good for sharing data
with other languages, which makes it easy to integrate pgBadger's result into
other monitoring tools.
You may want to expose your reports but not the data, using the --anonymize
option pgBadger will be able to anonymize all literal values in the queries.
Sometime select to copy a query from the report could be a pain. There's now
a click-to-select button in front of each query that allow you to just use
Ctrl+C to copy it on clipboard
The use of the new -X option also allow pgBadger to write out extra files to
the outdir when creating incremental reports. Those files are the CSS and
Javascript code normally repeated in each HTLM files.
Warning: the behavior of pgBadger in incremental mode has changed. It will now
always cleanup the output directory of all the obsolete binary file. If you were
using those files to build your own reports, you can prevent pgBadger to remove
them by using the --noclean option. Note that if you use the retention feature,
all those files in obsolete directories will be removed too.
Here is the complete list of changes.
- Javascript improvement to use only one call of sql_select and
sql_format. Use jQuery selector instead of getElementById to
avoid js errors when not found. Thanks to Julien Rouhaud for the
patches.
- Add -R | --retention command line option to set the maximum number of
week reports to preserve in the output directory for incremental mode.
Thanks to Kong Man for the feature request.
- Session count is immediately decreased when a FATAL error is received
in the current session to prevent overcount of simultaneous session
number. Thanks to Josh Berkus for the report.
- Fix issue in incremental mode when parsing is stopped after rotating
log and rotated log has new lines. The new file was not parsed at all.
Thanks to CZAirwolf for the report.
- Fix revert to single thread when last_line_parsed exists. Thanks to
Bruno Almeida for the report.
- Fix issue in handling SIGTERM/SIGINT that cause pgbadger to continue.
- Add autoclean feature to pgbadger in incremental mode. pgbadger will
now removed automatically obsolete binary files unless you specify
--noclean at command line.
- Add new command line option --anonymize to obscure all literals in
queries/errors to hide confidential data. Thanks to wmorancfi for the
feature request.
- Fix single "SELECT;" as a query in a report. Thanks to Marc Cousin for
the report.
- Add a copy icon in front of each query in the report to select the
entire query. Thanks to Josh Berkus for the feature request.
- Fix wrong move to beginning of a file if the file was modified after
have been parsed a time. Thanks to Herve Werner for the report.
- Allow pgBadger to write out extra files to outdir when creating
incremental reports. Require the use of the -X or --extra-files option
in incremental mode. Thanks to Matthew Musgrove for the feature request.
- Fix incomplete handling of XZ compressed format.
- Fix move to offset in incremental mode with multiprocess and incomplete
condition when file is smaller than the last offset. Thanks to Herve
Werner for the report.
- Allow/improve incremental mode with multiple log file and multiprocess.
- Fix incorrect location of temporary file storing last parsed line in
multiprocess+incremental mode. Thanks to Herve Werner for the report.
- Fix remote ssh command error sh: 2: Syntax error: "|" unexpected.
Thanks to Herve Werner for the report.
- Fix missing database name in samples of top queries reports. Thanks to
Thomas Reiss for the report.
- Add minimal documentation about JSON output format.
- Add execute attribute to pgbadger in the source repository, some may
find this more helpful when pgbadger is not installed and executed
directly from this repository.
- Fix issue with csv log format and incremental mode. Thanks to Suya for
the report and the help to solve the issue. There also a fix to support
autovacuum statistic with csv format.
- Fix bad URL to documentation. Thanks to Rodolphe Quiedeville for the report.
- Two minor change to made easier to use Tsung scenario: Remove the first
empty line and replace probability by weight. Now it is possible to use
the scenario as is with Tsung 1.5.
- Fix incremental mode where weeks on index page start on sunday and week
reports start on monday. Thanks to flopma and birkosan for the report.
- Replace label "More CPU costly" by "Highest CPU-cost". Thanks to Marc
Cousin for the suggestion.
- Add query latency percentile to General Activity table (percentiles are
90, 95, 99). Thanks to Himanchali for the patch.
- Fix typon pgbadger call. Thanks to Guilhem Rambal for the report.
- Add JSON support for output format. JSON format is good for sharing data
with other languages, which makes it easy to integrate pgBadger's result
into other monitoring tools like Cacti or Graphite. Thanks to Shanzhang
Lan for the patch.
- Update documentation about remote mode feature.
- Update documentation to inform that the xz utility should be at least in
version 5.05 to support the --robot command line option. Thanks to Xavier
Millies-Lacroix for the report.
- Fix remote logfile parsing. Thanks to Herve Werner for the report.
2014-05-05 version 5.1-1
- Fix parsing of remote log file, forgot to apply some patches.
Thank to Herve Werner for the report.
2014-05-04 version 5.1
This new release fixes several issues and adds several new features like:
* Support to named PREPARE and EXECUTE queries. They are replaced by
the real prepare statement and reported into top queries.
* Add new --exclude-line command line option for excluding immediately
log entries matching any regex.
* Included remote and client information into the most frequent events.
* pgBadger is now able to parse remote logfiles using a password less
ssh connection and generate locally the reports.
* Histogram granularity can be adjusted using the -A command line option.
* Add new detail information on top queries to show when the query is a
bind query.
* Support to logfile compressed using the xz compression format.
* Change week/day menu in incremental index, it is now represented as
usual with a calendar view per month.
* Fix various compatibility issue with Windows and Perl 5.8
Here is the full list of changes:
- fixed calendar display and correct typo. Thanks to brunomgalmeida
for the patch.
- revert to single thread if file is small. Thanks to brunomgalmeida
for the patch.
- print calendars 4+4+4 instead of 3+4+4+1 when looking at full year.
Thanks to brunomgalmeida for the patch.
- Add --exclude-line option for excluding log entries with a regex based
on the full log line. Thanks to ferfebles for the feature request.
- Fix SQL keywords that was beautified twice.
- Remove duplicate pg_keyword in SQL beautifier.
- Fix increment of session when --disable-session is activated.
- Fix missing unit in Checkpoints Activity report when time value is
empty. Thanks to Herve Werner for the report.
- Fix double information in histogram data when period is the hour.
- Add support to named PREPARE and EXECUTE queries. Calls to EXECUTE
statements are now replaced by the prepared query and show samples
with parameters. Thanks to Brian DeRocher for the feature request.
- Included Remote and Client information into the most frequent events
examples. Thanks to brunomgalmeida for the patch.
- Fix documentation about various awkward phrasings, grammar, and
spelling. Consistently capitalize "pgBadger" as such, except for
command examples which should stay all-lowercase. Thanks to Josh
Kupershmidt for the patch.
- Fix incremental mode on Windows by replacing %F and %u POSIX::strftime
format to %Y-%m-%d and %w. Thanks to dthiery for the report.
- Remove Examples button when there is no examples available.
- Fix label on tips in histogram of errors reports.
- Fix error details in incremental mode in Most Frequent Errors/Events
report. Thanks to Herve Werner for the report.
- Fix Sync time value in Checkpoints buffers report. Thanks to Herve
Werner for the report.
- Fix wrong connections per host count. Thanks to Herve Werner for the
report.
- Allow pgBadger to parse remote log file using a password less ssh
connection. Thanks to Orange OLPS department for the feature request.
- Histogram granularity can be adjusted using the -A command line
option. By default they will report the mean of each top queries or
errors occurring per hour. You can now specify the granularity down to
the minute. Thanks to Orange OLPS department for the feature request.
- Add new detail information on top queries to show when the query is
a bind query. Thanks to Orange OLPS department for the feature request.
- Fix queries that exceed the size of the container.
- Add unit (seconds) to checkpoint write/sync time in the checkpoints
activity report. Thanks to Orange OLPS department for the report.
- Fix missing -J option in usage.
- Fix incomplete lines in split logfile to rewind to the beginning of
the line. Thanks to brunomgalmeida for the patch.
- Fix tsung output and add tsung xml header sample to output file.
- Make it possible to do --sample 0 (prior it was falling back to the
default of 3). Thanks to William Moran for the patch.
- Fix xz command to be script readable and always have size in bytes:
xz --robot -l %f | grep totals | awk "{print $5}"
- Add support to logfile compressed by the xz compression format.
Thanks to Adrien Nayrat for the patch.
- Do not increment queries duration histogram when prepare|parse|bind
log are found, but only with execute log. Thanks to Josh Berkus for
the report.
- Fix normalization of error message about unique violation when
creating intermediate dirs. Thanks to Tim Sampson for the report.
- Allow use of Perl metacharacters like [..] in application name.
Thanks to Magnus Persson for the report.
- Fix dataset tip to be displayed above image control button. Thanks
to Ronan Dunklau for the fix.
- Renamed the Reset bouton to "To Chart" to avoid confusion with unzoom
feature.
- Fix writing of empty incremental last parsed file.
- Fix several other graphs
- Fix additional message at end of query or error when it was logged
from application output. Thanks to Herve Werner for the report.
- Fix checkpoint and vacuum graphs when all dataset does not have all
values. Thanks to Herve Werner for the report.
- Fix week numbered -1 in calendar view.
- Change week/day menu in incremental index, it is now represented as
usual with a calendar view per month. Thanks to Thom Brown for the
feature request.
- Load FileHandle to fix error: Can not locate object method "seek"
via package "IO::Handle" with perl 5.8. Thanks to hkrizek for the
report.
- Fix count of queries in progress bar when there is compressed file
and multiprocess is enabled. Thanks to Johnny Tan for the report.
- Fix debug message "Start parsing at offset"
- Add ordering in queries times histogram. Thanks to Ulf Renman for
the report.
- Fix various typos. Thanks to Thom Brown for the patch.
- Fix Makefile error, "WriteMakefile: Need even number of args at
Makefile.PL" with Perl 5.8. Thanks to Fangr Zhang for the report.
- Fix some typo in Changelog
2014-02-05 version 5.0
This new major release adds some new features like incremental mode and SQL
queries times histogram. There is also a hourly graphic representation of the
count and average duration of top normalized queries. Same for errors or events,
you will be able to see graphically at which hours they are occurring the most
often.
The incremental mode is an old request issued at PgCon Ottawa 2012 that concern
the ability to construct incremental reports with successive runs of pgBadger.
It is now possible to run pgbadger each days or even more, each hours, and have
cumulative reports per day and per week. A top index page allow you to go
directly to the weekly and daily reports.
This mode have been build with simplicity in mind so running pgbadger by cron
as follow:
0 23 * * * pgbadger -q -I -O /var/www/pgbadger/ /var/log/postgresql.log
is enough to have daily and weekly reports viewable using your browser.
You can take a look at a sample report at http://dalibo.github.io/pgbadger/demov5/index.html
There's also a useful improvement to allow pgBadger to seek directly to the
last position in the same log file after a successive execution. This feature
is only available using the incremental mode or the -l option and parsing a
single log file. Let's say you have a weekly rotated log file and want to run
pgBadger each days. With 2GB of log per day, pgbadger was spending 5 minutes
per block of 2 GB to reach the last position in the log, so at the end of the
week this feature will save you 35 minutes. Now pgBadger will start parsing
new log entries immediately. This feature is compatible with the multiprocess
mode using -j option (n processes for one log file).
Histogram of query times is a new report in top queries slide that shows the
query times distribution during the analyzed period. For example:
Range Count Percentage
--------------------------------------------
0-1ms 10,367,313 53.52%
1-5ms 799,883 4.13%
5-10ms 451,646 2.33%
10-25ms 2,965,883 15.31%
25-50ms 4,510,258 23.28%
50-100ms 180,975 0.93%
100-500ms 87,613 0.45%
500-1000ms 5,856 0.03%
1000-10000ms 2,697 0.01%
> 10000ms 74 0.00%
There is also some graphic and report improvements, like the mouse tracker
formatting that have been reviewed. It now shows a vertical crosshair and
all dataset values at a time when mouse pointer moves over series. Automatic
queries formatting has also been changed, it is now done on double click
event as simple click was painful when you want to copy some part of the
queries.
The report "Simultaneous Connections" has been relabeled into "Established
Connections", it is less confusing as many people think that this is the number
of simultaneous sessions, which is not the case. It only count the number of
connections established at same time.
Autovacuum reports now associate database name to the autovacuum and autoanalyze
entries. Statistics now refer to "dbname.schema.table", previous versions was only
showing the pair "schema.table".
This release also adds Session peak information and a report about Simultaneous