Parse Nginx Proxy Manager

[GAS85]

Hey, love your product!
I read #2331 and #892, but still do not catch how I can parse this log correctly.

1. There are 2 status codes - on the Proxy and from the backend, like 200 200 or - 301, but I can't skip first field that should be $upstream_status
2. I would like to use $scheme, but when I add %H it does not work. So I skip it
3. Also there is no URL and Visitors counted.
4. Status codes only 404s are shown on the graph.

• goaccess version: 1.7.

Log configuration in nginx:

[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"

goaccess parameters:

(cat /var/docker/data/npm/data/logs/*_access.log ; zcat /var/docker/data/npm/data/logs/*_access.*.gz) | goaccess -a -o html --log-format '[%d:%t %^] %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to $^] "%u" "%R"' --time-format %T --date-format %d/%b/%Y --http-protocol=yes - > all_report.html

Logs example:

[11/Jan/2023:02:16:53 +0100] - - 301 - GET http mycooldomain.com "/" [Client 10.10.10.10] [Length 166] [Gzip -] [Sent-to dokuwiki] "Expanse indexes customers\xE2\x80\x99 network perimeters. If you have any questions or concerns, please reach out to: scaninfo@xxx." "-"
[11/Jan/2023:03:17:44 +0100] - 200 200 - GET https mycooldomain.com "/" [Client 10.10.10.10] [Length 38893] [Gzip -] [Sent-to dokuwiki] "Expanse indexes customers\xE2\x80\x99 network perimeters. If you have any questions or concerns, please reach out to: scaninfo@xxx." "-"
[11/Jan/2023:04:11:52 +0100] - - 301 - GET http mycooldomain.com "/_ignition/health-check/" [Client 10.10.10.10] [Length 166] [Gzip -] [Sent-to dokuwiki] "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" "-"
[11/Jan/2023:04:11:52 +0100] - 404 404 - GET https mycooldomain.com "/_ignition/health-check/" [Client 10.10.10.10] [Length 715] [Gzip 2.96] [Sent-to dokuwiki] "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" "http://mycooldomain.com/_ignition/health-check/"
[11/Jan/2023:04:11:54 +0100] - - 301 - GET http mycooldomain.com "/public/_ignition/health-check/" [Client 10.10.10.10] [Length 166] [Gzip -] [Sent-to dokuwiki] "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" "-"
[11/Jan/2023:04:11:54 +0100] - 404 404 - GET https mycooldomain.com "/public/_ignition/health-check/" [Client 10.10.10.10] [Length 715] [Gzip 2.96] [Sent-to dokuwiki] "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" "http://mycooldomain.com/public/_ignition/health-check/"
[11/Jan/2023:04:11:56 +0100] - - 301 - GET http mycooldomain.com "/wp-includes/wlwmanifest.xml" [Client 10.10.10.10] [Length 166] [Gzip -] [Sent-to dokuwiki] "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" "-"
[11/Jan/2023:04:11:56 +0100] - 404 404 - GET https mycooldomain.com "/wp-includes/wlwmanifest.xml" [Client 10.10.10.10] [Length 715] [Gzip 2.96] [Sent-to dokuwiki] "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" "http://mycooldomain.com/wp-includes/wlwmanifest.xml"
[11/Jan/2023:09:27:27 +0100] - - 301 - GET http mycooldomain.com "/" [Client 10.10.10.10] [Length 166] [Gzip -] [Sent-to dokuwiki] "Expanse indexes customers\xE2\x80\x99 network perimeters. If you have any questions or concerns, please reach out to: scaninfo@xxx." "-"
[11/Jan/2023:11:33:06 +0100] - 200 200 - GET https mycooldomain.com "/" [Client 10.10.10.10] [Length 38893] [Gzip -] [Sent-to dokuwiki] "Expanse indexes customers\xE2\x80\x99 network perimeters. If you have any questions or concerns, please reach out to: scaninfo@xxx." "-"

[allinurl]

Please give this a shot, let me know how it goes:

goaccess access.log --log-format='[%d:%t %^] %^ %s %^ %e %m %^ %v "%U" [%^ %h] [%^ %b] %^"%u" "%R"' --no-strict-status --date-format=%d/%b/%Y --time-format=%T --http-protocol=no

[GAS85]

Thanks for answer, I modify a bit for my purpose and it works well.
Basically there are 5 kind of logs with 3 different formats will be generated by NPM:

1. Proxy logs - Services that are set as Proxy proxy-host-*_access.log
2. Dead Host - Services that are set as "404" Hosts dead-host-*_access.log
3. Default Host - Request that are not fitting into any configuration default-host_access.log
4. Fallback Host - IDK what is here, but a lot of trash fallback_access.log
5. Redirection Hosts - Services configure as redirect redirection-host-*_access.log

Problem - they have a bit different Logs format, so we need different command for them:

1. Proxy Logs:

   --log-format='[%d:%t %^] %C %^ %s %^ %m %^ %v "%U" [%^ %h] [%^ %b] %^"%u" "%R"' \
   --date-format=%d/%b/%Y \
   --time-format=%T

   Command could be:

   goaccess access.log --log-format='[%d:%t %^] %C %^ %s %^ %m %^ %v "%U" [%^ %h] [%^ %b] %^"%u" "%R"' --no-strict-status --date-format=%d/%b/%Y --time-format=%T

2. Dead Host Logs:

   --log-format='[%d:%t %^] %s %^ %m %^ %v "%U" [%^ %h] [%^ %b] %^"%u" "%R"' \
   --date-format=%d/%b/%Y \
   --time-format=%T \

   Command could be:

   goaccess access.log --log-format='[%d:%t %^] %s %^ %m %^ %v "%U" [%^ %h] [%^ %b] %^"%u" "%R"' --no-strict-status --date-format=%d/%b/%Y --time-format=%T

3. Default Host Logs:

   --log-format='%h %^ %^ [%d:%t %^] "%m %U %H" %s %b "%^" "%u"' \
   --date-format=%d/%b/%Y \
   --time-format=%T

   Command could be:

   goaccess access.log --log-format='%h %^ %^ [%d:%t %^] "%m %U %H" %s %b "%^" "%u"' --no-strict-status --date-format=%d/%b/%Y --time-format=%T

4. Fallback host has the same format as Dead Hosts.
5. Redirection hosts has the same format as Dead Hosts.

[allinurl]

Glad that worked and thanks for sharing those findings. There's a request #213 to support multiple log format. I may get to it sooner than later. Stay tuned.