trouble parsing nginx log

[ITwrx]

really looking forward to trying out goaccess but i'm having trouble getting my nginx log parsed. goaccesss reports "Message: Nothing valid to process."

an example log entry: "192.168.1.1 - - [08/Jun/2014:14:50:58 -0400] "GET /some_slug HTTP/1.1" 200 5082 localhost"http://192.168.1.2/some_slug2" "Mozilla/5.0 (X11; $(X11; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0""

and my relevant nginx.conf stuff: "## Log format:
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $bytes_sent ''$server_name'
'"$http_referer" "$http_user_agent" ';"

what i have right now in goaccess.conf: "log-format %a - %u [%d/%b/%Y:%T -%z ] "%r" %s %b $v "%R" "%U""

and for date-format in goaccess.conf: "date-format %d/%b/%Y:%T %z"

thanks in advance

[allinurl]

Hi there,

The following works for me (assuming you do not have quotes between the request)

log-format %h %^[%d:%^] "%r" %s %b %^"%R" "%u"
date-format %d/%b/%Y 

Let me know if that works for you!

[ITwrx]

unfortunately it didn't work, but i don't know what you mean by "(assuming you do not have quotes between the request)" so maybe i need to do something different. what i posted above are exact copies except ip addresses and stuff have been changed. everything else is exact. thanks for the help.

[allinurl]

I wasn't sure if each log entry is enclosed in double quotes (first and last char in this case), so if your log entry looks exactly like this:

"192.168.1.1 - - [08/Jun/2014:14:50:58 -0400] "GET /some_slug HTTP/1.1" 200 5082 localhost"http://192.168.1.2/some_slug2" "Mozilla/5.0 (X11; $(X11; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0"" 

Then I'd use these formats:

log-format "%h %^[%d:%^] "%r" %s %b %^"%R" "%u""
date-format %d/%b/%Y 

Also, I don't see a double quote in your sample entry before localhost, however, in the nginx config it appears there should be one right before $server_name. I might be reading it wrong though.

[ITwrx]

oh, ok. i don't have double quotes around the log entries. those were meant to mean "exact copy in here" but i screwed that up b/c this

"## Log format:
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $bytes_sent '
'$server_name'
'"$http_referer" "$http_user_agent" ';"
(not counting the wrapping double quotes)
is actually my nginx.conf entry, not the one with the double quote before localhost($server_name) that i somehow posted above.

i'm not sure what's wrong. what do the "%^" 's mean in your log-format version?

[allinurl]

What flags are you using to run goaccess? Could you try running only the sample entry you posted?This is how I ran your sample:

goaccess -f test.log -c --no-global-config

Then I selected the third option NCSA Combined Log Format and pressed c to edit the log format so I could add the %^ right before the referer, so the formats looks like this

%h %^[%d:%^] "%r" %s %b %^"%R" "%u"

The %^ tells goaccess to ignore a token or not to parse it.

[ITwrx]

ok, i ran your command above against the test.log and it was successful. excellent! i then edited my goaccess config to have the "%^" before the referrer on the NCSA combined log format entry and ran "goaccess -f /srv/http/www-logs/supersite.net-access.log -a" and it ran successfully. however, the command+flags i was trying to run previously was "goaccess -f /srv/http/www-logs/supersite.net-access.log -a > report.html" and i get a permisssion denied on that. i ran it in a directory owned by user http. what user does goaccess run as? also, why are we telling goaccess to ignore the referrer? Thanks for all the help. that ncurses is pretty sporty, btw...

[allinurl]

Typically goaccess should be run as root or whoever owns /srv/http/www-logs/supersite.net-access.log. Running ls -la /srv/http/www-logs/supersite.net-access.log should tell you who owns the log.

The %^ right before the referrer is ignoring $server_name, in the sample entry is ignoring localhost.

[ITwrx]

oh, ok. i just wasn't thinking clearly on the whole permissions issue. i'll take care of that. thanks, though. so, how come we're telling goaccess to ignore the $server_name? do i not have it formatted properly or something?

[allinurl]

Good point, If you are able to generate the report/output with valid data then your format should be fine. Currently goaccess does not use $server_name at all.

[ITwrx]

well, now that i think about it, i don't need $server_name anyways as my logs are separated by site already.
what i'd like to be able to do with goaccess eventually is maybe use the cli version for myself but be able to auto-generate(cron) monthly html reports for each individual site on a given server which also shows previous months, similar to what awstats is able to display. so maybe more config options for the html report generation and maybe access to the template for the report (if it's not already avail) and/or be able to integrate goaccess' html report into a site's cms. To me, that would really set goaccess apart as awstats looked kind of interesting to integrate to server OS/CMS and piwik is larger+heavier and has it's own situation going on as well.
i'll let you get back to it. thanks again and keep up the good work.

[allinurl]

Glad it worked!.

You should be able to generate monthly html reports for each individual site by grepping the virtualhost part (if log entries are stored in the same log) and piping it to goaccess. So a cronjob that looks like the following should do it:

*/10 * * * *   zgrep -i 'example.com:' /var/log/apache2/access.* | /usr/bin/goaccess --no-progress -p /home/user/.goaccessrc > /var/www/example.com/report.html

Currently you can get raw statistics by generating a JSON output and integrate it into your CMS.

Thanks for all the suggestions, will definitely be adding more options to goaccess.

[ssbarnea]

I would like to configure nginx in such was that goaccess would recognize the log format without having to configure it. What would you recommend? Also added at http://serverfault.com/questions/702587/what-would-be-a-portable-nignx-http-access-log-format-to-use-with-nginx

[allinurl]

Quoting from this post:

Nginx by default will output a combined log format (NCSA).

GoAccess will automatically recognize it if you pick the first option from the configuration dialog (or permanently uncomment the NCSA Combined Log Fo

If you are interested to log the request time, then you need to modify your nginx config file and add your custom log format:

vi /etc/nginx/nginx.conf

then add:

log_format timed_combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" 

and change the access log to use the new format:

access_log /var/log/nginx/timed.log timed_combined;

and restart nginx

Make sure your access.log is now logging the request time. Then you can add %T to your goaccess log-format as:

log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u" %T

[ip75]

see carefully locale.
export LANG=en_US.UTF-8 before run goaccess if your month name in English