How to parse multiple log sources using log-file config option?

[MaraScott]

Hi guys,

Is there anyway to provide multiple log sources to goaccess with a log-file parameter which could looks like : log-file /var/log/*/*/*.log ?

---

the idea behind is to build a common directory to all my logs from host and container within 1 directory like
/my/var/log

in this directory I coul bind volumes like that
docker run -v /my/var/log/myapp/var/log:/var/log [...]
docker run -v /my/var/log/myotherapp/var/log:/var/log [...]
docker run -v /my/var/log/anapp/var/log:/var/log [...]
+
mkdir -p /my/var/log/host/var/log + configure app to redirect logs to that specific dir

Once I have done that, I can bind /my/var/log/ to goaccess container, define log-file /var/log///*.log and the process will take care of parsing all logs, define the proper log-format and generate report(s)

I m not sure it meant to be like that, but that will be so awesome

What do you think ?

[allinurl]

Actually this should work. v1.2 did in fact implement the ability to parse multiple source. However, you will need to feed an actual access log to log-file. The expansion such as access.log* is usually done by the shell and not goaccess.

[MaraScott]

"The expansion such as access.log*" what do you mean ?

[allinurl]

That's right. The shell expands the * into something else (in this instance, the names of the files in the current working directory). But passing absolute paths to log-file should let you specify multiple sources.

[MaraScott]

if I understood, you mean that log-file /var/log/* should allow me to pass all files in /var/log/ to goaccess ?
however it will work for only one log format or can we tell goaccess to self define the log format for each files ?

[allinurl]

Using log-file /var/log/* won't work. It's your shell that expands *, not goaccess. You need to add to your config file each log as:

log-file /var/log/apache2/access.log
log-file /var/log/apache2/access.log.1
log-file /var/log/apache2/access.log.2
log-file /var/log/nginx/access.log
log-file /var/log/nginx/sub/access.log
...

[MaraScott]

my bad, my english level need some improvment :)
thks

can we tell goaccess to self define the log format for each files ?

[allinurl]

No worries :)

What do you mean with self define the log-format? #587 will address this if it's what you are asking.

[MaraScott]

Interesting issue.

Actually what I mean is to let goaccess define by itself which log format it need to process.

I am new to log management so I might say s**t however I m thinking.

If you got log files which contains COMBINED, VCOMBINED, COMMON, VCOMMON & W3C formats in your data dir, goaccess will identified them and process them accordingly to their own format.

Is that possible ?

access.log and error.log from apache2.4 doesn t have the same format

error: [Fri Sep 09 10:42:29.902022 2011] [core:error] [pid 35708:tid 4328636416] [client 72.15.99.187] File does not exist: /usr/local/apache2/htdocs/favicon.ico

access: 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

other_vhosts_access: localhost:80 127.0.0.1 - - [17/Apr/2017:06:36:27 +0000] "GET /robots.txt HTTP/1.1" 200 432 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I m asking this because a ELK stack might have this ability however goaccess is a much lighter and easy to use log analyzer.

[allinurl]

I see, currently it doesn't have the ability to parse multiple log formats. #213 will address this.

[allinurl]

Closing this. #213 will address different log formats.