Setup in-house analytics

[hnarayanan]

This is not to track any specific users or anything, but simply to ensure that there are not too many server side errors hiding in sketchy parts of the site. One example: https://github.com/elastic/examples/tree/master/ELK_nginx

[hnarayanan]

Related helpers:

https://harishnarayanan.org/app/kibana#/discover?_g=(refreshInterval:(display:'5%20seconds',pause:!f,section:1,value:5000),time:(from:now%2Fd,mode:quick,to:now%2Fd))&_a=(columns:!(request,response,bytes,agent,user_agent.os,geoip.country_name,remote_ip),filters:!(),index:nginx_elk_example,interval:auto,query:(query_string:(analyze_wildcard:!t,query:'-referrer:*app*%20-referrer:*elasticsearch*%20-request:*app*%20-request:*elasticsearch*')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:geoip.country_code3,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=nginx_elk_example&type=histogram

-referrer:*app* -referrer:*elasticsearch* -request:*app* -request:*elasticsearch*

[hnarayanan]

Extra (yucky) nginx configuration for ELK.

  location ~ ^/app/.* {
    proxy_pass http://127.0.0.1:5601;
    proxy_set_header Host $host;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  }

  location ~ ^/bundles/.* {
    proxy_pass http://127.0.0.1:5601;
    proxy_set_header Host $host;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  }

  location ~ ^/elasticsearch/.* {
    proxy_pass http://127.0.0.1:5601;
    proxy_set_header Host $host;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  }

[hnarayanan]

One significantly easier way to do this is to use a simple program like GoAccess.

Install and configure GoAccess (on Ubuntu 14.04)

sudo apt-get install libncursesw5-dev libglib2.0-dev libgeoip-dev libtokyocabinet-dev
wget http://tar.goaccess.io/goaccess-0.9.8.tar.gz
tar -xzvf goaccess-0.9.8.tar.gz
export PREFIX=$HOME/goaccess-0.9.8
cd goaccess-0.9.8/
./configure --enable-geoip --enable-utf8 --prefix=$PREFIX
make
make install
export PATH=$PREFIX/bin:$PATH

Copy goaccess.conf.txt to $PREFIX/etc/goaccess.conf.

Fetch and prepare Nginx logs

scp -Cr ubuntu@harishnarayanan.org:/var/log/nginx .
cd nginx
zcat -f harishnarayanan.org.* > combined.log

Run GoAccess on the server logs

goaccess -p $PREFIX/etc/goaccess.conf -f combined.log -a > report.html