logtrix

Examples

Parsing a log file

try (CrawlLogIterator log = new CrawlLogIterator(Paths.get("crawl.log"))) {
    for (CrawlDataItem line : log) {
        System.out.println(line.getStatusCode());
        System.out.println(line.getURL());
    }
}

Grouping the summary by various things

CrawlSummary.byRegisteredDomain(log);
CrawlSummary.byHost(log);
CrawlSummary.byKey(log, item -> item.getCaptureBegan().toString().substring(0, 4)); // by year

Limit top N results

CrawlSummary.build(log).topN(10); // top 10 status codes, mime-types etc

Working with status codes

StatusCodes.describe(404);      // "Not found"
StatusCodes.describe(-4);       // "HTTP timeout"
StatusCodes.isError(-4);        // true
StatusCodes.isServerError(503); // true

Command-line interface

Output a JSON crawl summary grouped by registered domain:

java -jar target/*.jar -g registered-domain crawl.log

For more options:

java -jar target/*.jar --help

Compiling

Install Maven and then run:

mvn package