Java WARC

This library makes reading WARC files in Java extremely easy. It is open source and compatible with the latest WARC standard (WARC-1.1).

The library is a fork of the "WARC Reader" provided by Mixnode.

Usage examples

These are basic usage examples for the library. When the library was created the main goal was to make something easy to use and lightweight. Because of this the examples are rather short and self-describing.

Stream a WARC file from an URL

Stream a WARC file from an URL and print the payload (response body) to the console.

final URL warcUrl = new URL(
   "https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2018-43/segments/1539583508988.18/warc/CC-MAIN-20181015080248-20181015101748-00000.warc.gz");

WarcRecordStreamFactory.streamOf(warcUrl)
   .filter(WarcRecord::isResponse)
   .map(entry -> ((ResponseContentBlock) entry.getWarcContentBlock()).getPayloadAsString())
   .forEach(System.out::println);

Read WARC records one by one

Read WARC records from a file one by one using the WarcReader class.

final WarcReader warcReader = new WarcReader(new FileInputStream(
    new File("C:\\warc-test\\CC-MAIN-20180716232549-20180717012549-00001.warc.gz")));

boolean hasNext = true;
while (hasNext) {
    try {
        final Optional<WarcRecord> optionalWarcRecord = warcReader.readRecord();

        optionalWarcRecord
            .filter(WarcRecord::isResponse)
            .map(warcRecord -> ((ResponseContentBlock) warcRecord.getWarcContentBlock())
                .getPayloadAsString())
            .ifPresent(System.out::println);

        hasNext = optionalWarcRecord.isPresent();
    } catch (WarcParsionException e) {
        e.printStackTrace();
    }
}

Reactive extensions

If you want a Flux of WarcRecords you should use the reactive module like:

final URL warcUrl = new URL(
    "https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2018-43/segments/1539583508988.18/warc/CC-MAIN-20181015080248-20181015101748-00000.warc.gz");

WarcRecordFluxFactory.buildWarcRecordFlux(warcUrl)
    .filter(WarcRecord::isResponse)
    .map(entry -> ((ResponseContentBlock) entry.getWarcContentBlock()).getPayloadAsString())
    ...

Installation

The library is available in maven central.

You can use it with maven:

<dependency>
  <groupId>com.github.bottomless-archive-project</groupId>
  <artifactId>java-warc</artifactId>
  <version>1.2.0</version>
</dependency>

Or gradle:

implementation 'com.github.bottomless-archive-project:java-warc:1.2.0'

If you want to use the reactive module use you can use it with maven:

<dependency>
  <groupId>com.github.bottomless-archive-project</groupId>
  <artifactId>java-warc-reactive</artifactId>
  <version>1.2.0</version>
</dependency>

Or gradle:

implementation 'com.github.bottomless-archive-project:java-warc-reactive:1.2.0'