Introduction

This tool is aimed at helping web developers to make ajax crawlable applications.

It is intended to be a simulator of crawlers that rely on the [ajax crawling] (https://developers.google.com/webmasters/ajax-crawling/) protocol.

It comes with a web application that enables to launch crawl simulations and explore the result.

By default it stores crawl simulations in memory, but it is able to use a Mongo DB database as backend to support data persistence.

Build

This tool is written in Java 8. It build relies on Maven 3. To compile it and get a standalone executable jar, just launch the following command at the repository root:

mvn package

The executable jar is web-launcher/target/web-launcher-1.0-SNAPSHOT-jar-with-dependencies.jar

Run

Once the project is built, it can be simply run thanks to the following command.

java -jar web-launcher/target/web-launcher-1.0-SNAPSHOT-jar-with-dependencies.jar

Use the -h option to discover the launcher options.

java -jar web-launcher/target/web-launcher-1.0-SNAPSHOT-jar-with-dependencies.jar -h

Once the tool is launched, open this URL with your browser:

http://localhost:8080/

This URL changes if the HTTP port is not the default one or if the app does not run in the local system.

Technologies

Here is a non exhaustive list of the technologies used in this tool.

Crawl

• Apache HTTP Client A powerful HTTP client
• jsoup A Java library that is able to parse and manipulate real-word HTML documents

Storage

• MongoDB A NoSQL document-oriented database
• Jongo A high-level MongoDB driver

Web Server

• Fluent HTTP A simple and yet powerful web server
• Guice A dependency injection library
• Jackson A serialization/deserialization library that supports object mapping and many formats such as JSON

Web Client

• AngularJS A very powerful Javascript framework
• Bootstrap 3 A very famous web presentation framework

Testing

• JUnit A very famous Java testing tool
• Hamcrest An library that provides matchers to write concise and readable tests
• Mockito A famous Java mock factory
• Fongo A fake Mongo Java implementation that runs in memory
• RestAssured A web service testing tool
• Wiremock A web service mocking tool
• Awaitility A tool that makes asynchronous testing simple