FatCache / camelcrawler Public

forked from jiaminghong/camelcrawler

Notifications You must be signed in to change notification settings
Fork 0
Star 0

Distributed web crawler engine 🌎🕷️ written in Scala. Akka framework used for concurreny and data persistence on Redis

0 stars 3 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
front-end		front-end
project		project
src		src
target		target
README.md		README.md
build.sbt		build.sbt

Repository files navigation

Camel Crawler

To-Do

Web Crawler

Write basic crawler to get list of URLs for single domain
Feed base root URLs from text file
Convert the list of domains into a data frame/structure for analysis
Connect crawler to a persistence database [MySQL]

Middleware develops API for the database - Redis

Connect to a database - Redis
Create APIs...

Team Members

Abdusamed
Ming
Chi

About

Distributed web crawler engine 🌎🕷️ written in Scala. Akka framework used for concurreny and data persistence on Redis

Report repository

Releases

No releases published

Packages

No packages published

Languages

XSLT 69.8%
Scala 15.7%
CSS 14.5%