TOP News: Real Time News Scraping and Recommendation System

Implemented a data pipeline which monitors, scrapes and dedupes latest news (MongoDB, Redis, RabbitMQ);
Designed data monitors for obtaining latest news from famous websites and recommend to web server.
Successfully fetch useful data from original news websites by building news scrapers.
Build dedupers which filter same news by using NLP (TF-IDF) to analyze similarities of articles scraped from news websites.
Use Tensorflow for machine learning which can shows news according to users interests. Build a single-page web.

System Overview

Name		Name	Last commit message	Last commit date
Latest commit History 62 Commits
backend_server		backend_server
images_for_readme		images_for_readme
news_recommendation_service		news_recommendation_service
news_topic_modeling_service		news_topic_modeling_service
pipeline		pipeline
web_server		web_server
.gitignore		.gitignore
README.md		README.md
WORKPLAN.md		WORKPLAN.md
dashboard_launcher.sh		dashboard_launcher.sh
end_service.sh		end_service.sh
requirements.txt		requirements.txt
start_news_pipline.sh		start_news_pipline.sh
start_web.sh		start_web.sh