Scrapy-Project

This is a project for scraping data from websites

Packages Used

• scrapy
• psycopg2 or psycopg2-binary

Database Used

• PostgreSQL

File name and Directory:

• scrapeQuote.py
• Directory: helloScrappy>helloScrappy>spiders>scrapeQuote.py

Installation

After cloning the project at first please install the following packages.

• Install scrapy
• Install psycopg2 or psycopg2-binary

• scrapy-fake-useragent

You can install these packages using:

• pip install -r requirements.txt

For Database

In this project PostgreSQL is used.For installation of PostgreSQL please follow the link below.

For Ubuntu:

• https://www.digitalocean.com/community/tutorials/how-to-install-postgresql-on-ubuntu-20-04-quickstart

• https://kodemonk.dev/blog/installing-pgadmin-4-in-ubuntu-20-04

For Windows:

• https://www.postgresql.org/download/windows/
• https://www.postgresqltutorial.com/install-postgresql/

After installing the packages please create a database. Provide your database name and password in the following line

conn = psycopg2.connect(dbname="Your database name",user="postgres",password="Your Password", host="localhost")

• You can also do this by creating a database named 'newdata' and importing the mydb.sql file in pgAdmin4.

Running the project

After completing the database setup and installing the packages, Please go to the spiders directory inside helloScrappy directory and run the following command:

• scrapy crawl quotes

This will crate a database table with all the data in postgresql database.