Reorganize python backend modules

[waseem-polus]

As a...

member of the development team

Wants to...

organize python code into a clean file structure

So that...

more easily dockerize, and host code on AWS lambda

Acceptance Criteria

• separate python code into 3 independent modules (that do not call each other): Scrapers, Cleaners, Models
• These modules are controlled by a main script (app.y) that controls when each module is called
• Each module should have an entry point function
• Each module has the dataflow:
  1. pull from database
  2. perform tasks
  3. push to database
  4. report failure to app.script
• Update entry points in AWS Lambda
• Restructure repo to the following file structure

root
│
├── backend (currently called scrapers)
│   ├── src
│   │   ├── database.py
│   │   ├── scrapers
│   │   │   ├── __init__.py
│   │   │   ├── # scraper files here
│   │   ├── cleaners
│   │   │   ├── __init__.py
│   │   │   └── # cleaner files here
│   │   ├── models
│   │   │   ├── __init__.py
│   │   │   └── # model files here
│   └── app.py
│   └── Dockerfile
│   └── # lambda function entry points
│
└── frontend
    # frontend code here

Resources

• Import statements and file structure reference