Serverless Web Scraping in Python for AI, Fun and Profit

(Using Step Functions and Lambdas)

This material is also covered in Chapter 7 of Pragmatic AI

Web Scraping for AI/ML consists of three phases:

A. Doing the Work

B. Scheduling the Work

C. Modeling the Work

A. Web Scraping Techniques (Doing the Work)

• Part 1: Jupyter Notebook: Pandas, Requests and APIs
• Part2: Jupyter Notebook: Scraping

B. Orchestrating Retrieval (Scheduling the Work)

• Part3: Jupyter Notebook: Using AWS Lambda and AWS Step Functions

C. Wiring Results into A Machine Learning Pipeline (Modeling the Work)

• TBD