Do not use puppeteer and playwright for scraping. Or any other browser automation framework for that matter. Why?
We only use the CDP to obtain the page source and to get the absolute coordinates for an arbitrary CSS selector. That's all what is needed for efficient scraping.
- To obtain the page source of the browser's current page. Implemented in page_source.js
- To get the absolute coordinates for an arbitrary CSS selector. Implemented in coords.js
Mouse movements and typing is handled by pyautogui
or other means, but not with JavaScript or with the CDP! Reason: Browser based mouse and keyboard emulation is very easy detectable!
- Analyzing key strokes: TypeNet: Deep Learning Keystroke Biometrics
- Research how to mimic human mouse movements: BeCAPTCHA-Mouse: Synthetic Mouse Trajectories and Improved Bot Detection
The bot challenge that can be found here bot.incolumitas.com/#botChallenge will be solved in the following quick tutorial.
The example code can be found in example.py
.
I am using an Ubuntu 18.04 system with Python3
(with pipenv
) and a recent Node
version.
The browser google-chrome
must be installed.
Clone the repo:
git clone https://github.com/NikolaiT/stealthy-scraping-tools
cd stealthy-scraping-tools
Activate an environment with:
pew new -p python3 sst
pew workon sst
Then install pyautogui
:
pip install pyautogui
Install node modules:
npm install chrome-remote-interface
And then run the bot with:
python example.py
The Dockerfile is based on Ubuntu 20.04
.
The Dockerfile uses xvfb
from the python module PyVirtualDisplay
.
I use pyautogui
for mouse and keyboard automation.
I use fluxbox
as a tiny window manager and x11vnc
(Virtual Network Computing server program) as a means to inspect the docker image and see what is going on.
Build the Dockerfile:
docker build -t sst:0.0.1 .
Hint: Avoid chrome in docker crashing: stephen-fox/chrome-docker#8
1. Option 1: Run chrome with --disable-dev-shm-usage
2. Option 2: Set /dev/shm size to a reasonable amount `docker run -it --shm-size=1g` replacing 1g with whatever amount you want.
The docker option --shm-size=2g
is really important:
docker run --cap-add=NET_ADMIN --network="host" --shm-size=2g sst:0.0.1
- Look at Kernel/OS level mouse/keyboard control commands (Ditch
pyautogui
) - Use the math from ghost-cursor
- Create a set of typign recordings and use it to derive rules for bot writing