Playwright is a testing tool for web application, useful also for web scraping, released on 2021.
BEST CHOICE: This is among the preferred tools we use It's the best choice when there's need of a fully rendered browser to scrape a website.
The best configuration we've found up to date against antibot systems consists in:
- playwright_stealth module
- a function to randomize mouse movement
- selection of a consistent combination of device to emulate and browser
- slow_mo option to reduce the rendering speed of the browser
- headless mode
You can find our standard base configuration here.
This configuration is more computing power intensive than a simply scrapy installation so is used only when a fully rendered browser is needed, actually it works pretty well against: