The crawler opened source by tap4.ai

Simple and powerfull all in one Telegram Bot to scrap / crawl webpages using Requests, html5lib and Beautifulsoup

SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.

Use browser to re-copy a web page

your friendly neighborhood web crawler

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

Web crawler for extracting internal site links info for SEO auditing & optimization purposes

Declarative, scriptable web robot (crawler) and scrapper

Generic Interfaces to Addressable Objects

Tegenaria is a crawler framework based on golang

Fast Crawlbase API crawling library

Example to demonstrate the usage of cached queues across multiple requests.

武汉东湖高新片区光谷&软件园二手房房价爬虫。data source: 房天下

基于Spring Boot、Scrapy 的可视化爬虫配置与管理

Useful functions for connecting to the network in the PHP based applications.

Shark (Plunder)可配置、插件化的爬虫引擎，二次开发框架。Configurable, pluginable crawler engine, secondary development framework.

An advanced web-crawler written in PHP.

Simple crawler using apache nutch and elasticsearch

Blazingly Fast, High Performant, Scalable Web Crawler Engine 💨

An Android app crawling framework, making automatic crawling mobile apps super easy! (if possible, iOS will be supported after Android version)