Repositories list

• scrapy-zyte-api

  Public
  Zyte API integration for Scrapy

  plugincrawlerproxy+ 2scrapingscrapy

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •20•36•11•4•Updated Dec 18, 2024Dec 18, 2024

• scrapy-playwright

  Public
  🎭 Playwright integration for Scrapy

  hacktoberfestchrome-headlesspython-asyncio+ 9headless-browserjavascript-rendererfirefox-headlessplaywrightplaywright-pythonwebkit-headlesspython+ 2

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •119•1.1k•19•2•Updated Nov 14, 2024Nov 14, 2024

• scrapy-zyte-smartproxy

  Public
  Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

  plugincrawlerproxy+ 3scrapingscrapycrawler-detection

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •88•359•6•2•Updated Nov 7, 2024Nov 7, 2024

• scrapy-spider-metadata

  Public
  Python

  •

  MIT License

  •2•4•2•0•Updated Oct 16, 2024Oct 16, 2024

• zyte-spidermon

  Public
  Python

  •0•0•0•0•Updated Sep 5, 2024Sep 5, 2024

• scrapy-settings-log

  Public
  An extension that allows a user to display all or some of their scrapy spider settings at runtime.

  Python

  •

  GNU General Public License v3.0

  •4•0•0•0•Updated Aug 20, 2024Aug 20, 2024

• scrapy-feedexporter-dropbox

  Public
  Scrapy feed exporter for Dropbox

  Python

  •3•1•0•0•Updated Jul 11, 2024Jul 11, 2024

• scrapy-crawlera-fetch

  Public archive
  Scrapy Downloader Middleware for Crawlera Fetch API

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •10•8•5•5•Updated Jun 5, 2024Jun 5, 2024

• scrapy-feedexporter-azure-storage

  Public
  Python

  •5•2•0•3•Updated May 18, 2024May 18, 2024

• scrapy-feedexporter-onedrive

  Public
  Export to OneDrive

  Python

  •1•1•0•0•Updated Feb 23, 2024Feb 23, 2024

• scrapy-djangoitem

  Public
  Scrapy extension to write scraped items using Django models

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •87•498•10•3•Updated Oct 15, 2023Oct 15, 2023

• scrapy-feedexporter-google-sheets

  Public
  Python

  •0•1•0•0•Updated Aug 15, 2023Aug 15, 2023

• scrapy-incremental

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1•1•0•0•Updated May 1, 2023May 1, 2023

• scrapy-splash

  Public
  Scrapy+Splash for JavaScript integration

  scrapyheadless-browsers

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •454•3.2k•65•15•Updated Feb 6, 2023Feb 6, 2023

• scrapy-feedexporter-google-drive

  Public
  Python

  •2•3•1•0•Updated Jan 4, 2023Jan 4, 2023

• scrapy-statsd

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •4•6•1•3•Updated Oct 19, 2022Oct 19, 2022

• scrapy-dotpersistence

  Public
  A scrapy extension to sync `.scrapy` folder to an S3 bucket

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •8•17•2•1•Updated Mar 28, 2022Mar 28, 2022

• scrapy-magicfields

  Public
  Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •7•56•0•0•Updated Mar 16, 2022Mar 16, 2022

• scrapy-pagestorage

  Public
  A scrapy extension to store requests and responses information in storage service

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •6•26•2•0•Updated Mar 11, 2022Mar 11, 2022

• scrapy-snowflake-stage-exporter

  Public
  Snowflake database loading utility with Scrapy integration

  snowflakescrapyetl

  Python

  •2•4•0•0•Updated Dec 3, 2021Dec 3, 2021

• scrapy-deltafetch

  Public
  Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls

  hacktoberfesthacktoberfest2021

  Python

  •49•269•14•5•Updated Oct 17, 2021Oct 17, 2021

• scrapy-headless

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •8•29•2•2•Updated Apr 28, 2021Apr 28, 2021

• scrapy-jsonschema

  Public
  Scrapy schema validation pipeline and Item builder using JSON Schema

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •12•44•1•0•Updated Mar 26, 2021Mar 26, 2021

• scrapy-feedexporter-sftp

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •3•6•1•1•Updated May 14, 2020May 14, 2020

• scrapy-jsonrpc

  Public
  Scrapy extension to control spiders using JSON-RPC

  Python

  •72•296•7•2•Updated Aug 26, 2019Aug 26, 2019

• scrapy-hcf

  Public
  Scrapy spider middleware to use Scrapinghub's Hub Crawl Frontier as a backend for URLs

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •6•4•1•1•Updated Aug 28, 2018Aug 28, 2018

• scrapy-monkeylearn

  Public
  A Scrapy pipeline to categorize items using MonkeyLearn

  Python

  •13•37•0•0•Updated Apr 28, 2017Apr 28, 2017

• scrapy-splitvariants

  Public
  Scrapy spider middleware to split an item into multiple items using a multi-valued key

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1•20•0•0•Updated Feb 8, 2017Feb 8, 2017

• scrapy-streaming

  Public
  Python

  •5•18•2•6•Updated Oct 12, 2016Oct 12, 2016

• scrapy-querycleaner

  Public
  Scrapy spider middleware to clean up query parameters in request URLs

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1•25•0•0•Updated Jun 30, 2016Jun 30, 2016