Skip to content

Latest commit

 

History

History
96 lines (70 loc) · 2.77 KB

README.md

File metadata and controls

96 lines (70 loc) · 2.77 KB
Web Transpose. Simple APIs to get data from the internet.

Web Crawler & AI Web Scraper APIs for building new web experiences.

pip install webtranspose

Introduction · Installation · Docs


Introduction

In the near future, nobody will open websites. Instead, we will be directly served the information we are seeking. New web experiences will combine the information from many websites into a single, unified experience.

Web Transpose is a collection of API tools that allow building these new web experiences simple.

Crawl

import webtranspose as webt

crawl = webt.Crawl(
    "https://www.example.com",
    max_pages=100,
    render_js=True,
)
await crawl.crawl() # crawl.queue_crawl() for async

Scrape

import webtranspose as webt

schema = {
    "Merchant Name": "string",
    "Title of Product": "string",
    "Product Photo URL": "string",
}

scraper = webt.Scraper(
    schema, 
    render_js=True, 
    api_key="YOUR_WEBTRANSPOSE_API_KEY"
)
out_json = scraper.scrape("https://www.example.com")

Installation

Non-Python Users: 📄 API Docs.

This repo contains a local lite installation of Web Transpose. This is a good option if you want to run Web Transpose locally on your machine for quick use cases.

pip install webtranspose[lite]

However, if you wish to leverage the full tools of Web Transpose and use in production, you should install the full version.

pip install webtranspose

Enterprise Support

Web Transpose serves enterprises small and large. We partner with companies for the long term with hands-on support and custom solutions.

Please email me directly at mike@webtranspose.com for enquiries.