New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically Scrape a HUGE amount of url at scale #143

Open

benjaminelkrieff opened this issue Nov 18, 2020 · 0 comments

benjaminelkrieff commented Nov 18, 2020 •

edited

Loading

Hello Everyone.

I am currently working on a project whose goal is to create scrapers in a dynamic way and then being able to process a huge amount of urls at scale.

For example, I have in DDS two websites: www.xxx.it associated with Scraper IT and www.xxx.ca associated with Scraper CA.

I want to send an infinite amount of urls to DDS, and I want all the urls in the form xxx.it/* to be scraped by Scraper IT, and all the urls in the form xxx.ca/* to be scraped by Scraper CA. I want this to be automatically inferred and done by DDS.

Is there a way to achieve this with the current implementation ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment