New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Look at robots.txt #14

Open

briatte opened this issue Nov 1, 2017 · 1 comment

Assignees

Labels

enhancement important

Owner

briatte commented Nov 1, 2017

https://rud.is/b/2017/09/25/speeding-up-digital-arachinds/

briatte added the enhancement label

briatte self-assigned this

briatte mentioned this issue

Check robots.txt for crawl delays #16

Open

briatte added the important label

Owner Author

briatte commented Aug 16, 2018

Use https://github.com/dmi3kno/polite ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment