Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Respect robots.txt #376

Closed
stijn-uva opened this issue Mar 9, 2020 · 1 comment
Closed

Respect robots.txt #376

stijn-uva opened this issue Mar 9, 2020 · 1 comment
Labels
crawler feature

Comments

@stijn-uva
Copy link
Contributor

If I recall correctly, Hyphe currently does not respect robots.txt files. Depending on their ethical framework it may however be important for researchers to be able to exclude pages that are explicitly marked as non-crawlable.

Could this be made an option? I don't know exactly how Hyphe interfaces with Scrapy, but perhaps Scrapy's robots.txt middleware could be toggled for this?

A nice bonus would be if it were possible to also exclude links marked with rel="nofollow", but I see this is already covered in another issue (#86).
@boogheta
Copy link
Member

boogheta commented Mar 9, 2020

Yes that definitely would be a nice feature, and not too complex to implement among other advanced crawl features at least via the API for a start

@stijn-uva stijn-uva mentioned this issue Sep 30, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
crawler feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Setting to make scrapy ignore/follow robots.txt digitalmethodsinitiative/hyphe
2 participants
@boogheta @stijn-uva