Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add robots.txt to block bots #94

Closed
3j14 opened this issue Jul 23, 2023 · 0 comments
Closed

Add robots.txt to block bots #94

3j14 opened this issue Jul 23, 2023 · 0 comments
Labels
priority: low

Comments

@3j14
Copy link
Member

3j14 commented Jul 23, 2023

Description
Add a robots.txt file to block bots from crawling the page. As GewässerCampus is a dynamic web app, crawling the page is not helpful. For SEO, the website (www.gewaessercampus.de) is more useful.

I noticed that a significant chunk of request on /measurements come from various bots. These add unnecessary strain.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority: low
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@3j14