Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Robots.txt deploy for domserver on online instance #2654

Closed
vmcj opened this issue Apr 18, 2024 · 2 comments · Fixed by #2808
Closed

Robots.txt deploy for domserver on online instance #2654

vmcj opened this issue Apr 18, 2024 · 2 comments · Fixed by #2808
Labels
frontend good first issue web interface

Comments

@vmcj
Copy link
Member

vmcj commented Apr 18, 2024 •
edited by eldering
Loading

As suggested by @meisterT, we should deploy a robots.txt to prevent spiders from triggering Symfony errors when they visit our POST only routes.
@eldering
Copy link
Member

Could we include that by default in the domjudge repo itself? I guess you'd never want this behaviour.

@vmcj
Copy link
Member Author

vmcj commented Aug 24, 2024

The traditional robots.txt will not work when you deploy DOMjudge in a subdirectory (https://stackoverflow.com/q/4837334)
So we can either set a <meta> or a header, see: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Because I prefer to not fill the DOM with extra tags if not needed I think we should do this with another header (and copy this manually to the domjudge-scripts repo).

@vmcj vmcj transferred this issue from DOMjudge/domjudge-scripts Aug 24, 2024
vmcj added a commit to vmcj/domjudge that referenced this issue Nov 19, 2024
@vmcj
Prevent indexing by search engines by default
042c4eb 
Some search engines trigger the markdown-preview which is POST only so
we would get sentry errors for this. That lead to the discussion that we
don't want search engines to index DOMjudge instances without the user
specifically choosing this.

Fixes: DOMjudge#2654
@vmcj vmcj mentioned this issue Nov 19, 2024
Merged
vmcj added a commit to DOMjudge/domjudge-scripts that referenced this issue Nov 19, 2024
@vmcj
Prevent indexing of DOMjudge instance
27d8cb6 
See: DOMjudge/domjudge#2654
@vmcj vmcj mentioned this issue Nov 19, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Nov 19, 2024
@vmcj
Prevent indexing by search engines by default
7c890e8 
Some search engines trigger the markdown-preview which is POST only so
we would get sentry errors for this. That lead to the discussion that we
don't want search engines to index DOMjudge instances without the user
specifically choosing this.

Fixes: #2654
vmcj added a commit to DOMjudge/domjudge-scripts that referenced this issue Nov 21, 2024
@vmcj
Prevent indexing of DOMjudge instance
7036259 
See: DOMjudge/domjudge#2654
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
frontend good first issue web interface
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prevent indexing by search engines by default vmcj/domjudge
2 participants
@eldering @vmcj