Add robots.txt file to doc preview websites for PRs and releases #38694
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kwankyu
changed the title
kwankyu
added
c: documentation
s: needs review
p: blocker / 1
and removed
s: needs review
labels
fchapoton
approved these changes
Sep 22, 2024
|
Documentation preview for this PR (built with commit a08bbfb; changes) is ready! 🎉 |
|
Thanks! |
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Sep 27, 2024
… and releases
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->
We are experiencing an extraordinary surge of access to the doc preview
websites for PRs and releases.
<img width="924" alt="Screenshot 2024-09-22 at 2 24 56 PM"
src="https://github.com/user-
attachments/assets/e7d49ae9-b14e-485f-95bf-3dee2d4f3bc6">
We suspect that it is by web crawlers indexing the websites. This may be
a good thing for our doc sites for stable releases, but a threat of
excessive use of limited bandwidth (100GB monthly) of our netlify (web
publishing service) resource.
Hence we add "robots.txt" file to the websites to discourage web
crawlers.
### 📝 Checklist
<!-- Put an `x` in all the boxes that apply. -->
- [x] The title is concise and informative.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.
### ⌛ Dependencies
<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
URL: sagemath#38694
Reported by: Kwankyu Lee
Reviewer(s): Frédéric Chapoton
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are experiencing an extraordinary surge of access to the doc preview websites for PRs and releases.
We suspect that it is by web crawlers indexing the websites. This may be a good thing for our doc sites for stable releases, but a threat of excessive use of limited bandwidth (100GB monthly) of our netlify (web publishing service) resource.
Hence we add "robots.txt" file to the websites to discourage web crawlers.
📝 Checklist
⌛ Dependencies