Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add robotsNoIndex parameter #900

Merged
merged 3 commits into from
May 22, 2022
Merged

add robotsNoIndex parameter #900

merged 3 commits into from
May 22, 2022

Conversation

martin2250
Copy link
Contributor

What does this PR change? What problem does it solve?
This PR adds the ability to use the noindex and nofollow parameters for specific pages in production mode, preventing e.g. the imprint from showing up in search engines. It also lists these pages in the robots.txt file.

Was the change discussed in an issue or in the Discussions before?
#712 also touches this line, but it adds other functionality as well and doesn't include robots.txt.

Cheers!

PR Checklist

  • This change adds/updates translations and I have used the template present here.
  • I have enabled maintainer edits for this PR.
  • I have verified that the code works as described/as intended.
  • This change adds a Social Icon which has a permissive license to use it.
  • This change does not include any CDN resources/links.
  • This change does not include any unrelated scripts such as bash and python scripts.
  • This change updates the overridden internal templates from HUGO's repository.

@Thunder33345
Copy link

Thunder33345 commented May 6, 2022 •
edited

You shouldnt use Disallow: in robots.txt as it will stop crawlers from seeing the noindex directive, thus leaving the possibility of site still being indexed anyways.

However, robots.txt disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant and show the URL in the results. If you wish to explicitly block a page from being indexed, use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, don't disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.
Cite: Google's Robots FAQs

@sonarcloud
Copy link

sonarcloud bot commented May 8, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@martin2250
Copy link
Contributor Author

@Thunder33345 Thanks, good call!

@adityatelange adityatelange merged commit 7b9c836 into adityatelange:master May 22, 2022
romainx pushed a commit to romainx/hugo-PaperMod that referenced this pull request Jul 26, 2022
@martin2250 @romainx
Add robotsNoIndex parameter (adityatelange#900)
d1297cc 
- adds the ability to use the noindex and nofollow parameters for specific pages in production mode, preventing e.g. the imprint from showing up in search engines.
- Usage: In post front-matter add,
    ---
    robotsNoIndex: true
    ---
minyeamer pushed a commit to minyeamer/hugo-PaperMod that referenced this pull request Sep 16, 2022
@martin2250
Add robotsNoIndex parameter (adityatelange#900)
a569500 
- adds the ability to use the noindex and nofollow parameters for specific pages in production mode, preventing e.g. the imprint from showing up in search engines.
- Usage: In post front-matter add,
    ---
    robotsNoIndex: true
    ---
dunklecat pushed a commit to dunklecat/hugo-PaperModX that referenced this pull request Oct 8, 2022
@martin2250 @dunklecat
Add robotsNoIndex parameter (adityatelange#900)
9f327a5 
- adds the ability to use the noindex and nofollow parameters for specific pages in production mode, preventing e.g. the imprint from showing up in search engines.
- Usage: In post front-matter add,
    ---
    robotsNoIndex: true
    ---
kylethedeveloper pushed a commit to kylethedeveloper/hugo-PaperMod that referenced this pull request Feb 21, 2023
@martin2250
Add robotsNoIndex parameter (adityatelange#900)
908f1c7 
- adds the ability to use the noindex and nofollow parameters for specific pages in production mode, preventing e.g. the imprint from showing up in search engines.
- Usage: In post front-matter add,
    ---
    robotsNoIndex: true
    ---
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adityatelange adityatelange adityatelange approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@martin2250 @Thunder33345 @adityatelange