Prevent deprecated documentation from showing up in search engine top results

[awaelchli]

📚 Documentation

Currently when we search keywords like "pytorch lightning trainer" we get results that point to very very very outdated docs!
It should instead point to the latest stable documentation pages.

Investigate these options here:

• https://docs.readthedocs.io/en/stable/faq.html#how-can-i-avoid-search-results-having-a-deprecated-version-of-my-docs
• https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path

adding a robots.txt is probably enough

cc @alemkhenter

[dipam7]

Hey, @awaelchli I can work on this one. Can you help me understand how the docs are generated and where are the old docs stored?

Thanks

[awaelchli]

Hi, that's great.
Here in the contributing file you find the section about the docs:
https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md#documentation

The docs are hosted on RTD and every time a new lightning version is released, a new build is triggered and stored on the RTD server.

Here is another resource I found:
https://docs.readthedocs.io/en/latest/versions.html#hidden

I am not sure if some of these settings require admin access to the RTD account for lightning
cc @edenlightning @Borda

[Borda]

YES! I am very much to have reached only stable/latest docs, mind sending PR ;]

[dipam7]

Hey, @awaelchli it seems like you were right. RTD creates a default robots.txt file on the RTD server and I would need access to that to change anything. But it seems like a simple Disallow: /path/to/version/ line in the robots.txt should work to avoid earlier versions from being indexed.

I may be able to add a robots.txt under /docs but I don't know how the tree structure of the versions on the server so it'll be best to edit the file autogenerated by RTD.

[awaelchli]

Actually I think this needs to be entirely configured in the pytorch lightning account at RTD. The docs here explain about how one can choose version to be "hidden", i.e. invisible to search engines:

https://docs.readthedocs.io/en/stable/versions.html#version-states

@Borda do you have access to RTD and can set old versions as "hidden"?

[dipam7]

Yes @awaelchli , that's what I meant. Sorry if that was not clear. It has to be done on the RTD account.

[Borda]

@Borda do you have access to RTD and can set old versions as "hidden"?

yes, I can so hide all except last release?

[awaelchli]

Yes, I think everything except stable could be hidden.

[Borda]

[awaelchli]

Oh I didn't know it would affect this list. I think we still need to provide links to all version on the page itself, at least to 1.0. So I'm wondering, is it even possible to hide from search results, but not there in the list?

[Borda]

@awaelchli I have set https://pytorch-lightning.readthedocs.io/robots.txt
not sure what else I can do with other/custom robots.txt

[awaelchli]

I think that is exactly what we need! Looks good. Were you able to edit directly this robots file?

[Borda]

I think that is exactly what we need! Looks good. Were you able to edit directly this robots file?

no, it is generated by setting a version as hidden, https://docs.readthedocs.io/en/latest/versions.html#hidden

[aribornstein]

Resolved this issue.