Mark outdated docs

[pfitzseb]

This adds a notice to all old docs pointing visitors to /stable as well as a <meta content="noindex" name="robots"/> annotation to stop search engines from showing the page.

Diff was generated with https://gist.github.com/pfitzseb/4b910449e21b51fed4b4a2a7c20912ff.

Edit: The diff is huge and does stuff like replacing " with ", but all of that should be perfectly safe.

[bkamins]

Thank you. This PR is impossible to review :). I will just wait till you feel it is ready and review selected files, then merge it and test on-line if all is OK.

[pfitzseb]

Ok, so the reason I marked this as WIP is that I was thinking about unconditionally adding the JS in Documenter.jl, but I think this approach is better because it allows us to statically insert these noindex-annotations, which should be better for SEO. The changes are also self contained (apart from the reformatting Gumbo does when printing the rest of the DOM), so it'd be easy to adjust the script further down the line if necessary.

[bkamins]

@nalimilan - I have checked it manually and it looks good (admittedly I have not checked all 1064 files, but a random sample looks OK). Do you want to check it additionally somehow?

This is a structure of the folders after this PR:

CNAME         stable        v0.12.0  v0.17.1  v0.20.2  v0.22.3  v0.8.0
dev           v0.10         v0.13    v0.18    v0.21    v0.22.4  v0.8.5
docs          v0.10.0       v0.13.0  v0.18.0  v0.21.0  v0.22.5  v0.9
index.html    v0.10.1       v0.13.1  v0.18.1  v0.21.1  v0.22.6  v0.9.0
latest        v0.11         v0.14    v0.18.2  v0.21.2  v0.22.7  v0.9.1
release-0.10  v0.11.0       v0.14.0  v0.18.3  v0.21.3  v0.6     v1
release-0.11  v0.11.0-docs  v0.14.1  v0.18.4  v0.21.4  v0.6.11  v1.0
release-0.12  v0.11.1       v0.15    v0.19    v0.21.5  v0.7     v1.0.0
release-0.13  v0.11.2       v0.15.0  v0.19.0  v0.21.6  v0.7.0   v1.0.1
release-0.14  v0.11.3       v0.15.1  v0.19.1  v0.21.7  v0.7.1   v1.0.2
release-0.15  v0.11.4       v0.15.2  v0.19.2  v0.21.8  v0.7.2   v1.1
release-0.16  v0.11.5       v0.16    v0.19.3  v0.22    v0.7.3   v1.1.0
release-0.17  v0.11.6       v0.16.0  v0.19.4  v0.22.0  v0.7.7   v1.1.1
release-0.18  v0.11.7       v0.17    v0.20    v0.22.1  v0.7.8   versions.js
release-0.19  v0.12         v0.17.0  v0.20.0  v0.22.2  v0.8

and it is the same as gh-pages branch:

CNAME         stable        v0.12.0  v0.17.1  v0.20.2  v0.22.3  v0.8.0
dev           v0.10         v0.13    v0.18    v0.21    v0.22.4  v0.8.5
docs          v0.10.0       v0.13.0  v0.18.0  v0.21.0  v0.22.5  v0.9
index.html    v0.10.1       v0.13.1  v0.18.1  v0.21.1  v0.22.6  v0.9.0
latest        v0.11         v0.14    v0.18.2  v0.21.2  v0.22.7  v0.9.1
release-0.10  v0.11.0       v0.14.0  v0.18.3  v0.21.3  v0.6     v1
release-0.11  v0.11.0-docs  v0.14.1  v0.18.4  v0.21.4  v0.6.11  v1.0
release-0.12  v0.11.1       v0.15    v0.19    v0.21.5  v0.7     v1.0.0
release-0.13  v0.11.2       v0.15.0  v0.19.0  v0.21.6  v0.7.0   v1.0.1
release-0.14  v0.11.3       v0.15.1  v0.19.1  v0.21.7  v0.7.1   v1.0.2
release-0.15  v0.11.4       v0.15.2  v0.19.2  v0.21.8  v0.7.2   v1.1
release-0.16  v0.11.5       v0.16    v0.19.3  v0.22    v0.7.3   v1.1.0
release-0.17  v0.11.6       v0.16.0  v0.19.4  v0.22.0  v0.7.7   v1.1.1
release-0.18  v0.11.7       v0.17    v0.20    v0.22.1  v0.7.8   versions.js
release-0.19  v0.12         v0.17.0  v0.20.0  v0.22.2  v0.8

[nalimilan]

I trust you that it's OK. ;-)

In the future, will it be necessary to apply these changes after each new release? Or is Documenter supposed to handle this?

[bkamins]

In the future, will it be necessary to apply these changes after each new release? Or is Documenter supposed to handle this?

This is a good point.

@pfitzseb - we probably need some script that would handle this after the release. Right?
I can run this script when making a release but we need to have it stored somewhere. So the questions are:

1. is the thinking correct?
2. if yes do you have such a script (and if yes could you please share it so that I describe the release procedure in the documentation for the future)

Thank you!

[pfitzseb]

Ok, so I read up on SEO a bit more today and apparently modern search engines can handle dynamic meta tags just fine, so my current thinking is to get a PR to Documenter done that automatically displays the warning for future old versions.

Let's hold off on merging this PR in the meantime, although I don't really anticipate any significant changes.

[bkamins]

OK - waiting for the update then. Thank you!

[bkamins]

Why is this closed?

[pfitzseb]

Because I force pushed (what I thought is) a new commit without ever actually creating that commit.
This now uses JuliaDocs/DocumenterTools.jl#51 to create the warning. Also see JuliaDocs/Documenter.jl#1577.

[bkamins]

But should we merge this PR now, or wait for the Documenter.jl changes?
Also I assume that the linked changes - when merged and tagged will in the future automatically mark old versions if they are not marked yet?

[pfitzseb]

Yeah good question. I'd probably wait until those PRs are merged.

And no, nothing will happen automatically for old versions. It's (will be) as easy as DocumenterTools.OutdatedWarning.generate(".") though.

[bkamins]

OK - let us wait then. Can you please bump this PR when you feel it is ready to be merged. Thank you!

[bkamins]

@pfitzseb - what is the status here? (no rush, but I do not want to wait too long with the decision as then everyone will forget the status - thank you!)

[pfitzseb]

Uhm, good question. I think this should be good to go now; I've updated this PR with the output of OutdatedWarning.generate from JuliaDocs/DocumenterTools.jl#51, which should be ready to merge.

We'll need one more run before a new Documenter version is released that automatically adds the warning though.

[bkamins]

In summary (just to double check) - merging this now should be OK?

[pfitzseb]

Yeah, I don't anticipate any more (breaking) changes to either of those Documenter PRs.

[bkamins]

I am merging it then. I will review later how it looks like on production.

@nalimilan @pfitzseb - also could you please have a look then to make sure that all is OK? Thank you!

[pfitzseb]

Had a quick look through the various versions and everything looks fine to me.

[bkamins]

I also checked and all looks good. Thank you!