Scaladoc markdown preprocessor

[BarkingBad]

PoC of preprocessing markdown to get advantage of scaladoc linking and snippet compiler but preserve the flexibility to reuse docs in Jekyll or another templating engine.

[julienrf]

I think we should not try to support both rendering the dotty website with scaladoc, and processing the md files to be rendered by Jekyll (on docs.scala-lang.org).

Instead, we should just focus on doing one thing: processing the md files that will be then rendered by Jekyll only. This will save us from introducing the postprocessing-layout feature.

Reviewing the changes made to the md files is hard, I suggest the following process to be sure they are correct:

1. update the markdown files in the docs.scala-lang repo to contain the fixes that have been applied in the dotty repo (the files in docs.scala-lang have been taken from commit 270bd8b, so we need to see what changes have been made to the docs/ files since that commit and to backport them to the docs.scala-lang repo).
2. copy the files from docs.scala-lang to the dotty repo, without changing anything to them

[BarkingBad]

Thank you Julien for your patience to review this big chunk of code :D I will address all of your concerns in the threads.

[BarkingBad]

I think we should not try to support both rendering the dotty website with scaladoc, and processing the md files to be rendered by Jekyll (on docs.scala-lang.org).

Instead, we should just focus on doing one thing: processing the md files that will be then rendered by Jekyll only. This will save us from introducing the postprocessing-layout feature.

I agree

Reviewing the changes made to the md files is hard, I suggest the following process to be sure they are correct:

1. update the markdown files in the docs.scala-lang repo to contain the fixes that have been applied in the dotty repo (the files in docs.scala-lang have been taken from commit 270bd8b, so we need to see what changes have been made to the docs/ files since that commit and to backport them to the docs.scala-lang repo).
2. copy the files from docs.scala-lang to the dotty repo, without changing anything to them

I was thinking about adding one more feature to the markdown front-matters, like file source. We can easily generate it because we know the source-links from scaladoc. These links would be used as a button target for Edit on github. From the files perspective they won't change, as this entry in the front-matter would be autogenerated for dotty markdown files by scaladoc. I wonder, whether we could add something similar for the markdowns hosted at docs.scala-lang to stay concise and help users introduce changes faster (no overhead of finding correct file and forking the repo)?

[julienrf]

I wonder, whether we could add something similar for the markdowns hosted at docs.scala-lang to stay concise and help users introduce changes faster (no overhead of finding correct file and forking the repo)?

Yes, that’s a good point. We should find a way to make this work.

[BarkingBad]

I'm thinking about opening 2 smaller PRs (one with css-fixes and one with markdown update) then would rebase this PR on top of the changes to reduce LOC of this PR and ease the whole process

[BarkingBad]

CSSes PR: #13335

[BarkingBad]

Markdown backports to the docs.scala-lang repo: scala/docs.scala-lang#2160

[BarkingBad]

Moved markdowns back to dotty: #13350

[julienrf]

I did another pass where I added a few more comments. Please also note that former comments have not been addressed yet.

[julienrf]

Should we use an enumeration type?

[BarkingBad]

It is already ChoiceSetting. It is hard to extends Setting to be compatible with my custom type as I would have to modify doSet method from dotty.tools.dotc.config.Settings for my new enumeration ClassTag

[julienrf]

OK. This is out of the scope of this PR, but we might need to allocate some resources at some point to clean the codebase. It looks very fragile now. If we add another format to the list we will have to manually look for all the places we pattern match on a format to check that we correctly handle it. If it was an enumeration the compiler would just give us exhaustivity warnings.

[julienrf]

I think it is urgent to move forward and clean the documentation authoring workflow. As long as we duplicate the markdown sources here and on docs.scala-lang, divergence will happen (see where we are in just a couple of months).

Hopefully, this work will help us maintain the documentation source files so that the content at docs.scala-lang.org will be easier to keep up to date, even for the compiler team.

By the way, I see that this PR changes some things on the source files that will be published to dotty.epfl.ch. I am not able to review those changes. I leave them up to you.