WIP: merge Scala3doc #10081

abgruszecki · 2020-10-25T19:37:21Z

No description provided.

romanowski

Changes in sbt plugin and build definition looks ok however I think it should be rebased on top of #10073 since there we've maintained a history of scala3doc and use more recent version of code, without need for small Kotlin project build with gradle as well changes in GH actions

romanowski · 2020-10-25T22:01:49Z

scala3doc/src/main/scala/dotty/dokka/DottyDokkaConfig.scala

@@ -19,7 +19,8 @@ case class DottyDokkaConfig(docConfiguration: DocConfiguration) extends DokkaCon

  private object OurConfig extends DokkaConfiguration.PluginConfiguration:
    override def getFqPluginName = "ExternalDocsTooKey"
-    override def getSerializationFormat: DokkaConfiguration$SerializationFormat = DokkaConfiguration$SerializationFormat.JSON
+    override def getSerializationFormat: DokkaConfiguration$SerializationFormat =
+      DokkaConfiguration$SerializationFormat.JSON.asInstanceOf[DokkaConfiguration$SerializationFormat]


Why do need to cast here? Some change in dotty type interference?

This became necessary when I bumped up Dotty compiler version to latest. I'm not sure if this is related to type inference, my first guess is that Kotlin was emitting some strange JVM bytecode and we changed how we translated that to Scala types. Another of the litany of things we should open an issue for in the Dotty repo.

ok, I will take a look

romanowski · 2020-10-25T22:08:26Z

.github/workflows/ci.yaml

@@ -133,6 +133,37 @@ jobs:
        run: sbt ";sjsJUnitTests/test ;sjsCompilerTests/test"
        shell: cmd

+  build_docs:


I was under impression that we want to have a dedicated fast running CI for scala3docs and having it as a step in main build will not be possible.

I thought about it and I think something like what we have here is the cleanest way. build_docs actually runs concurrently with other "jobs", so it finishes very quickly (in 4 minutes). We might still make it a separate workflow so that we can disable the primary workflow when only Scala3doc has changed.

romanowski · 2020-10-25T22:10:53Z

project/Build.scala

@@ -1440,6 +1460,130 @@ object Build {
      settings(commonBenchmarkSettings).
      enablePlugins(JmhPlugin)

+    def asScala3doc: Project = {


IMHO we should keep most of the settings in scala3doc/build.sbt, since if we move them to top level all changes here will requrie to run full community build (and we may not want that).

I don't think that's doable. The primary reason behind merging the two repos together is that we want their builds to be unified, so that Scala3doc can be easily compiled with latest compiler (which is practically necessary for quickly fixing compiler issues to help Scala3doc) and so that compiler sub-projects and community build projects can be easily documented with latest Scala3doc. I can't see any good way of doing that other than having Scala3doc be a sub-project here.

romanowski · 2020-10-25T22:11:41Z

sbt-dotty/src/dotty/tools/sbtplugin/DottyPlugin.scala

+      // Configuration for the doctool
+      resolvers ++= Seq(
+        Resolver.jcenterRepo,
+        Resolver.bintrayRepo("kotlin", "kotlin-dev"),


I think we should add those only if useScala3doc := true

I'll ask someone that understands how SBT works if that will work.

ok. I will try to make it work

What I was worried here about is when exactly the value of useScala3doc is checked. It has to be checked after plugin users could have already set it, otherwise it will always be false. Also whatever we do here should work correctly with users adding their own resolvers.

The following will do what you want:

resolvers ++= { if (useScala3doc.value) { Seq(...) } else Nil }

@sjrd Excellent, thanks for checking up on this!

Will include that in #10092

nicolasstucki

Contents of docs where copied to scala3doc/dotty-docs/docs. This makes it really hard to see any diffs. It would much simpler if we delete docs` to make git handle these as file moves.

nicolasstucki · 2020-10-27T07:33:42Z

scala3doc/README.md

+Scala3doc (name subject to change) is the documentation tool for
+[Dotty](https://github.com/lampepfl/dotty), which is scheduled to become
+Scala 3. It's based on [Dokka](https://github.com/Kotlin/dokka), the
+documentation tool for Kotlin. It uses Tasty-Reflect to access definitions,


Suggested change

documentation tool for Kotlin. It uses Tasty-Reflect to access definitions,

documentation tool for Kotlin. It uses the TastyInspector to access definitions,

nicolasstucki · 2020-10-27T07:40:11Z

scala3doc/dotty-docs/docs/blog/_posts/2017-09-07-third-dotty-milestone-release.md

@@ -0,0 +1,151 @@
+---
+layout: blog-page


Can we deleter the blog post file original blogpost file? This way git will consider it a move and we won't lose the history. Same for all other blog posts.

We can't, these files are actually slightly different. We will do that when replacing Dottydoc as the tool for rendering the Dotty blog, which should be done soon-ish.

nicolasstucki · 2020-10-27T07:41:54Z

scala3doc/dotty-docs/docs/blog/index.html

@@ -0,0 +1,32 @@
+---
+layout: main


Can we delete docs/blog/index.html?

Same as here: #10081 (comment)

nicolasstucki · 2020-10-27T08:00:54Z

scala3doc/src/main/scala/dotty/dokka/tasty/BasicSupport.scala

+  self: TastyParser =>
+  import reflect._
+
+  object SymOps extends SymOps[reflect.type](reflect)


It looks like this is encoding what is described in #9530. We should improve extensions to support this use case.

Yeah, I was not happy when I needed to introduce Cake Patter that cause me so much pain in Scala 2

nicolasstucki · 2020-10-27T08:05:16Z

scala3doc/src/main/scala/dotty/dokka/tasty/TastyParser.scala

+    )
+
+/** Parses a single Tasty compilation unit. */
+case class TastyParser(reflect: Reflection, inspector: DokkaBaseTastyInspector, config: DottyDokkaConfig)


Suggested change

case class TastyParser(reflect: Reflection, inspector: DokkaBaseTastyInspector, config: DottyDokkaConfig)

case class TastyParser(qctx: QuoteContext, inspector: DokkaBaseTastyInspector, config: DottyDokkaConfig)

We should always use the full path for reflection. Using Reflection directly may not work in the future.

Adressed in 78b7d49 in #10092

nicolasstucki · 2020-10-27T08:06:47Z

scala3doc/src/main/scala/dotty/dokka/tasty/TypesSupport.scala

+        case AppliedType(tpe, _) => inner(tpe)
+        case tp @ TermRef(qual, typeName) =>
+          qual match
+            case _: Type | _: NoPrefix => Some(tp.termSymbol)


Suggested change

case _: Type | _: NoPrefix => Some(tp.termSymbol)

case _: TypeRepr | _: NoPrefix => Some(tp.termSymbol)

Need to rebase and rename reflection Type to TypeRepr

Done in dd285bb in #10092

nicolasstucki · 2020-10-27T08:20:33Z

scala3doc/src/main/scala/dotty/dokka/tasty/TastyParser.scala

+    result()
+  }
+
+  class InspectorDriver extends Driver:


We will need to find a way to share this code. That can be done later.

nicolasstucki · 2020-10-27T08:23:11Z

scala3doc/src/main/scala/dotty/dokka/tasty/TastyParser.scala

+    override protected def newCompiler(implicit ctx: Context): Compiler = new TastyFromClass
+
+    def run(filesToDocument: List[String])(implicit ctx: Context): Unit =
+      doCompile(newCompiler, filesToDocument)


What can filesToDocument contain? .tasty and .jar file? If so it is ok.

List of .tasty files. We do support jars however we extract them to temporary directory and use extracted .tasy files. I think we will change that once #10061 is ready

nicolasstucki · 2020-10-27T08:25:09Z

scala3doc/src/main/scala/dotty/dokka/tasty/TastyParser.scala

+  private val topLevels = Seq.newBuilder[Documentable]
+
+  def processCompilationUnit(using ctx: quoted.QuoteContext)(root: ctx.reflect.Tree): Unit =
+    val parser = new TastyParser(ctx.reflect, this, config)


Should pass QuoteContext rather than Reflection

Adressed in 78b7d49 in #10092

nicolasstucki · 2020-10-27T08:26:52Z

scala3doc/src/main/scala/dotty/dokka/tasty/TastyParser.scala

+
+  private val topLevels = Seq.newBuilder[Documentable]
+
+  def processCompilationUnit(using ctx: quoted.QuoteContext)(root: ctx.reflect.Tree): Unit =


Suggested change

def processCompilationUnit(using ctx: quoted.QuoteContext)(root: ctx.reflect.Tree): Unit =

def processCompilationUnit(using QuoteContext)(root: qctx.reflect.Tree): Unit =

qctx acts like the ctx in the compiler. You have to import scala.quote._ first.

Adressed in 78b7d49 in #10092

bishabosha · 2020-10-27T11:05:36Z

how does this relate to #10092?

abgruszecki · 2020-10-27T12:38:55Z

@bishabosha that one is this one rebased on top of Scala3doc's migrated Git history + some additional tweaks to the configuration + changes to the code which were already requested (such as removing Scala2 dependencies).

bishabosha · 2020-10-27T12:41:34Z

@abgruszecki thank you for explaining, does this one need to remain open?

romanowski · 2020-10-27T18:08:56Z

@nicolasstucki

Contents of docs where copied to scala3doc/dotty-docs/docs. This makes it really hard to see any diffs. It would much simpler if we delete docsto make git handle these as file moves. This is our next step after this PR is merged. We will removescala3doc/dotty-docs` as it is now, then either change scala3doc to support existing documentation (this means a bit of hackery) or copy documentation again and have a single commit with changes.

abgruszecki · 2020-10-27T18:10:07Z

Closing in favour of #10092.

abgruszecki added 5 commits October 25, 2020 17:49

Copy scala3doc code

6780498

Scala3doc: update Dotty dependency to latest snapshot

8b8dc6d

Scala3doc: comment out code causing an infinite loop

35d8c54

Scala3doc: fix all warnings

2804ce0

Clean up and split up Scala3doc

385182a

abgruszecki mentioned this pull request Oct 25, 2020

Migrate scala3doc codebase into dotty #10073

Closed

abgruszecki force-pushed the wip/doc-repo-merge branch from 7c832dd to d27609f Compare October 25, 2020 19:40

abgruszecki added 3 commits October 25, 2020 20:47

Configure SBT build

36442ff

Add testing the doctool to community build

e6790f4

Add generating documentation to the CI

e9871ac

abgruszecki force-pushed the wip/doc-repo-merge branch from d27609f to e9871ac Compare October 25, 2020 19:47

romanowski requested changes Oct 25, 2020

View reviewed changes

abgruszecki mentioned this pull request Oct 26, 2020

Add Scala3doc as the optional doctool #10090

Closed

smarter added this to the 3.0.0-M1 milestone Oct 26, 2020

nicolasstucki reviewed Oct 27, 2020

View reviewed changes

nicolasstucki suggested changes Oct 27, 2020

View reviewed changes

tgodzik mentioned this pull request Oct 27, 2020

Disallow given as <ident> syntax scalameta/scalameta#2153

Merged

abgruszecki closed this Oct 27, 2020

	documentation tool for Kotlin. It uses Tasty-Reflect to access definitions,
	documentation tool for Kotlin. It uses the TastyInspector to access definitions,

	case class TastyParser(reflect: Reflection, inspector: DokkaBaseTastyInspector, config: DottyDokkaConfig)
	case class TastyParser(qctx: QuoteContext, inspector: DokkaBaseTastyInspector, config: DottyDokkaConfig)

	case _: Type \| _: NoPrefix => Some(tp.termSymbol)
	case _: TypeRepr \| _: NoPrefix => Some(tp.termSymbol)


		private val topLevels = Seq.newBuilder[Documentable]

		def processCompilationUnit(using ctx: quoted.QuoteContext)(root: ctx.reflect.Tree): Unit =

	def processCompilationUnit(using ctx: quoted.QuoteContext)(root: ctx.reflect.Tree): Unit =
	def processCompilationUnit(using QuoteContext)(root: qctx.reflect.Tree): Unit =

WIP: merge Scala3doc #10081

WIP: merge Scala3doc #10081

Conversation

abgruszecki commented Oct 25, 2020

romanowski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolasstucki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bishabosha commented Oct 27, 2020

abgruszecki commented Oct 27, 2020

bishabosha commented Oct 27, 2020

romanowski commented Oct 27, 2020

abgruszecki commented Oct 27, 2020