Gradle Runner 2.0

[IgnatBeresnev]

Background

Dokka's current Gradle plugin has many problems:

• It is not idiomatic, not up to modern standard and does not follow best practices (see Improve compatibility with the Gradle API, and follow best practices #2700).
• It's very difficult to use and confusing to configure, especially for beginners (confirmed by our UX researches and user interviews)
• It has a lot of internal technical debt, as multiple generations of maintainers worked on it without passing down the knowledge, and it wasn't looked after much this past year.
• There are things that do not cause trouble per se, but that we'd want to change given the chance (for example, Gradle integration tests).

Trying to improve the current plugin and resolve problems gradually proved to be difficult as it's gets overwhelming quick. Moreover, with this many changes needed, it would lead to multiple releases with noticeably broken backward compatibility.

At this point it seems like the best course of action is to just rip the bandaid off, and to redesign and reimplement the Gradle plugin from scratch, taking into consideration all of the experience and knowledge we have now.

This issue exists to keep track of progress and to provide updates to the community.

High-level goals

• Redesign the configuration DSL:
  • Use the knowledge we got from conducting UX researches and user interviews.
  • Comb through existing issues with the configuration tag and take them into account
  • Make sure the DSL respects project isolation and is configuration cache compatible from the get-go
  • Address our internal wants and wishes
• Make sure the new plugin is idiomatic and follows best practices, both on the surface and under the hood
• Significantly improve or even re-write user-facing documentation.

We won't be able to implement all feature requests and fix all the bugs, but we'll make sure to lay a solid foundation to address them in the future.

High-level plans

We plan to start with designing the new configuration DSL, and it'll take some thinking time and rounds of internal and external review. It's important to get it (mostly) right the first time around, as we want it to be very stable for at least the first couple of major releases after it's out.

As for re-writing the plugin internals, a regular contributor @aSemy started working on it in January (#2839), which grew into a separate project called Dokkatoo (double pun intended). Dokkatoo has been adopted by a number of big projects (including Gradle itself), and has proved itself to be very idiomatic and modern. We hope Dokkatoo will be contributed upstream into Dokka, laying the very foundation for the Gradle plugin 2.0.

We also plan to test the new plugin on a wide range of projects and platforms, with various Kotlin and Gradle versions, and to provide migration guides -- all to ease the transition as much as possible.

More specific details and plans will follow.

---

Milestone: milestone/28

[CLOVIS-AI]

Hi, good luck with this endeavor!

Apart from what was already mentioned, my biggest pain point with Dokka is the high memory usage. We have had to increase the Gradle heap space on all projects which use Dokka.

We switched all our projects to Dokkatoo, mostly because of build cache and configuration cache support (which decrease memory usage by a lot, since Dokka is invoked a lot more rarely).

If that is of any help, we would be interested in using our projects to test future experimental versions of Dokka.

[hfhbd]

I want to further process the documentation so dokka should export the docs as json (whatever) file and then run the html file generator on the json file. This would also allow you to intercept the generation and add your own transformer.

One use-case: I want to dynamically generate a class diagram and add it to the documentation, so at the moment I need to call the Gradle task by myself, but it should automatically wired using inputs/outputs.

[jeffdgr8]

While not directly related to the Gradle plugin, it could be relevant when considering a new configuration API or intermediate output format. One thing I'd like to see Dokka support in the future is a redesigned platform / source set selector. See my comment here for how I envision this working ideally.

[adam-enko]

"it just works" configuration

I want to be able to apply Dokka to a project and it just works, with sensible defaults requiring no configuration.

Performance

Currently Dokka is still quite slow/heavy. This makes it difficult to recommend using Dokka to generate a Javadoc JAR, because some projects publish to Maven Local for testing, and Dokka makes this step too slow.

Split out Dokka Configuration

It would be helpful to split out the Dokka Configuration classes into a separate dependency.

Currently Dokkatoo requires DGP as a compileOnly dependency and only uses them in the isolated Gradle Worker. Using an implementation dependency doesn't work, because it intrinsically links Dokkatoo to a specific Dokka version (which at the moment is configurable - Dokkatoo can run almost any Dokka version) and it would pollute the Gradle buildscript classpath with Dokka classes.

Because Dokkatoo uses a compileOnly dependency on Dokka this increases the risk of accidentally using a 'missing' class during the Dokkatoo Gradle plugin configuration (which can be caught by testing, but I'd prefer not to).

Machine-readable output

+1 for dumping machine-readable output #621

This could also integrate better with Knit https://github.com/Kotlin/kotlinx-knit

Split up Dokka Generator steps

It suspect would be helpful to split up the Dokka Generator into more discrete steps. These steps could then be done with separate Gradle tasks, which would aid with caching/performance/parallelization.

For example, split up the sources/classes/dependencies analysis from markup generation, and store the analysed files in a directory, to be loaded later by the markup generator.

Dokka subproject Module vs aggregating project Publication configuration options

Clarify/simplify some of the Dokka format configuration options. For example, which HTML output options are applicable to a subproject (a Dokka Module) vs an aggregated publication? Ideally the buildscript configuration DSL should only show options that will do something (currently the behaviour can be unclear adamko-dev/dokkatoo#111)

More & automatic external source links

Automatic & more comprehensive external source links. Currently Dokka has some automatic detection, e.g. if it's an Android project then enable the Android external source link. But it would be great if Dokka checked all of the dependencies for a project, and had built-in configuration for enabling external source links for all of the Kotlin libs (like Serialization, Coroutines, etc).

[kenyee]

I integrated this with a multi-module project w/o problems, but then tried integrating with a much more complex Android multi-module multi-flavor application and it is hitting all sorts of interesting issues with compiling a releaseInt version which is missing a resource (but we never build it).
Would be nice to be able to tell it to run against a specific flavor when generating documetation. The current Dokka plugin appears to be trying to build all flavors...

EDIT: been meaning to update this. I fixed the issues with the extra flavor and also tried excluding it from all sourcesets....but Dokka at the root module still tries compiling it. I'll bet that's why Dokka is considered slow...