Skip to content

Commit

Permalink
Merge pull request #513 from typelevel/preview/ast
Browse files Browse the repository at this point in the history
preview server - render document AST under /ast path postfix
  • Loading branch information
jenshalm authored Sep 5, 2023
2 parents 88d8b96 + c0f03ec commit bece16d
Show file tree
Hide file tree
Showing 8 changed files with 196 additions and 14 deletions.
2 changes: 1 addition & 1 deletion build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ lazy val basicSettings = Seq(
(if (priorTo2_13(scalaVersion.value)) Seq("-Ypartial-unification") else Nil)
)

val mimaPreviousVersions = Set("0.19.0", "0.19.1", "0.19.2")
val mimaPreviousVersions = Set("0.19.0", "0.19.1", "0.19.2", "0.19.3")

val previousArtifacts = Seq(
mimaPreviousArtifacts := mimaPreviousVersions
Expand Down
6 changes: 5 additions & 1 deletion core/shared/src/main/scala/laika/render/ASTRenderer.scala
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,11 @@ object ASTRenderer extends ((TextFormatter, Element) => String) {
}

def attributes(attr: Iterator[Any], exclude: AnyRef = NoRef): String = {
def prep(value: Any) = value match { case opt: Options => options(opt); case other => other }
def prep(value: Any) = value match {
case opt: Options => options(opt)
case t: ResolvedInternalTarget => s"InternalTarget(${t.relativePath},${t.internalFormats})"
case other => other
}
val it = attr.asInstanceOf[Iterator[AnyRef]]
val res = it
.filter(_ ne exclude)
Expand Down
3 changes: 2 additions & 1 deletion docs/src/01-about-laika/01-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,8 @@ Content Organization
* Produce [Versioned Documentation] based on simple configuration steps and an integrated version switcher
dropdown in the default Helium theme.

* Use the integrated [Preview Server](../02-running-laika/01-sbt-plugin.md#using-the-preview-server) with live updates to preview your site while editing.
* Use the integrated [Preview Server](../02-running-laika/01-sbt-plugin.md#using-the-preview-server)
with live updates to preview your site while editing.


Library API
Expand Down
15 changes: 14 additions & 1 deletion docs/src/02-running-laika/01-sbt-plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,20 @@ If you are using an IDE with auto-save you might need to tweak its preferences
for seeing changes while editing the Markdown sources.
IntelliJ, for example, only auto-saves when you run or compile an application or when you switch to a
different application in the OS.
You can either use `cmd-S` to manually force saving or change the preferences to auto-save in fixed time intervals.
You can either use `cmd-S` to manually force saving or change the preferences to auto-save in fixed time intervals.


### Preview of the Document AST

Introduced in version 0.19.4 the preview server can now also render the document AST for any markup source document.
Simply append the `/ast` path element to your URL, e.g. `localhost:4242/my-docs/intro.md/ast`.
Note that this does not prevent you from using `/ast` as an actual path segment in your site,
the server will be able to distinguish those.

The AST shown is equivalent to the AST passed to the actual renderer after the final rewrite phase.
In case of writing custom render overrides it is the most accurate representation of the nodes you can match on.
When writing rewrite rules for earlier phases the actual nodes to match on might differ
(e.g. directives and links might still be unresolved).


Plugin Settings
Expand Down
13 changes: 13 additions & 0 deletions docs/src/02-running-laika/02-library-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -735,3 +735,16 @@ ServerBuilder[IO](parser, inputs)
By default, the port is 4242, the poll interval is 3 seconds, and EPUB or PDF downloads will not be included
in the generated site.


### Preview of the Document AST

Introduced in version 0.19.4 the preview server can now also render the document AST for any markup source document.
Simply append the `/ast` path element to your URL, e.g. `localhost:4242/my-docs/intro.md/ast`.
Note that this does not prevent you from using `/ast` as an actual path segment in your site,
the server will be able to distinguish those.

The AST shown is equivalent to the AST passed to the actual renderer after the final rewrite phase.
In case of writing custom render overrides it is the most accurate representation of the nodes you can match on.
When writing rewrite rules for earlier phases the actual nodes to match on might differ
(e.g. directives and links might still be unresolved).

137 changes: 137 additions & 0 deletions preview/src/main/scala/laika/preview/ASTPageTransformer.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
/*
* Copyright 2012-2023 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package laika.preview

import laika.api.Renderer
import laika.api.builder.OperationConfig
import laika.ast.{
Block,
BlockContainer,
BlockSequence,
CodeBlock,
Document,
DocumentTreeRoot,
RewritePhase,
RootElement,
Section,
Title
}
import laika.bundle.{ BundleOrigin, ExtensionBundle }
import laika.format.{ AST, HTML }
import laika.parse.{ Failure, Success }
import laika.parse.code.languages.LaikaASTSyntax
import laika.rewrite.OutputContext
import laika.rewrite.nav.PathTranslator

import scala.annotation.tailrec

private[preview] object ASTPageTransformer {

object ASTPathTranslator extends ExtensionBundle {

override val origin: BundleOrigin = BundleOrigin.Library
val description: String = "AST URL extension for preview server"
private val outputName = "ast"

override def extendPathTranslator
: PartialFunction[ExtensionBundle.PathTranslatorExtensionContext, PathTranslator] = {
case context =>
PathTranslator.postTranslate(context.baseTranslator) { path =>
if (path.suffix.contains("html")) {
val base = path.withoutFragment / outputName
path.fragment.fold(base)(base.withFragment)
}
else path
}
}

}

private val renderer = Renderer.of(AST).build

private val highlighter = LaikaASTSyntax.rootParser

private val messagePrefix = "Rendering of AST failed: "

private def callout(message: String): Block =
BlockSequence(messagePrefix + message).withStyles("callout", "error")

private def sequenceToASTBlock(bs: BlockContainer): Block = {
renderer.render(bs) match {
case Left(error) => callout(error.message)
case Right(ast) =>
highlighter.parse(ast) match {
case Success(spans, _) => CodeBlock("laika-ast", spans)
case f: Failure => callout(f.message)
}

}
}

private def splitAtFirstSection(blocks: Seq[Block]): (Seq[Block], Seq[Block]) =
blocks.span(b => !b.isInstanceOf[Section])

@tailrec
private def transformBlocks(blocks: Seq[Block], acc: Seq[Block] = Nil): Seq[Block] = {

val (start, end) = splitAtFirstSection(blocks)

// due to the nature of Laika's internal section builder start is expected to always be empty,
// but there is no need to fail on unexpected content
val noSection =
if (start.isEmpty) None
else Some(sequenceToASTBlock(BlockSequence(start)))
val (section, rest) =
if (end.isEmpty) (None, Nil)
else (Some(transformSection(end.head.asInstanceOf[Section])), end.tail)

val newAcc = acc ++ noSection.toSeq ++ section.toSeq
if (rest.isEmpty) newAcc
else transformBlocks(rest, newAcc)
}

private def transformSection(section: Section): Section = {
val (first, remaining) = splitAtFirstSection(section.content)

val firstBlock = sequenceToASTBlock(section.withContent(first))

section.withContent(firstBlock +: transformBlocks(remaining))
}

private def transformRoot(root: RootElement): RootElement = {

val (first, remaining) = splitAtFirstSection(root.content)

val title = first.collectFirst { case title: Title => title }
val firstBlock = sequenceToASTBlock(root.withContent(first))
val rest = firstBlock +: transformBlocks(remaining)

root.withContent(title.toSeq ++ rest)
}

private def transformDocument(doc: Document): Document =
doc.copy(content = transformRoot(doc.content))

def transform(tree: DocumentTreeRoot, config: OperationConfig): DocumentTreeRoot = {
val rules = config.rewriteRulesFor(tree, RewritePhase.Render(OutputContext(HTML)))
tree.rewrite(rules) match {
case Right(rewritten) => rewritten.modifyTree(_.mapDocuments(transformDocument))
case Left(error) => tree.mapDocuments(_.copy(content = RootElement(callout(error.message))))
}
}

}
26 changes: 16 additions & 10 deletions preview/src/main/scala/laika/preview/SiteTransformer.scala
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ import java.io.ByteArrayOutputStream
private[preview] class SiteTransformer[F[_]: Async](
val parser: TreeParser[F],
htmlRenderer: TreeRenderer[F],
astRenderer: TreeRenderer[F],
binaryRenderers: Seq[(BinaryTreeRenderer[F], String)],
inputs: InputTreeBuilder[F],
staticFiles: ResultMap[F],
Expand Down Expand Up @@ -76,8 +77,8 @@ private[preview] class SiteTransformer[F[_]: Async](
}
}

def transformHTML(tree: ParsedTree[F]): F[ResultMap[F]] = {
htmlRenderer
def transformHTML(tree: ParsedTree[F], renderer: TreeRenderer[F]): F[ResultMap[F]] = {
renderer
.from(tree)
.toOutput(StringTreeOutput)
.render
Expand All @@ -96,11 +97,15 @@ private[preview] class SiteTransformer[F[_]: Async](
}

val transform: F[SiteResults[F]] = for {
tree <- parse
rendered <- transformHTML(tree)
ebooks <- Async[F].fromEither(transformBinaries(tree).leftMap(ConfigException.apply))
tree <- parse
html <- transformHTML(tree, htmlRenderer)
ast <- transformHTML(
tree.modifyRoot(ASTPageTransformer.transform(_, parser.config)),
astRenderer
)
ebooks <- Async[F].fromEither(transformBinaries(tree).leftMap(ConfigException.apply))
} yield {
new SiteResults(staticFiles ++ rendered ++ ebooks)
new SiteResults(staticFiles ++ ast ++ html ++ ebooks)
}

}
Expand Down Expand Up @@ -164,16 +169,17 @@ private[preview] object SiteTransformer {
}

for {
p <- parser.map(adjustConfig)
html <- htmlRenderer(p.config, p.theme)
bin <- renderFormats.traverse(f =>
p <- parser.map(adjustConfig)
html <- htmlRenderer(p.config, p.theme)
ast <- htmlRenderer(p.config.withBundles(Seq(ASTPageTransformer.ASTPathTranslator)), p.theme)
bin <- renderFormats.traverse(f =>
binaryRenderer(f, p.config, p.theme).map((_, f.description.toLowerCase))
)
vFiles <- Resource.eval(StaticFileScanner.collectVersionedFiles(p.config))
apiFiles <- collectAPIFiles(p.config)
} yield {
val allInputs = inputs.merge(asInputTree(apiFiles))
new SiteTransformer[F](p, html, bin, allInputs, vFiles, artifactBasename)
new SiteTransformer[F](p, html, ast, bin, allInputs, vFiles, artifactBasename)
}

}
Expand Down
8 changes: 8 additions & 0 deletions preview/src/test/scala/laika/preview/PreviewRouteSpec.scala
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,14 @@ class PreviewRouteSpec extends CatsEffectSuite with InputBuilder {
run(uri"/dir", Status.Ok, Some(MediaType.text.html), stringBody("<p>foo</p>"))
}

test("serve the AST of a rendered document") {
val expected =
"""<pre><code class="nohighlight"><span class="type-name">RootElement</span><span> - </span><span class="keyword">Blocks</span><span>: </span><span class="number-literal">1</span><span>
|</span><span class="tag-punctuation">. </span><span class="type-name">Paragraph</span><span> - </span><span class="keyword">Spans</span><span>: </span><span class="number-literal">1</span><span>
|</span><span class="tag-punctuation">. . </span><span class="type-name">Text</span><span> - </span><span class="string-literal">&#39;foo&#39;</span></code></pre>""".stripMargin
run(uri"/doc.html/ast", Status.Ok, Some(MediaType.text.html), stringBody(expected))
}

test("serve a static document") {
run(uri"/dir/image.jpg", Status.Ok, Some(MediaType.image.jpeg), stringBody("img"))
}
Expand Down

0 comments on commit bece16d

Please sign in to comment.