Servedocs improvement + literate jl docs

docs/make.jl

@@ -14,7 +14,8 @@ makedocs(
         "Home" => "index.md",
         "Manual" => [
             "Functionalities" => "man/functionalities.md",
-            "Extending LiveServer" => "man/extending_ls.md"
+            "Extending LiveServer" => "man/extending_ls.md",
+            "LiveServer + Literate" => "man/ls+lit.md"
             ],
         "Library" => [
             "Public"    => "lib/public.md",

docs/src/index.md

@@ -49,6 +49,8 @@ julia> servedocs()
 
 Open a browser and go to `http://localhost:8000/` to see your docs being rendered; try modifying files (e.g. `docs/index.md`) and watch the changes being rendered in the browser.
 
+You can also use LiveServer with both Documenter and [Literate.jl](https://github.com/fredrikekre/Literate.jl).
+This is explained [here](man/ls+lit.md).
 
 ## How it works
 

docs/src/lib/internals.md

@@ -70,7 +70,7 @@ LiveServer.update_and_close_viewers!
 #### Helper functions associated with `servedocs`
 
 ```@docs
-LiveServer.servedocs_callback
+LiveServer.servedocs_callback!
 LiveServer.scan_docs!
 ```
 

docs/src/man/ls+lit.md

@@ -0,0 +1,170 @@
+# LiveServer + Literate
+
+(_Thanks to [Fredrik Ekre](https://github.com/fredrikekre) and [Benoit Pasquier](https://github.com/briochemc) for their input; a lot of this section is drawn from an early prototype suggested by Fredrik._)
+
+You've likely already seen how LiveServer could be used along with Documenter to have live updating documentation (see [`servedocs`](/man/functionalities/#servedocs-1) if not).
+
+It is also easy to use LiveServer with both Documenter and [Literate.jl](https://github.com/fredrikekre/Literate.jl), a package for literate programming written by Fredrik Ekre that can convert julia script files into markdown.
+This can be particularly convenient for documentation pages with a lot of code examples.
+
+Only two steps are required to have this working (assuming you have already added Literate to your environment):
+
+1. pick a folder structure
+1. modify the `docs/make.jl` file to contain a line calling Literate
+
+### Folder structure
+
+There are effectively two recommended ways, pick whichever one you prefer.
+In the first case, the script files `.jl` to be compiled by Literate are at the _same location_ as the output file so that you would have:
+
+```
+docs
+└── src
+    ├── index.jl
+    └── index.md
+```
+
+if you're happy with this, then you can jump to the [next step](#Modifying-the-make-file-1) to change the make file.
+
+However you may not be happy with this, in particular if you have lots of such files and a mix of files which are generated by `Literate` and some which aren't, then typically you might prefer to keep all scripts in a separate folder.
+You would just have to make sure that the output is properly redirected to `docs/src`.
+Your folder structure would then look something like:
+
+```
+docs
+├── lit
+│   └── index.jl
+└── src
+    └── index.md
+```
+
+The only thing you have to do in this case is to specify to `servedocs` where the "literate folder" is; this is a keyword argument and for the example above we would have:
+
+```julia
+servedocs(literate=joinpath("docs", "lit"))
+```
+
+### Modifying the make file
+
+The only thing you have to do here is add a few lines to specify which files should be compiled by `Literate`.
+Assuming you have taken the second path in the situation above, your `make.jl` file should look like:
+
+```julia
+using Documenter, Literate
+
+src = joinpath(@__DIR__, "src")
+lit = joinpath(@__DIR__, "lit")
+
+for (root, _, files) ∈ walkdir(lit), file ∈ files
+    splitext(file)[2] == ".jl" || continue
+    ipath = joinpath(root, file)
+    opath = splitdir(replace(ipath, lit=>src))[1]
+    Literate.markdown(ipath, opath)
+end
+
+makedocs(
+    sitename = "Test",
+    modules = [Test],
+    pages = ["Home" => "index.md"]
+    )
+```
+
+If you were happy with the `.jl` and `.md` files being in the same location, simply replace the `lit = ` line by
+
+```julia
+lit = src
+```
+
+What the for loop does is simple: it loops over the files in the folder where it's likely to encounter `.jl` files and for those it encounters:
+
+1. it retrieves the path to the file (`ipath`)
+1. it constructs the output path in `docs/src` (`opath`)
+1. it compiles the file `ipath` and saves the output at `opath`
+
+## Complete example
+
+Here's a step-by-step example to get started which should help put all the pieces together.
+
+Let's start by creating a dummy repo
+
+```julia-repl
+pkg> generate testlit
+julia> cd("testlit")
+pkg> activate testlit
+pkg> add Documenter Literate LiveServer
+pkg> dev .
+```
+
+add a `docs/` folder with the appropriate structure so that the `testlit` folder ends up like
+
+```
+.
+├── Manifest.toml
+├── Project.toml
+├── docs
+│   ├── literate
+│   │   └── man
+│   │       └── pg1.jl
+│   ├── make.jl
+│   └── src
+│       ├── index.md
+│       └── man
+└── src
+    └── testlit.jl
+```
+
+where the file `pg1.jl` contains
+
+```julia
+# # Test literate
+
+# We can include some code like so:
+
+f(x) = x^5
+f(5)
+```
+
+the file `index.md` contains
+
+```
+# Test
+
+A link to the [other page](/man/pg1.md)
+```
+
+and the file `make.jl` contains
+
+```julia
+using Documenter, Literate
+
+src = joinpath(@__DIR__, "src")
+lit = joinpath(@__DIR__, "literate")
+
+for (root, _, files) ∈ walkdir(lit), file ∈ files
+    splitext(file)[2] == ".jl" || continue
+    ipath = joinpath(root, file)
+    opath = splitdir(replace(ipath, lit=>src))[1]
+    Literate.markdown(ipath, opath)
+end
+
+makedocs(
+    sitename = "testlit",
+    modules = [testlit],
+    pages = ["Home" => "index.md",
+             "Other page" => "man/pg1.md"]
+    )
+```
+
+Now `cd("testlit/")` and do
+
+```julia-repl
+julia> servedocs(literate=joinpath("docs", "literate"))
+```
+
+if you navigate to `localhost:8000` you should end up with
+
+![](/assets/testlit.png)
+
+if you modify `testlit/docs/literate/man/pg1.jl` for instance writing `f(4)` it will be applied directly:
+
+![](/assets/testlit2.png)

src/utils.jl

@@ -1,5 +1,5 @@
 """
-    servedocs_callback(filepath, watchedfiles, path2makejl)
+    servedocs_callback!(docwatcher, filepath, path2makejl, literate)
 
 Custom callback used in [`servedocs`](@ref) triggered when the file corresponding to `filepath`
 is changed. If that file is `docs/make.jl`, the callback will check whether any new files have
@@ -9,82 +9,119 @@ Otherwise, if the modified file is in `docs/src` or is `docs/make.jl`, a pass of
 triggered to regenerate the documents, subsequently the LiveServer will render the produced pages
 in `docs/build`.
 """
-function servedocs_callback(fp::AbstractString, vwf::Vector{WatchedFile}, makejl::AbstractString)
-    ismakejl = (fp == makejl)
-    # if the file that was changed is the `make.jl` file,
-    # assume that maybe new files are referenced and so refresh the
-    # vector of watched files as a result.
-    if ismakejl
-        watchedpaths = (wf.path for wf ∈ vwf)
-        for (root, _, files) ∈ walkdir(joinpath("docs", "src")), file ∈ files
-            fpath = joinpath(root, file)
-            fpath ∈ watchedpaths || push!(vwf, WatchedFile(fpath))
-        end
-        # check if any file that was watched has died
-        deadfiles = Int[]
-        for (i, wf) ∈ enumerate(vwf)
-            isfile(wf.path) || push!(deadfiles, i)
-        end
-        deleteat!(vwf, deadfiles)
+function servedocs_callback!(dw::SimpleWatcher, fp::AbstractString, makejl::AbstractString,
+                             literate::String="")
+    # if the file that was changed is the `make.jl` file, assume that maybe new files are # referenced and so refresh the vector of watched files as a result.
+    if fp == makejl
+        # it's easier to start from scratch (takes negligible time)
+        empty!(dw.watchedfiles)
+        scan_docs!(dw, literate)
     end
-    # only trigger for changes appearing in `docs/src` otherwise a loop gets triggered
-    # changes from docs/src create change in docs/build which trigger a pass which
-    # regenerates files in docs/build etc...
-    if ismakejl || occursin(joinpath("docs", "src"), fp)
+    if splitext(fp)[2] ∈ (".md", ".jl")
         Main.include(makejl)
         file_changed_callback(fp)
     end
     return nothing
 end
 
+
 """
-    scan_docs!(dw::SimpleWatcher)
+    scan_docs!(dw::SimpleWatcher, literate="")
 
 Scans the `docs/` folder in order to recover the path to all files that have to be watched and add
-those files to `dw.watchedfiles`. The function returns the path to `docs/make.jl`.
+those files to `dw.watchedfiles`. The function returns the path to `docs/make.jl`. A list of
+folders and file paths can also be given for files that should be watched in addition to the
+content of `docs/src`.
 """
-function scan_docs!(dw::SimpleWatcher)
+function scan_docs!(dw::SimpleWatcher, literate::String="")
     src = joinpath("docs", "src")
     if !(isdir("docs") && isdir(src))
         @error "I didn't find a docs/ or docs/src/ folder."
     end
     makejl = joinpath("docs", "make.jl")
     push!(dw.watchedfiles, WatchedFile(makejl))
     if isdir("docs")
-         # add all files in `docs/src` to watched files
-         for (root, _, files) ∈ walkdir(joinpath("docs", "src")), file ∈ files
-             push!(dw.watchedfiles, WatchedFile(joinpath(root, file)))
-         end
+        # add all files in `docs/src` to watched files
+        for (root, _, files) ∈ walkdir(joinpath("docs", "src")), file ∈ files
+            push!(dw.watchedfiles, WatchedFile(joinpath(root, file)))
+        end
+    end
+    if !isempty(literate)
+        isdir(literate) || @error "I didn't find the provided literate folder $literate."
+        for (root, _, files) ∈ walkdir(literate), file ∈ files
+            push!(dw.watchedfiles, WatchedFile(joinpath(root, file)))
+        end
     end
+
+    # When using literate.jl, we should only watch the source file otherwise we would double
+    # trigger: first when the script.jl is modified then again when the script.md is created
+    # which would cause an infinite loop if both `script.jl` and `script.md` are watched.
+    # So here we remove from the watchlist all files.md that have a files.jl with the same path.
+    remove = Int[]
+    if isempty(literate)
+        # assumption is that the scripts are in `docs/src/...` and that the generated markdown
+        # goes in exactly the same spot so for instance:
+        # docs
+        # └── src
+        #     ├── index.jl
+        #     └── index.md
+        for wf ∈ dw.watchedfiles
+            spath = splitext(wf.path)
+            spath[2] == ".jl" || continue
+            k = findfirst(e -> splitext(e.path) == (spath[1], ".md"), dw.watchedfiles)
+            k === nothing || push!(remove, k)
+        end
+    else
+        # assumption is that the scripts are in `literate/` and that the generated markdown goes
+        # in `docs/src` with the same relative paths so for instance:
+        # docs
+        # ├── lit
+        # │   └── index.jl
+        # └── src
+        #     └── index.md
+        for (root, _, files) ∈ walkdir(literate), file ∈ files
+            spath = splitext(joinpath(root, file))
+            spath[2] == ".jl" || continue
+            path = replace(spath[1], Regex("^$literate") => joinpath("docs", "src"))
+            k = findfirst(e -> splitext(e.path) == (path, ".md"), dw.watchedfiles)
+            k === nothing || push!(remove, k)
+        end
+    end
+    deleteat!(dw.watchedfiles, remove)
     return makejl
 end
 
+
 """
-    servedocs(; verbose=false)
+    servedocs(; verbose=false, literate="")
 
 Can be used when developing a package to run the `docs/make.jl` file from Documenter.jl and
 then serve the `docs/build` folder with LiveServer.jl. This function assumes you are in the
 directory `[MyPackage].jl` with a subfolder `docs`.
 
-* `verbose` is a boolean switch to make the server print information about file changes and connections.
+* `verbose` is a boolean switch to make the server print information about file changes and
+connections.
+* `literate` is the path to the folder containing the literate scripts, if left empty, it will be
+assumed that they are in `docs/src`.
 """
-function servedocs(; verbose::Bool=false)
+function servedocs(; verbose::Bool=false, literate::String="")
     # Custom file watcher: it's the standard `SimpleWatcher` but with a custom callback.
     docwatcher = SimpleWatcher()
-    set_callback!(docwatcher, fp->servedocs_callback(fp, docwatcher.watchedfiles, makejl))
+    set_callback!(docwatcher, fp->servedocs_callback!(docwatcher, fp, makejl, literate))
 
-    makejl = scan_docs!(docwatcher)
+    # Retrieve files to watch
+    makejl = scan_docs!(docwatcher, literate)
 
-    # trigger a first pass of Documenter
+    # trigger a first pass of Documenter (& possibly Literate)
     Main.include(makejl)
 
     # note the `docs/build` exists here given that if we're here it means the documenter
     # pass did not error and therefore that a docs/build has been generated.
     serve(docwatcher, dir=joinpath("docs", "build"), verbose=verbose)
-
     return nothing
 end
 
+
 #
 # Miscellaneous utils
 #
@@ -96,6 +133,7 @@ Set the verbosity of LiveServer to either `true` (showing messages upon events)
 """
 setverbose(b::Bool) = (VERBOSE.x = b)
 
+
 """
     example()