IJulia notebooks for documentation?

[StefanKarpinski]

I'm really keen on the idea of writing Julia docs using IJulia notebooks. Notebooks support Markdown text formatting with embedded math equations using MathJax, and of course, if you run a notebook locally, you can execute all the code yourself, which I think will help people learn more effectively and making it easy to step through examples will help keep the examples up-to-date and catch cases where code changes breaks the examples more quickly. I've also found that IJulia notebooks are a great way to write documentation in the first place.

However, it would be nice if the source format, rather than the not-very-human-readable .ipynb JSON format, were something more like Markdown with embedded code examples (and output) that could be transformed – via code execution – to notebook form. This would be a bi-directional transformation:

• markdown w/ code and output + IJulia kernel ⟹ IJulia notebook
• IJulia notebook ⟹ markdown w/ code and output

The first transformation occurs via code execution; the second by just outputting the content of the notebook to markdown format, with input and output cells included as embedded code. Or maybe this process is too complicated and .ipynb should just be the source format.

So what are the features that we need from IPython notebook to support this? Obviously, we need the ability to link from one notebook to another. What else?

[StefanKarpinski]

Relevant IPython issues: ipython/ipython#3806, ipython/ipython#977.

cc: @Carreau, @fperez.

[Carreau]

One of the problem you will have if how to store many output in markdown (json store'em'all), so you will almost never be able to get a round trip .ipynb->md->.ipynb.

We have some kind of importer that will split a python source marked with comment into cells, but it will give you only something partial. that you will probably have to tweak most of the time.

One could use rst with specific sphinx directive to generate a notebook.

.ipynb -> md already kind of exist in nbconvert. It is not perfect as usually a human written document will not have html in it, but the automatic converter have issues to deal with that.

The converter does not add hints for back-conversion to ipynb, it shouldn't be too hard to do though.

[stevengj]

If there is graphical output (e.g. a plot), then markdown format is not so convenient... I'm not sold on the need for this, as opposed to just using notebook files directly (+ generated HTML/nbviewer output for web access).

[Carreau]

This repo did something similar (IScala Notebook) with a script to import its .xipynb.

[stevengj]

Here is some info on linking between notebooks.

[StefanKarpinski]

That seems to be a pretty major roadblock, unfortunately.

[Carreau]

The SO answer is wrong. There is a redirect handler. If you link to ./this-notebook.ipynb#header anchor it will redirect you to the right notebook on click. It also works on nbviewer.

We even have a PR that allow that across folder in filesysytem and get rid of uuid.

Envoyé de mon iPhone

Le 20 oct. 2013 à 01:18, Stefan Karpinski notifications@github.com a écrit :

That seems to be a pretty major roadblock, unfortunately.

—
Reply to this email directly or view it on GitHub.

[JeffBezanson]

This would be great for the manual, but the help system is also very important. Am I correct that this would not really address the help system?

[johnmyleswhite]

I was thinking the same thing: IJulia seems great for making what R calls vignettes (free-form text manuals), but less useful for a help system where you want structured data about functions.

[simonbyrne]

Perhaps something analogous to the Mathematica help system: for those who haven't used it, if you look up help from within Mathematica itself, you get something that looks similar to a HTML help page, however you can actually edit and re-run the code parts to see how things work (from what I can tell, it seems to run in its own kernel which is cleared when you leave the page).

A possible vision:

1. Help files are stored in source in some sort of markdown + code for easy storage and editing.
2. During the build process, these are run and stored as ijulia notebooks (this would also function as a type of unit test), and would generate plots etc.
   • Static text-based and static html-based versions can also be generated from this.
3. If I call help from:
   a. an IJulia console, it opens the IJulia version, while also opening a temporary kernel (or this could be loaded on demand, since most people probably won't want to play around with the examples).
   b. a text-based console, it opens the static text-based version.
4. the static html pages could be hosted on the website.

[simonbyrne]

On a related note, generating the output at build time (as opposed to saving it in the repo) would help avoid some problems with inconsistent documentation (e.g. #4747).

[pao]

On a related note, generating the output at build time (as opposed to saving it in the repo) would help avoid some problems with inconsistent documentation (e.g. #4747).

Also #2501.

[ViralBShah]

@shashi has an early converter that converts our documentation to IJulia notebooks.

[shashi]

Here https://github.com/shashi/rst2ipynb is the script that can use Pandoc to parse rST and generate valid ipynb files. Has a lot of rough edges, it's possible to fix all of them in a stop-gap manner. The rough edges are there mainly because Pandoc doesn't parse Sphinx variant of the rST.

So a better way of implementing this is as a Sphinx extension (or a Pandoc backend?), which I tried to do till their OOP got to me. Will try again if there is interest.

[MikeInnes]

Also worth noting that Markdown.jl could handle this kind of thing easily, which might be useful for turning inline documentation into notebooks. Would be kinda cool to type ?fft and have a notebook generated on the fly.

[johnmyleswhite]

+1 for focusing on Markdown.

[shashi]

+1. Definitely a much better direction than sticking to Sphinx.

[staticfloat]

@one-more-minute I really like the "notebook generated from markdown" approach. Seems like something that fits into the Julia way of doing things very neatly.

[simonbyrne]

Possibly relevant to generating ipython notebooks: ipython/ipython#4930

[Carreau]

+1. I, at least, have no objection in requiring Julia for a markdown to notebook converter.

Envoyé de mon iPhone

Le 28 nov. 2014 à 12:42, Simon Byrne notifications@github.com a écrit :

Possibly relevant to generating ipython notebooks: ipython/ipython#4930

—
Reply to this email directly or view it on GitHub.

[mauro3]

Should this be closed or is this still relevant?

[stevengj]

I think it can be closed.

[simonbyrne]

Why is it no longer relevant?

[stevengj]

Because we've gone in another direction for documentation of Julia itself. It still might be interesting for someone to do a notebook/manual conversion package, but that's more an issue for nbconvert or something else than for Julia Base.