First draft markdown flavor JEP

[fcollonval]

This draft PR is to share the output of the Notebook format workshop on specifying the Markdown flavor description in notebook file.

Please discussed this proposal in the associate issue: resolve #98.

[rgbkrk]

This may allow going a bit more broad than I'd recommend, since mimetype can be a lot of unrelated and yet well specified types.

Conceptually though, calling it markdown_format would make sense. Limit it to an enum of possible variants:

• marked
• commonmark
• gfm
• commonmark+latex

[stevejpurves]

Apart from being well established way to identify the content the interesting thing using mimetype here is thst. is could potentially be used to specify other content like YAML, or HTML or SQL which could be rendered appropriately in a "markdown" cell - maybe we should think of markdown cell more as an "input" cell.

Immediately opens up the case for notebooks with hetrogenous inputs (for better or worse) including markdown variants, or things like YAML, or SQL and for that content to be properly identified - I believe this content is all already in use in notebooks but not well supported

[rgbkrk]

Rather than do it per cell, I'd aim for the notebook document itself to have the markdown format specified at the top level. There are text/markdown outputs after all.

[stevejpurves]

I'd love to see some more discussion on this point, We were a bit torn when discussing this at the workshop and a few of the open questions point to this..

A couple of scenarios we thought about here think about here:

1. a notebook is created from scratch in a single client, and edited there
2. a notebook is created by someone in client A, and then shared, opened and one cell is edited by someone else in a different client, client B. The two clients render/support different markdown flavours.

So when setting the cell mimetype type per cell

• felt like it was more in line with upcoming proposals to allow inputs to be fully mimebundle based
• In scenario (2), client B would have to make a best efforts to render the markdown cells on displaying the notebook. When a user edited a markdown cell - the client would then (what?) assume the content was it's markdown flavour and update the mimetype accordingly?
• opens up the doors for hetrogenous notebooks with different "Markdown" cells containing any mimetye identified content e.g. SQL, yaml and for that to be rendered appropriately by extensions (no matter what is intended, once. mimetype is available it'll likely be leveraged for this)

When setting the markdown type per notebook:

• in scenario (2) what would happen on edit? would a client change the notebook wide markdown type to it's own or leave it as be? would it try to convert content to it's own flavour (probably it's not in a position to!)
• or the mimetype is taken as a hint, to indicate what renderer is best suited to handle the content in the notebook
• should clients expose UI to change this?

It feels like without stepping out of this JEP and into the larger discussion new mimetype based inputs, which could accomodate a commonmark markdown fallback alongside -- it'll be hard to reconcile these issues around "editing" here but adding the mimetype field to either improves the experience of rendering, provided the content is consistent with the declared type!

[stevejpurves]

also note the discussion on this issue

[rgbkrk]

TIL there's a way to declare a variant of markdown.

[jjallaire]

Registered variants are here: https://www.iana.org/assignments/markdown-variants/markdown-variants.xhtml

[rowanc1]

Both quarto and myst have been added to the registry in the last few weeks based on follow ups from the notebook format meeting. If would be great to add them to the JEP!

[fcollonval]

Thanks for the follow up @rowanc1 ? Would you mind providing guidance how the process to add such variant?

[jjallaire]

You can register a new variant by filling out this form: https://www.iana.org/form/protocol-assignment

[krassowski]

I think that this proposal should ideally address the question of security model for proposed outputs. The current Jupyter Notebook security model prohibits execution of JS and forces sanitation of HTML/CSS in all Markdown cells. Since outputs in this proposal are modelled on code cells (which can be trusted or not), would it be worth re-visiting the trust question?

Please see more thoughts on potential trust evolution in #95 (comment).

[fcollonval]

We have started looking at this at the SSC meetings. We have decided to give at least another 2 weeks of discussion before moving forward.

[jjallaire]

What is the current thinking around the precise content type of the fallback ("Rendered Markdown source")? This might be clear in the revised text of the proposal but I wanted to be sure. The scenarios I'm thinking offer for renderers like nbconvert or for notebook editors that don't know about the specified mimetype. In these cases it seems like editing tools should provide a standard Jupyter Markdown version of the content -- e.g. if a callout supported by MyST or Quarto is in the markdown, the fallback should be something suitable e.g. a markdown blockquote). The alternative would be to provide e.g. text/html however if you are attempting to render a PDF or DOCX this won't be helpful (whereas Jupyter Markdown will be).

I also wonder if given that notebook editors won't immediately be aware of this extension whether the actual source should be always written in Jupyter Markdown, and the source that uses an extended markdown variant be provide separately. This would allow "variant aware" editors to deal with the content properly but allow the rest of the ecosystem to keep working without changes? This might already be anticipated in an aspect of the proposal I'm not appreciating, but in the case that its not is definitely worth considering.