plotly.io HTML functions, modular renderers framework, future flags system

[jonmmease]

Overview

This PR is a big one! It tackles three version 4 goals that ended up being so intertwined that I decided to handle them in the same PR.

1. Adds to_html and write_html functions to the plotly.io module and re-bases plotly.offline.plot on top of these new functions. This finishes porting the existing figure export logic into the plotly.io module as initially described in New module proposal: plotly.io #1098.
2. Adds a new plotly.io.renderers configuration object to control how figures are rendered, and a new plotly.io.show function for displaying figures. Reimplements plotly.offline.init_notebook_mode and plotly.offline.iplot on top of the new renderers framework.
3. A new _plotly_future_ module that will allow users to opt-in to future (v4) default behaviors before v4 is released.

Despite the breadth of these changes, I believe this PR is 100% backward compatible for users, and so I would like to target releasing this for plotly.py version 3.8.0

plotly.io HTML functions

This PR adds two new public functions to plotly.io: to_html and write_html. Per the convention discussed in #1098, to_html inputs a figure and returns an html representation as as string. And write_html inputs a figure and a file string/object and writes the html representation to the specified file.

post_script

The signatures for these functions map pretty closely to plotly.offline.plot with one exception. plotly.offline.plot includes a set of image export options that can be used to add JavaScript to the exported figure that will ask plotly.js to export the figure to an image and then ask the browser to download it.

This is not a workflow that we recommend anymore now that orca is available and integrated into plotly.py (See #1120), so I didn't want to add this support directly to to_html/write_html. Instead, I added a post_script argument that can be used to add any custom JavaScript snippet to the resulting HTML. This custom HTML is executed after the plot is created (in a .then block) and it can include '{plot_id}' placeholders that are automatically replaced with the id of the div that the Plotly.js plot is attached to. Here's the docstring

    post_script: str or None (default None)
        JavaScript snippet to be included in the resulting div just after
        plot creation.  The string may include '{plot_id}' placeholders that
        will then be replaced by the `id` of the div element that the
        plotly.js figure is associated with.  One application for this script
        is to install custom plotly.js event handlers.

Another use-case I had in mind where this would have come in handy was this question on the forums of how to add custom JavaScript to a plot exported by plotly.py to open hyperlinks in response to marker clicks. https://community.plot.ly/t/hyperlink-to-markers-on-map/17858/6

responsive output

One other change I made while I was in there was to remove our custom window resize event handler logic and replace it with the (relatively) new plotly.js 'responsive': true config parameter. So if the input figure does not have a predefined layout.width and layout.height thento_html will add 'responsive': true to the figure config.

plotly.io.renderers

This PR introduces a new plotly.io.renderers configuration object. This work ended up following the design I laid out in #1459 pretty much to the letter, and I've updated the proposal description there to match the details of this implementation.

Design Influences

This design is somewhat inspired by the approach used by the Altair project (https://altair-viz.github.io/user_guide/renderers.html), the Altair implementation was a helpful reference in implementing this.

The design also draws on elements of the plotly.io.templates design (See #1224), and is intended to have a consistent user experience.

Default renderer

I've added a new configuration object named renderers in the plotly.io module. This sits alongside the plotly.io.templates and plotly.io.orca configuration objects. Users can view and specify the current default renderer using property assignment. For example, to specify the equivalent of init_notebook_mode(connected=False):

import plotly.io as pio
pio.renderers.default = 'notebook+plotly_mimetype'

repr and show

There are now two ways to display figures. If one or more mime-type based renderers are defined and the plotly.io.renderers.render_on_display property is set to True, then the go.Figure instances will display themselves automatically as their notebook representation. If plotly.io.renderers.render_on_display is set to False, then mimetype based rendering will not be performed when figures are displayed.

In addition, all renderer types (including external renderers that will be described below) can be invoked using a new plotly.io.show function. This function will display the figure as a side effect and will return None.

The show functions will also allow the user to override the default renderer. e.g.

import plotly.io as pio
pio.renderers.default = 'notebook' 

fig = go.Figure(...)

# By default show will display figure inline, override to display in browser tab
pio.show(fig, renderer='browser')

Then to display a figure with the equivalent of plotly.offline.iplot:

pio.show(fig)

Available Renderers

Here is a detailed description of the renderers that are included in this PR:

plotly_mimetype

The plotly_mimetype renderer displays figures using the application/vnd.plotly.v1+json mime type that is handled in JupyterLab by the @jupyterlab/plotly-extension extension, and is handled natively by both nteract and the Visual Studio Code.

For user friendliness, the plotly_mimetype renderer is also aliased as jupyterlab, vscode, and nteract

HTML-mimetype

Several renderers that produce bundles with mimetype text/html are included.

notebook

This renderer is equivalent to plotly.offline.init_notebook_mode(connected=False) and plotly.offline.iplot. It works in the classic Jupyter Notebook, and works offline. This renderer is also useful for notebooks that will be converted to HTML using nbconvert/nbviewer as it will produce standalone HTML files that include interactive figures.

When the renderer is activated in loads the full plotly.js bundle (~3MB) into the notebook and registers it as 'plotly' with requirejs (which is included in the classic notebook). Then when figures are renderer, they reference plotly.js using requirejs.

notebook_connected

Same as notebook except that the initialization phase uses the plotly CDN rather than loading the full plotly.js bundle into the notebook. This results in much smaller notebooks but requires an internet connection for use.

This is the best rendering mode for use in kaggle notebooks, so it has also been aliased as 'kaggle'

colab

Google Colab has a different architecture than either JupyterLab or the classic notebook in that each output cell is an iframe, so there can be no global initialization for the notebook. Also, requirejs is not available by default. The Colab renderer borrows insight from the Altair colab renderer and produces full HTML file bundles that include plotly.js using a <script> tag.

iframe

This renderer was inspired by @Nikolai-Hlubek's observations in #1469. It works be saving figures to standalone html files (using plotly.io.write_html) in a directory next to the notebook. Then the HTML mimebundles that are inserted into the notebook are iframe elements with the relative path to these HTML files.

I haven't tested the performance characteristics yet, but the approach works well in the classic notebook and in JupyterLab and it will result in much smaller notebooks for cases where a notebook contains many large figures. Export to standalone HTML still works fine as long as a copy of the iframe_figures directory is kept alongside the HTML file.

One limitation for this approach is that it doesn't currently support responsive resizing. And I'm not sure if this is something that's possible through an iframe.

HTML-external

As an alternative to the mime-type rendering approach, where renderers subclass MimetypeRenderer, there is the external rendering approach, where renderers subclass ExternalRenderer. These renderers are external in the sense that they render the figure outside of the notebook, and don't include any reference to the figure in the notebook itself.

browser

The 'browser' renderer displays interactive figures in a new tab/window in the users default browser. This is done without the use of temporary files (which would be challenging to manage) by creating a single use HTTPServer on a local port that responds to exactly one request with the figure's contents and then shuts down.

This approach has no Jupyter/notebook dependencies works fine in the QtConsole, ipython console, and even the vanilla python repl.

firefox and chrome

There are also predefined renderers that allow the user to specify that the new tabs should be opened in firefox or in chrome.

Static image renderers

A collection of static image renderers are included that rely on the orca integration to render figures as images. In addition to the Jupyter Notebook/JupyterLab, these renderers work well in even static contexts like the QtConsole/Spyder.

'png', 'svg', 'jpg', 'pdf'

Static rendereres are predefined for png, svg, jpg, and pdf formats. I'm especially excited about the pdf static renderer because this is picked up by the LaTeX-based PDF export functionality of nbconvert. This means that with the pdf renderer enabled, a notebook can be exported through LaTeX as a PDF and the figures will come along in vector form.

json

This renderer allows the figure's JSON structure to be displayed in the notebook using the JSON viewers in JupyterLab/VSCode. This is really handy for being able to navigate the structure of a large figure with expand/collapse toggling and search-based filtering. Here's a YouTube clip of the what this looks like https://www.youtube.com/watch?v=FRj1r7-7kiQ.

Combining renderers

Multiple renderers can be registered as the default by separating the renderer names with '+' characters. This is the same combination syntax used to combine templates in plotly.io.templates. When multiple mime type renderers are specified this way, a single bundle will be created with the render representation for each one. As motivation, consider

pio.renderers.default = 'notebook+plotly_mimetype+png'

A notebook with this renderer specification would display figures properly in the classic notebook ('notebook'), in jupyterlab/vscode/nteract ('plotly_mimetype'), in exported HTML ('notebook'), in the QtConsole/PyCharm ('png'), and in exported PDFs ('png').

Of course this would result in fairly large notebook sizes, but the user will have the flexibility to define where this code needs to render figures.

Auto-detecting renderers

In a few cases, we're able to detect when it's appropriate to enable a specific renderer by default. In particular, this PR includes logic to detect when plotly.py is running in Google Colab, Kaggle, or VSCode.

Customize renderer properties

Some renderers can expose additional configuration options. In this case, a new renderer can be constructed and named. Each built-in renderer is defined by a class accessible under plotly.io.base_renderers.

E.g. to specify a static image renderer with custom properties:

import plotly.io as pio
pio.renderers['my_png'] = pio.base_renderers.PngRenderer(width=1000, height=650, scale=1.5)
pio.renderers.default = 'my_png'

Or the properties of an existing renderer can be modified directly:

pio.renderers['png'].width = 1000
pio.renderers['png'].height = 650
pio.renderers['png'].scale = 1.5
pio.renderers.default = 'png'

Or the properties can be overridden temporarily in plotly.io.show:

pio.show(fig, renderer='png', width=1000, height=650, scale=1.5)

Registering new renderers

Users or third-party-libraries may register new renderers by subclassing plotly.io.base_renderers.MimetypeRenderer or plotly.io.base_renderers.ExternalRenderer

import plotly.io as pio

# Define custom mimetype renderer
class MyRenderer(pio.base_renderers.MimetypeRenderer):
    ...
    def activate():
        ...

    def to_mimebundle(self, fig_dict):
        ...

# Or a custom external renderer
class MyExternalRenderer(pio.base_renderers.ExternalRenderer):
    ...
    def activate():
        ...

    def render(self, fig_dict):
        ...

# Register for use by name
my_renderer = MyRenderer()
pio.renderers['my_renderer'] = my_renderer

# Set as default
pio.renderers.default = 'my_renderer'

Backward compatibility

For backward compatibility, plotly.offline.init_notebook_mode and plotly.offline.iplot will stay around, but they will be implemented on top of the new renderer framework.

If we are able to merge this work in version 3, the default renderer will be plotly_mimetype and iplot will call plotly.io.show. Calling plotly.offline.init_notebook_mode() will set the default renderer to 'plotly_mimetype+notebook' and plotly.offline.init_notebook_mode(connected=True) will set the default renderer to 'plotly_mimetype+notebook_connected'.

The _plotly_future_ system

Inspired by the Python __future__ mechanism for opting in to the default behavior of future version of the language, this PR introduces a _plotly_future_ module that can be used to enable features/behaviors that will be the default in plotly.py version 4.

In every case, these _plotly_future_ import must come before plotly itself is imported otherwise an error is raised.

For example, the default behavior of this PR for displaying figures matches that of plotly.py version 3. But the renderer_defaults flag can be used to enable the future v4 defaults, where figures will display themselves automatically.

from _plotly_future_ import renderer_defaults

Or, to enable the plotly theme that we plan to make the default in version 4

from _plotly_future_ import template_defaults

And to enable all version 4 features that are available (only the two above for now)

from _plotly_future_ import v4

I'd like to continue adding flags here as a way to merge in v4 updates before v4 itself. This way the community can weigh in on changes more easily (no need to download a separate version, just from _plotly_future_ import v4 at the top of your script), and we can more easily start working on updating documentation to the v4 style before v4 is released.

cc @jackparmer @chriddyp @nicolaskruchten @cldougl @michaelbabyn

TODO:

• Add animation options support from Adding animation opts to python interface #1503

[jonmmease]

I'm going to try to review and merge this PR soon, so let me know if anyone has any other comments!

[astrojuanlu]

@GorgiAstro if you have some spare minutes, can you check how does this work with documentation? perhaps you already noticed readthedocs/readthedocs.org#4367 (comment) Sorry @jonmmease I haven't been able to test this!

[ricardofbarros]

@jonmmease tested this PR in my classic notebook and the performance is way better 👍 There's no lag moving the chart and it loads way faster.

I installed this:

pip install https://11196-14579099-gh.circle-artifacts.com/0/dist/plotly-3.7.0%2B44.g2677dafe.tar.gz

and tested like this:

from _plotly_future_ import renderer_defaults
import plotly.io as pio
import plotly.graph_objs as go

import plotly

trace = go.Ohlc(x=df.index,
                open=df['open'],
                high=df['high'],
                low=df['low'],
                close=df['close'])

fig = dict(data=[trace], layout={'title': 'OHLC'})

pio.show(fig)

[GorgiAstro]

@GorgiAstro if you have some spare minutes, can you check how does this work with documentation? perhaps you already noticed rtfd/readthedocs.org#4367 (comment) Sorry @jonmmease I haven't been able to test this!

I tested an example Notebook from poliastro, by installing the dev version of Plotly (on top of the Python environment I use for poliastro) by means of pip install https://11196-14579099-gh.circle-artifacts.com/0/dist/plotly-3.7.0%2B44.g2677dafe.tar.gz

After running and saving the notebook, I ran the command make html in the poliastro/docs directory.

All renderers were tested, here are the results of the HTML export to Read-The-Docs:

• plotly_mimetype/jupyterlab/vscode/nteract: exported as png
• notebook: nothing
• notebook_connected: nothing
• iframe: the carl sagan video shows up instead, wtf
• browser: nothing (expected)
• png: exported as png (expected)
• svg: exported as svg (expected)
• jpg: exported as jpg (expected)
• pdf: only the filename appears in the read-the-docs
• json: "Data type cannot be displayed: application/json"

The notebook used for the test is located here: https://github.com/GorgiAstro/poliastro/blob/test-plotly-new-renders/docs/source/examples/Going%20to%20Mars%20with%20Python%20using%20poliastro.ipynb

Nbviewer: https://nbviewer.jupyter.org/github/GorgiAstro/poliastro/blob/ec994082825fcc9496e866ec2184f29b00dfec85/docs/source/examples/Going%20to%20Mars%20with%20Python%20using%20poliastro.ipynb

[jonmmease]

Thanks for giving it a try @GorgiAstro! My expectation is that notebook should behave identically to init_notebook_mode()+iplot and notebook_connected should behave identically to init_notebook_mode(connected=True)+iplot. Were either of these iplot methods working for you in poliastro docs in previous versions? From readthedocs/readthedocs.org#4367 (comment), I gather that this wasn't working previously, but correct me if I'm wrong.

If RTD is having a problem with the requirejs dependency, I wonder if the 'colab' renderer would work as this loads plotly.js using a script tag. Unfortunately this approach won't work in the classic notebook itself, but it would be interesting to know whether RTD picked it up alright.

[GorgiAstro]

You're right @jonmmease, the methods were not working in poliastro docs in previous versions either (@Juanlu001, can you confirm?).

I added the colab renderer to the notebook, it does not get picked up by RTD.

[jonmmease]

Thanks for confirming @GorgiAstro.

I'm planning to merge this tomorrow unless something comes up.

[jonmmease]

Merging.

Now that we have this renderers interface, if someone is interested in digging in perhaps we could come up with an html renderer that works in both the classic notebook and RTD. For example, if requirejs is still an issue, we could add a 'readthedocs' renderer that automatically loads requirejs from a CDN like @Juanlu001's hack from https://rtd-js-test.readthedocs.io/en/latest/Connected%20-%20require.js%20Hack.html.

[GorgiAstro]

Great to hear that Plotly 3.8.0 has been released 🎉

Here are the new combined renderers at work: https://nbviewer.jupyter.org/github/GorgiAstro/laser-orbit-determination/blob/master/02-orbit-determination-example.ipynb