Reinventing the wheel: jupyter-datatables package

[CermakM]

Hello there,

I like that there is a demand for this feature and that I am not the only one who sees that :). However, the functionality is really being duplicated here, which is perhaps shame and unnecessary reinventing of the wheel. Also, it is a great display of my failure in marketing the package properly :)

Feel free to check out the existing Jupyter DataTables

I'd very much appreciate if you fed back to me about the features that you miss in the aforementioned package and/or contributed yourself! :)

Also, Jupyter DataTables only supports Jupyter Notebooks at the moment, so it might be worthwhile to focus the attention to JupyterLab (or any other media you wish to be supported).

Cheers!
Marek

[mwouts]

Hello @CermakM , I'm glad to see you here! Let me also mention that from the beginning your work on jupyter-datables ranked high in the project references.

Maybe I should tell you a bit more about the purpose of this project. I'd love to be able to use a robust table component for both Jupyter and Dash (and possibly also PyCharm/VScode). I think that the DT package in R is inspiring: the same package can be used in notebooks, in the IDE, and in Shiny applications.

I am afraid we are very far from that in the Python world:

• In Jupyter notebook I can use qgrid. But it won't display nicely all my tables, and also I really prefer to type df than show_grid(df).
• In Jupyter Lab I can use ipysheet by @maartenbreddels and @martinRenou
• or maybe, I was an expert of ag-grid then I could use ipyaggrid by @DGothrek.
• In Dash I can use dash-datatable by @chriddyp, @alinastarkov, @Marc-Andre-Rivet, and others.

But clearly that's far from my ideal - I don't really want to learn using more than one table framework, and I want to do at least both Dash apps and notebooks with the same table.

What I'd love to trigger (rather than implementing here - itables is more a proof of concept than a real project), is to encourage someone/or a group with the necessary skils to propose a package that would

• be robust - it could ideally display tables of any shape or content (I'm ok with only showing an extract of large tables)
• be easy to use - default options should be good enough
• could be fined tuned by advanced users - as you may have seen, I like the idea of applying extracts of the javascript documentation to the Python package.

Let me know what you think. Also, @chriddyp, I'd love to ear what the Dash team thinks of allowing Jupyter users to use their table component in a notebook (I know that I can use plotly tables, and I also know that I can launch Dash app in Jupyter Lab, but what I really want here is a real table component in a notebook).

[CermakM]

Hey @mwouts !

Thanks! :)

Let's put our heads together :).

I totally get your idea. In fact, I share that ideology and I wrote Jupyter DataTables with that in mind as well (see further). Having a single environment-agnostic component that could be used to display the data would be wicked! :)

However, from what I've seen, you've been experimenting and investing your time in Jupyter Notebooks at the moment and basically re-implementing stuff that is already there in the jupyter-datatables package. Even the issues like #6 and #8 have already been solved there.
Don't get me wrong, I am not saying that because I want to promote the jupyter-datatables --- of course, I want, who wouldn't want contributors :) --- I am saying that because I know how much time I spent solving all of the problems around it.

To address your goals: Jupyter DataTables are basically pure JS, so that shouldn't be the problem when it comes to porting that to Jupyter Lab. What's holding it back is the custom JavaScript executor that I wrote for Jupyter Notebooks which takes care of the most aspects of Python-JS communication and is therefore an integral component: the jupyter-require package.

Once this is ported to JupyterLab, it should be a piece of cake to run DataTables there.

As far as Dash is concerned, I've never worked with it to that extent so that I could give an educated answer, but from what I understand, the components are written in React. Which means that it should be just a matter of wrapping the Jupyter DataTables into a React Component and passing in the data.

There is currently a gap IMHO in Jupyter Notebooks, which is the Python-JS communication and synchronous JS execution and that's what jupyter-require is trying to solve. To be honest I haven't invested my time in Jupyter Lab yet, so I am not sure how that works there. I'd appreciate your input on that very much. I would also be very glad if you took a look at jupyter-require and gave me your idea about the difficulty of porting that to Jupyter Lab extension. Once that is solved, we have 2/3 solved and Dash React component would be the next in the queue.

Looking forward to your inputs! :)

Cheers,
Marek

[maartenbreddels]

Thanks for pushing on this @mwouts!

I don't have time now to go into this deeply, but something to be aware of is that this is also possible with vuetify / ipyvuetify:

https://mybinder.org/v2/gh/mariobuikhuizen/ipyvuetify/master?filepath=examples%2FExamples%20template.ipynb

And we've used that in glue-jupyter to make a very custom table:
glue-viz/glue-jupyter#129

Which also works with a vaex dataframe (kernel side processing, 1 billion rows).

Hopefully that adds to the inspiration as well :)

[CermakM]

That is actually pretty neat! Thanks for the pointer @maartenbreddels

[mwouts]

Hello @CermakM , @maartenbreddels , thanks for your inputs!

I liked the idea behind the jupyter-require package, and also your readme there, that's very interesting, and the d3 example is fun! But maybe that is still in an early stage, and probably I would prefer to adopt a technology that has the potential to deliver the table to any of Jupyter Notebook, Lab, Jupyter NBconvert, and Dash. Plotly have a great expertise on this, since they deliver Plotly javascript plots in all of these environments, and thus I think we should have a look at how they do that.

@maartenbreddels, thanks for the screenshot, that's fairly impressive! I'll give a try to ipyvuetify, and see how it works. Would you have a sample notebook to reproduce that example (or a simpler table?)

Regarding the next steps, this is what I am thinking of:

• a review of the existing table framework (tables available in Jupyter). Recap the features in a summary table: do they work in Jupyter Notebook/Lab/NBconvert/Dash, offline, do they support inputs of any types, do they display nicely tables with many rows/columns/multiindices, do they implement a server mode for large tables, can we edit the table,... If you like the idea, I can share a notebook with a series of test, and we could run it on the candidate table frameworks.
• a reflexion on how to best connect the table component with Jupyter. Probably ipywidgets is the way to go? What are the limitations? Why does Plotly uses another solution?
• Can we offer a table framework in Jupyter that will be usable by people who know nothing about the underlying JS table component? Can we offer a full mapping of JS options for those who are expert of the JS components? Without giving the impression that you need to be an expert to use the Python component?

Let me know what you think!

[neon-ninja]

Personally, I chose to use itables over jupyter-datatables for the Advanced cell formatting / HTML in cells features. See https://uoa-eresearch.github.io/twitter_analysis/plot.html for an example of how I used these.

[mwouts]

Oh that's interesting! Thank you @neon-ninja for the link. May I ask if you had any previous experience with datatables.net? Was it easy to find that in the documentation?

[neon-ninja]

Yes, I used it directly for a leaflet js project a few years ago. Yes, your example / links were very helpful :)

[maartenbreddels]

@maartenbreddels, thanks for the screenshot, that's fairly impressive! I'll give a try to ipyvuetify, and see how it works. Would you have a sample notebook to reproduce that example (or a simpler table?)

The mybinder link: https://mybinder.org/v2/gh/mariobuikhuizen/ipyvuetify/master?filepath=examples%2FExamples%20template.ipynb

Personally I think Vue is a really good framework for people that don't want to do heavy frontend development like React, but aren't afraid of a little bit of html-like syntax, and sprinkle a bit of JS in the mix.

Embracing Vue, Vuetify seem like one of the best component frameworks, where it's v-data-table component is not an opinionated component, but something to make your custom table viewer with. It supports client and server pagination/sorting/searching etc, and integrates well with vuetify.

Via ipyvuetify it's easy to expose them as a widget for Python (via the template system shown in the binder link).

[mwouts]

Awesome! I love what you show there. When time permits I'll see if I can draft a version of itables based on ipyvuetify!