Make the sage notebook optional

[timokau]

If you're coming from the deprecation notice in the notebook

The legacy SageNB notebook has been effectively deprecated for quite some time.
We are now making that official.
But that does not mean that Sage will not have notebook functionality.
In fact, most of the functionality of the sage notebook can already be replaced by using the Juypter Notebook instead.
For an introduction on how to use the new notebook and convert existing notebooks, see the documentation.

There are some features that are not yet ported to the new notebook however.
We will not completely stop supporting the legacy notebook until those issues are resolved.
Those features are listed below. If there is some feature you're missing that is not listed or you want to give general feedback, feel free to leave a comment on this ticket. If you do not have an account yet, you can log in with a GitHub account.

Implementation details

Reasons to make sagenb optional

• most of its functionality has been replaced for a while by Jupyter

• it is not actively maintained -- merely on life support

• it has various open issues (looking at the "Critical" label in the github repo)

  • unmaintained dependencies (Unmaintained dependencies sagenb#440 (comment))
  • incompatibility with the latest flask-babel (sage-on-gentoo: flask-babel upgrade is breaking thing sagenb#437
  • incompatibility with python3 (Port to Python 3 sagenb#343)
  • an XSS vulnerability, unfixed for 3.5 years (Cross-site reference vulnerability sagenb#319)

• and for me personally: I don't want to package it. I feel like it causes a disproportionate amount of issues considering its relatively low value in today's Sage.

Reasons to keep sagenb around

• there are still some features depending on it

• there might still be people using it. An example would be @kiwifb's university: #25382 comment:16

In making it optional, we still make it possible to use it for now. And when it inevitably breaks further, the people relying on it can weigh the effort of keeping it alive themselves against the effort of switching to the new notebook.

Tickets and issues tracking progress/going in that direction

Necessary to keep all tested behaviour intact without sagenb ("pass the doctests")

• Add package phitigra: Graph editor that works with Jupyter #30540: port/remove the graph editor

• port/remove the graph database navigator

• port/remove the interactive debugger

• py3: do not include the notebook documentation in sage #25382: do not include sagenb docs in sage

Misc changes

• Add a deprecation warning to the legacy sage notebook #25852: add a deprecation warning to the sage notebook

• Make the sage notebook optional #25837: actually make sagenb optional

Functionality sagenb currently does better than Jupyter

• Improve introspection capabilities of Sage Jupyter Kernel #25015: object introspection (<obj>?) in the Jupyter notebook does not display HTML-formatted docs (esp. with latex formatting), while sagenb does

• search_doc broken with jupyter notebook #24784: search_doc does not work in the Jupyter notebook

• inconsistencies between the terminal version and the notebook #9048: automatic_names only works in sagenb

• Jupyter's converting to PDF "kind of sucks" (see Broken typeset in pdf export of Jupyter notebook #23330)

• Inline view in the Jupyter notebook #26101: With Jupyter, view(...) currently displays the pdf in an external pdf viewer, launched from the server. An inline view as in sagenb is nicer and would work remotely as well.

• Jupyter does not scroll large output (e.g. matrices) well; see this post on Stackoverflow

• probably supports matrix input, which does work in Sage cell server but not in Jupyter as of this Sage support discussion

Otherwise relevant

• upgrade sagenb and build sagenb in sage/python3 #22431: first step in making sagenb python3 compatible
• https://wiki.sagemath.org/IpythonNotebook (which features of the Sage notebook are missing in the jupyter notebook, to be updated)
• MathJax fonts -- I don't know if it was possible to change the font in the legacy notebook either, but it's not easy to configure a different font in Jupyter notebook. See e.g.:
  • Please add TeX fonts for MathJax, not just STIX-web (or make it very easy for end-users to do so) jupyter/notebook#2536
  • https://ask.sagemath.org/question/43082/configuring-mathjax-fonts/

CC: @embray @vbraun @jdemeyer @kcrisman @kiwifb @antonio-rojas @timokau @infinity0 @novoselt @pcpa @nthiery @slel @saraedum @haraldschilly @strogdon @tobihan @sagetrac-tmonteil @williamstein

Component: packages: standard

Keywords: notebook, sagenb

Branch/Commit: u/chapoton/25837 @ ec7931a

Reviewer: Dima Pasechnik, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/25837

[embray]

comment:1

Some issues I'm aware of related to features that are only in sagenb:

• Improve introspection capabilities of Sage Jupyter Kernel #25015 - object introspection (<obj>?) in the Jupyter notebook does not display HTML-formatted docs (esp. with latex formatting), while sagenb does. I believe Odile might be looking into this now. Not a significantly difficult issue but not trivial either.

• search_doc broken with jupyter notebook #24784 - search_doc does not work in the Jupyter notebook

• inconsistencies between the terminal version and the notebook #9048 - automatic_names only works in sagenb. This was also discussed recently on sage-devel: https://groups.google.com/d/msg/sage-devel/RoBZ36i1xo4/e6ZunaKRAwAJ I believe automatic_names() can be reimplemented in such a way that's mostly independent on the front-end, though one would still need front-end specific hooks for it to work. I haven't tried this yet though.

I think at a minimum these issues should be fixed before taking further steps at abandoning the old sagenb.

[kcrisman]

comment:2

I would also say that something "official" when one uses sagenb even when directly that says it is deprecated is necessary.

[timokau]

comment:4

I modified the ticket description to incorporate (read: shamelessly copy) your additions.

It would be nice if we could get the deprecation warning into 8.3 so that users will have at least one release to prepare. Thats a very optimistic estimate of course, but getting it in 8.3 would be nice either way.

[timokau]

comment:5

I have opened #25852 for that. I have only ever actually started the notebook once. Do you have an idea where best to add such a message? Just print it on the command line?

[kcrisman]

comment:6

Probably on the notebook itself as well, which would require changes in sagenb.

[slel]

comment:7

cc a few people who package Sage in various ways

[slel]

Changed keywords from notebook to notebook, sagenb

[embray]

comment:8

Another issue that was brought up here: https://ask.sagemath.org/question/43051/file-download-as-latex/

Converting a Jupyter notebook to PDF kind of sucks at the moment for notebooks that contain a lot of math in latex. There are two main cases in the notebook where latex can appear, and both have issues in sage:

1. If an object output from a code cell has _render_latex_ then the notebook will display the rendered latex with mathjax. However, when converting the notebook to PDF, nbconvert just takes the plain-text representation of the output, not the latex, and puts it in a verbatim block. Not good, especially not for Sage.

2. If a markdown cell contains equations in latex, nbconvert will render them correctly, but only if you have pandoc installed. This is already a known issue, but since pandoc does not come with Sage and is not a dependency this makes for a sucky user experience for something that should Just Work.

I'm not sure what to do about the first case; it seems like an issue for nbconvert but I haven't explored it too deeply. For the latter case (which probably also affects the first case) I wonder if we shouldn't just make pandoc another sage dependency.

[timokau]

comment:9

I've updated the ticket. The flask issue was resolved (thanks @fchapoton!), I added the pdf complaint and improved formatting.

[timokau]

comment:11

I've added a section intended for end users. Please improve on that if you have more knowledge about sagenb or the Jupyter Notebook, which you probably do.

[williamstein]

comment:15

Those features are listed below.

Where is the list? I can't find it.

I touched on this on page 26 of a talk I gave yesterday (https://share.cocalc.com/share/6cd832d3-c523-41e3-9e54-c8f2d2e8fa2a/travel/2018/2018-07-icms/talks/cocalc/slides/slides.html#1)

One example is that if you start running:

for i in range(10):
   print(i)
   sys.stdout.flush()
   time.sleep(1)

then close and open your browser (to simulate a network failure, say), you
should not loose output. In Jupyter, you do.

[embray]

comment:16

Replying to @williamstein:

Those features are listed below.

Where is the list? I can't find it.

It's in the description of the ticket, but perhaps that still needs more reformatting, because there are a number of thing "below" that sentence that include, but are not limited to, missing features.

Most of it falls under the heading "Functionality sagenb currently does better than Jupyter" though that list is far from complete. There's also a link to a wiki page listing other issues.

[jhpalmieri]

comment:19

"#24784: search_doc does not work in the Jupyter notebook" is now ready for review.

[fchapoton]

comment:20

Could someone (having the required rights) please now merge the 4 top pull requests in sagenb github, including the one related to the present ticket ?

https://github.com/sagemath/sagenb/pulls

[dimpase]

comment:21

Replying to @fchapoton:

Could someone (having the required rights) please now merge the 4 top pull requests in sagenb github, including the one related to the present ticket ?

https://github.com/sagemath/sagenb/pulls

done, I guess. See #26499 for the new release of sagenb.

[fchapoton]

Branch: u/chapoton/25837

[fchapoton]

New commits:

ec7931a

make sagenb optional

[jhpalmieri]

comment:42

When we talk about regressions, I think the context is: we are imagining people converting their notebooks from SageNB to Jupyter and not using SageNB any more (as opposed to: using SageNB for some tasks, Jupyter for others). In that context, it is also worthwhile to ask what features Jupyter has that SageNB doesn't. I don't think we are just losing things by moving from SageNB to Jupyter; aside from Python 3 compatibility, I think we gain some things by moving to Jupyter. So it is a trade-off. At what point does the balance tip to the Jupyter side? Maybe the Python 3 support is enough on its own, in fact.

[kcrisman]

comment:43

Maybe the Python 3 support is enough on its own, in fact.

Obviously that is the consensus decision. I'm just quoting this ticket right now when I mention these things, and would hate to have them disappear for all time. But as you said elsewhere, someone has to work on them.

[nthiery]

comment:44

A functionality that Jupyter is currently missing over sagenb is the rich text editor. This may possibly be resolved by:

https://github.com/jupytercalpoly/jupyterlab-richtext-mode

[dimpase]

comment:45

the most pressing issue is inability to normally use jsmol in Jupyter notebooks, needed to export 3d drawings for animations. We are having a lot of complaints about it.
There is something with the jsmol pop-up menu, which gets cut out by the notebook cell, and can't be used (thus making exports impossible)

jupyter/notebook#1872 discussing this has been opened in 2016 (pity that it was not resolved during ODK).

[nthiery]

comment:46

Indeed; too bad this was not brought up. That being said, we still are in good contact with the Jupyter team :-) So if we provide them with a specific and easy to reproduce example (e.g. through a binder link) on the above issue, there are chances that we can make progress. It would be worth testing how things are in jupyterlab too.

[kcrisman]

comment:47

By the way, I note that only jsmol can make stereographic plots for 3d viewing, though I don't know how often this capability is used. (People like it when advertising Sage!)

[dimpase]

comment:48

Replying to @kcrisman:

By the way, I note that only jsmol can make stereographic plots for 3d viewing, though I don't know how often this capability is used. (People like it when advertising Sage!)

can't plotly do it? see e.g. https://plot.ly/~empet/13037.embed

[kcrisman]

comment:49

By "only jsmol", I meant the only thing included standard in Sage. What is plotly? (Am I so behind that I missed a major new Sage internal plot mechanism? Yikes.)

[dimpase]

comment:50

I read your comments as "the only Python library that can do this sort if plotting"

plotly is Python library, no idea if it can work in Sage.

[embray]

comment:51

Ticket retargeted after milestone closed

[mkoeppe]

comment:52

This was done another ticket merged for 9.1 (see #29320), should be closed.

But #17590 (update sagenb SPKG.txt) needs work for 9.1

[jhpalmieri]

Changed reviewer from Dima Pasechnik to Dima Pasechnik, John Palmieri