Versioned installation of threejs

[paulmasson]

From Matthias Koeppe on #30915:

When we start using Sage with a system jupyter notebook (Meta-ticket #30306), there is an issue involving the installation directory share/jupyter/nbextensions/threejs, and the fact that there will be several unrelated (and possibly mutually incompatible) copies of that.

Here is an example.

Let's say:

• system jupyter is installed in /usr, so the notebook server is accessing /usr/share/jupyter/nbextensions.
• Sage-9.x may be installed in /home/x/sage-9.x/local/ and may need threejs r122.
• Sage-9.y may be installed in /home/x/sage-9.y/local/ and may need threejs r155.
• Let's say r122 and r155 are mutually incompatible.

As @enriqueartal has reported in #30915, our offline threejs graphics needs /usr/share/jupyter/nbextensions/threejs/. But the system does not provide it -- it is only provided by Sage (and, after #30915 is the result of a custom build (fork) of threejs).

If we create a symlink /usr/share/jupyter/nbextensions/threejs -> /home/x/sage-9.x/local/share/jupyter/nbextensions/threejs, then Sage 9.x will work with the system jupyter notebook; but Sage 9.y will not.

This means that we need a versioned installation scheme so that offline threejs graphics can access the specific version it needs even if the user is accessing Sage through the system notebook.

In this ticket we also rename directories from threejs to threejs-sage (in share/ and share/jupyter/nbextensions/) to reflect the fact that it is a Sage-specific fork. This will reduce the need for downstream patching.

(On this ticket we ignore the separate issue that installing such symlinks is not user-friendly -- see #30123 Repackage Sage's cropped threejs as a pip-installable package jupyter-threejs-sage for a follow-up.)

CC: @jcamp0x2a @kiwifb @mkoeppe @EmmanuelCharpentier @enriqueartal @jhpalmieri @seblabbe @antonio-rojas @defeo @novoselt @dimpase

Component: graphics

Keywords: threejs, jsmol, jupyter, sd111

Author: Matthias Koeppe, Joshua Campbell

Branch/Commit: 935dcef

Reviewer: Joshua Campbell, Matthias Koeppe, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30972

[paulmasson]

comment:1

I'm no expert on Python, especially the way implementations my differ, but it can access the local file system and Sage already has THREEJS_DIR. Why not set the symlink to Three.js or any package when the kernel initially loads? That way the system package will use whatever version accompanies the kernel.

[mkoeppe]

comment:2

That's not a solution because a user may want to use two different Sage versions at the same time.

[paulmasson]

comment:3

Replying to @mkoeppe:

That's not a solution because a user may want to use two different Sage versions at the same time.

Why exactly would someone want to do this? I can see a user not wanting to upgrade to the newest version, but why would they use an older version when the newer one will almost certainly be an improvement?

[kiwifb]

comment:4

Should we use JUPYTER_PATH for this kind of things? https://jupyter.readthedocs.io/en/latest/use/jupyter-directories.html

[mkoeppe]

comment:5

Replying to @paulmasson:

Replying to @mkoeppe:

That's not a solution because a user may want to use two different Sage versions at the same time.

Why exactly would someone want to do this?

Users have complex needs.

[paulmasson]

comment:6

Replying to @mkoeppe:

Replying to @paulmasson:

Replying to @mkoeppe:

That's not a solution because a user may want to use two different Sage versions at the same time.

Why exactly would someone want to do this?

Users have complex needs.

Considering that I don't even use Sage, I'll need a better answer than this before devoting more time to this issue. We are not required to support every feature desired by end users. We can simply tell them to run only one kernel at a time. Who is making the decision to support this option?

[mkoeppe]

comment:7

Replying to @kiwifb:

Should we use JUPYTER_PATH for this kind of things? https://jupyter.readthedocs.io/en/latest/use/jupyter-directories.html

I don't think this can be solved by using environment variables. The issue is that the notebook may access different kernels when the kernelspec is installed, and (apparently) the notebook webserver is expected to serve the local threejs files. Now these files are kernel-specific (because they are tied to the sagelib version).

Francois, how do you currently install these files in gentoo packaging?
(Overall I think we need some input from downstream packaging to make progress here.)

[kiwifb]

comment:9

For "sagelib" I disable https://github.com/sagemath/sage-prod/blob/develop/src/sage/repl/ipython_kernel/install.py#L281 because of the way I have to install documentation following Gentoo's guidelines (not everyone in Gentoo is happy with the current way of doing this) and I do the stuff from that line manually. For the rest, sagelib's kernel install is used - it wasn't always the case.

I just change the kernel specs to use the current python directly.

I don't support any versioning of sage whatsoever. Multiple pythons are supported and work thanks to the current Gentoo python infrastructure that supports it. But stuff in /usr/share/jupyter/ has to be somewhat python independent really.

fbissey@moonloop ~ $ ll /usr/share/jupyter/kernels/sagemath/
total 20K
drwxr-xr-x 2 root root 4.0K Nov 27 12:55 .
drwxr-xr-x 4 root root 4.0K Apr 21  2020 ..
lrwxrwxrwx 1 root root   30 Nov 27 12:53 doc -> ../../../doc/sage-9999/html/en
-rw-r--r-- 1 root root  134 Nov 27 12:53 kernel.json
lrwxrwxrwx 1 root root   93 Nov 27 12:53 logo-64x64.png -> ../../../../..//usr/lib/python3.8/site-packages/sage/ext_data/notebook-ipython/logo-64x64.png
lrwxrwxrwx 1 root root   87 Nov 27 12:53 logo.svg -> ../../../../..//usr/lib/python3.8/site-packages/sage/ext_data/notebook-ipython/logo.svg
fbissey@moonloop ~ $ ll /usr/share/jupyter/nbextensions/
total 12K
drwxr-xr-x 3 root root 4.0K Nov 27 12:55 .
drwxr-xr-x 5 root root 4.0K Oct 26 21:37 ..
lrwxrwxrwx 1 root root   16 Nov 27 12:53 jsmol -> /usr/share/jsmol
drwxr-xr-x 2 root root 4.0K Nov 19 15:56 jupyter-js-widgets
lrwxrwxrwx 1 root root   18 Nov 27 12:53 mathjax -> /usr/share/mathjax
lrwxrwxrwx 1 root root   23 Nov 27 12:53 threejs -> /usr/share/sage/threejs

sage-9999 is just sage straight from the git develop branch, so right now 9.2.beta3 (but in my case, Volker's develop branch).

Users who want versioning, usually are after reproducibility. In that case system packaging is not part of the solution. containers on the other are an ideal tool for that.

[mkoeppe]

Dependencies: #30915

[mkoeppe]

comment:11

I would propose the following installation scheme in share/ and (via symlink) share/jupyter/nbextensions/:

 - threejs-sage        (renamed to indicate that it is a fork)
   - r122/             (subdirectory for the version, replaces meaningless "build")
     - three.min.js

[mkoeppe]

comment:12

I would also propose to store the version of threejs that is required by the scripts / templates in src/sage/ext_data/threejs/ in a text file in that directory.

From a packaging point of view it makes no sense to read this from the version file that is installed in THREEJS_DIR.

[mkoeppe]

Branch: u/mkoeppe/versioned_installation_of_threejs

[mkoeppe]

Commit: 938bff4

[mkoeppe]

comment:14

Here's a draft (untested)

---

Last 10 new commits:

63530d2	Update script link generation
82b8888	Update online link
55449d8	Update upstream URL
eed14d3	Fix doctest and rename more 'scripts' to 'script'
c097439	Revert threejs_scripts method name changes
c2204a4	Merge branch 'u/gh-jcamp0x2a/threejs-r122' of git://trac.sagemath.org/sage into t/30972/versioned_installation_of_threejs
1fc3d4c	build/pkgs/threejs: Install in /threejs-sage/rVERSION/
c8e707c	src/sage/repl/ipython_kernel/install.py: Change from threejs to threejs-sage
5e1ca4e	src/sage/repl/ipython_kernel/install.py, src/sage/env.py, build/pkgs/sage_conf: Change from threejs to threejs-sage
938bff4	Read required threejs version from ext_data

[sagetrac-git]

Changed commit from 938bff4 to f847738

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

f847738

src/sage/repl/rich_output/backend_ipython.py: Another change threejs -> threejs-sage

[mkoeppe]

Author: Matthias Koeppe

[kiwifb]

comment:20

I guess I can accomodate that. I already ship threejs as a sage specific component in /usr/share/sage. I have to think if I will allow several versions of threejs on the system at once.

[sagetrac-git]

Changed commit from f847738 to d13177d

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

d13177d

src/sage/repl/rich_output/display_manager.py: Fix up imports

[mkoeppe]

comment:22

This version passes the testsuite. Manual testing would be appreciated.

[jcamp0x2a]

comment:38

Replying to @paulmasson:

I strongly object to the two authors reviewing this ticket. I have yet to receive a cogent answer as to why someone would want to run two or more versions of Sage at the same time. This ticket also needs more input from distribution managers, who may not want to support this behavior. There need to be other voices clearly in favor of the direction this ticket moves, a direction I for one do not support.

Apologies. I did not realize there were outstanding questions about the use case's validity, of which I can not personally attest. Just saw that it was in review for awhile yet functioning 'correctly' (to ticket's spec).

[mkoeppe]

comment:39

Paul, if you have a specific technical objection to the change that this ticket proposes, it would probably be helpful for the discussion to try to articulate it.

[mkoeppe]

comment:40

Let's discuss what the ticket changes step by step.

1. renames directories from threejs to threejs-sage (in share/ and share/jupyter/nbextensions/) to reflect the fact that it is a Sage-specific fork

for example in this change

-THREEJS_DIR = "@prefix@/share/threejs"
+THREEJS_DIR = "@prefix@/share/threejs-sage"

Any concerns or objections to that?

[mkoeppe]

Changed branch from u/gh-jcamp0x2a/versioned_installation_of_threejs to u/mkoeppe/versioned_installation_of_threejs

[mkoeppe]

Changed dependencies from #30915 to none

[mkoeppe]

Changed reviewer from Joshua Campbell, Matthias Koeppe to Joshua Campbell, Matthias Koeppe, ...

[mkoeppe]

Changed commit from 47e795b to 0b24127

[mkoeppe]

New commits:

0b24127

Merge tag '9.3.beta6' into t/30972/versioned_installation_of_threejs

[dimpase]

comment:46

please rebase

[slel]

comment:47

Use cases for having several versions of Sage installed

• install new version & keep old one for code you lack time to adapt
• stable release for normal use, beta for development work
• keep several versions around for the sake of bisecting
  breaking changes
• bare version and version with many optional packages installed

[sagetrac-git]

Changed commit from 0b24127 to 935dcef

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

935dcef

Merge tag '9.4.beta3' into t/30972/versioned_installation_of_threejs

[dimpase]

comment:49

Replying to @paulmasson:

Replying to @mkoeppe:

Replying to @paulmasson:

Replying to @mkoeppe:

That's not a solution because a user may want to use two different Sage versions at the same time.

Why exactly would someone want to do this?

Users have complex needs.

Considering that I don't even use Sage, I'll need a better answer than this before devoting more time to this issue. We are not required to support every feature desired by end users. We can simply tell them to run only one kernel at a time. Who is making the decision to support this option?

for instance one can have two copies of Sage built with different compilers. At least for Sage development purposes it's very normal to have several installations of Sage on a box.

[dimpase]

comment:50

How does one test this with, say, jupyterlab installed by Homebrew?

[mkoeppe]

comment:51

Replying to @dimpase:

How does one test this with, say, jupyterlab installed by Homebrew?

You would follow the instructions in our manual added in #30476
(https://doc.sagemath.org/html/en/installation/launching.html#setting-up-sagemath-as-a-jupyter-kernel-in-an-existing-jupyter-notebook-or-jupyterlab-installation)

[dimpase]

comment:52

Unfortunately these instructions are outdated: on macOS I get

% jupyter kernelspec install --user `pwd`/local/share/jupyter/kernels/sagemath                             
Traceback (most recent call last):
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/bin/jupyter-kernelspec", line 33, in <module>
    sys.exit(load_entry_point('jupyter-client==6.1.12', 'console_scripts', 'jupyter-kernelspec')())
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages/jupyter_client/kernelspecapp.py", line 266, in start
    return self.subapp.start()
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages/jupyter_client/kernelspecapp.py", line 132, in start
    self.kernel_spec_manager.install_kernel_spec(self.sourcedir,
  File "/usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages/jupyter_client/kernelspec.py", line 340, in install_kernel_spec
    shutil.copytree(source_dir, destination)
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 557, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 513, in _copytree
    raise Error(errors)
shutil.Error: [('/Users/dima/sage/local/share/jupyter/kernels/sagemath/doc', '/Users/dima/Library/Jupyter/kernels/sagemath/doc', "[Errno 2] No such file or directory: '/Users/dima/sage/local/share/jupyter/kernels/sagemath/doc'")]

[mkoeppe]

comment:53

Looks to me like someone forgot to build the documentation

[dimpase]

Changed reviewer from Joshua Campbell, Matthias Koeppe, ... to Joshua Campbell, Matthias Koeppe, Dima Pasechnik

[dimpase]

comment:54

OK, this works, sorry for noise.

[mkoeppe]

comment:55

Thanks!

[vbraun]

Changed branch from u/mkoeppe/versioned_installation_of_threejs to 935dcef