build/pkgs/sagelib: Invoke install cleaner explicitly, remove it from pkgs/sagemath-standard/setup.py

[mkoeppe]

Unless ./configure --enable-editable is used, the Sage library is installed directly into site-packages using setup.py install, and at the end of sage_setup's custom sage_install_and_clean invokes the installation cleaner, which removes "stale" installed files.

The installation cleaner is problematic when we start to install optional modules in the sage.* namespace via separate distributions such as sagemath-tdlib (#29864).

• pkgs/sagemath-standard/setup.py will no longer enable optional Cython modules based on which optional packages are installed
• but the installation cleaner needs to know about these optional Cython modules so it does not remove them right after their installation.

In this ticket we solve this problem by removing the use of the installation cleaner from pkgs/sagemath-standard/setup.py. Instead we invoke it separately at the end of the installation script build/pkgs/sagelib/spkg-install.

(An alternative solution, replacing this installation procedure by a standard installation procedure using wheels, as proposed in #32874, would make sage -b much slower. Also, the idea of just making ./configure --enable-editable the default, #32406, has met skepticism.)

Depends on #32874

CC: @kiwifb @jhpalmieri

Component: build

Work Issues: Do not build_ext

Branch/Commit: u/mkoeppe/pkgs_sagemath_standard_setup_py__move_distributions_logic_to_a_new_package_sagemath_optional @ edd0ea1

Reviewer: Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/32927

[mkoeppe]

Dependencies: #31386, #32899

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Branch: u/mkoeppe/pkgs_sagemath_standard_setup_py__move_distributions_logic_to_a_new_package_sagemath_optional

[mkoeppe]

Commit: 00d5f59

[mkoeppe]

Last 10 new commits:

77c957f	src/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher
9375ef8	pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
c6fc111	Prettier diagram
7d6a44c	Use :mod: as markup for packages/modules
f83d424	Improve ABC example
817a8d0	src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup
b555735	src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata
81e9c9a	Merge #32899
ecc0886	Merge #31386
00d5f59	build/pkgs/sagelib: Invoke install cleaner explicitly, remove it from pkgs/sagemath-standard/setup.py

[mkoeppe]

comment:9

Stalled in needs_review or needs_info; likely won't make it into Sage 9.5.

[sagetrac-git]

Changed commit from 00d5f59 to 2d08b69

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

2d08b69

Merge tag '9.5.rc3' into t/32927/pkgs_sagemath_standard_setup_py__move_distributions_logic_to_a_new_package_sagemath_optional

[mkoeppe]

Changed dependencies from #31386, #32899 to none

[sagetrac-git]

Changed commit from 2d08b69 to fad8104

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

fad8104

build/pkgs/sagelib/install-cleaner/setup.py: Remove primecount

[kiwifb]

comment:14

Is this installing a nested package? The kind of stuff that we finally ditched with R/rpy a few years ago.

[mkoeppe]

comment:15

No, the distribution in build/pkgs/sagelib/install-cleaner/ does not install anything and is not nested in anything. If it's clearer, I can also move it to pkgs/install-cleaner/

[kiwifb]

comment:16

I think that may be best. But I must say so far I am a bit confused. I do not see where that "install-cleaner" is ever used. Is it just functionality you are keeping in reserve for now or for a dependent ticket?

[kiwifb]

comment:23

Replying to @mkoeppe:

It's invoked by build/pkgs/sagelib/spkg-install

Oh yes I see. Like it says, that's not a bit I am usually concerned with downstream.

[mkoeppe]

comment:24

Yes, the only effect of this ticket on downstream packaging is that sagemath-standard's setup.py no longer does weird things with site-packages.

[kiwifb]

comment:25

If I was doing a bit of archeology on the sage-on-gentoo repo I think I would find bits where I suppressed some of that clean up. I seem to have let go of that part recently.

[mkoeppe]

Dependencies: #33855

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

351cccd	build/pkgs/sagelib: Invoke install cleaner explicitly, remove it from pkgs/sagemath-standard/setup.py
731e813	build/pkgs/sagelib/install-cleaner/setup.py: Remove primecount
7667836	pkgs/sage-legacy-install-cleaner: Move install cleaner here

[sagetrac-git]

Changed commit from 1db674d to 7667836

[mkoeppe]

Changed dependencies from #33855 to none

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e26da73

build/pkgs/sagelib/spkg-install: Use --build-base for cleaning

[sagetrac-git]

Changed commit from 7667836 to e26da73

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ed680c5	build/pkgs/sagelib: Invoke install cleaner explicitly, remove it from pkgs/sagemath-standard/setup.py
57044b6	build/pkgs/sagelib/install-cleaner/setup.py: Remove primecount
fc34e2a	pkgs/sage-legacy-install-cleaner: Move install cleaner here
67d3940	build/pkgs/sagelib/spkg-install: Use --build-base for cleaning
3fcce86	pkgs/sage-legacy-install-cleaner/setup.py: Remove use of sage_build
e802edd	build/pkgs/sagelib/spkg-install: Fix up
8bfdaee	pkgs/sage-legacy-install-cleaner/setup.py: Restore multiprocessing fix
edd0ea1	pkgs/sage-legacy-install-cleaner/setup.py: Update import

[sagetrac-git]

Changed commit from e26da73 to edd0ea1

[mkoeppe]

Changed work issues from FIXME in build/pkgs/sagelib/spkg-install to Do not build_ext

[mkoeppe]

comment:34

Not needed with #32874

[mkoeppe]

Dependencies: #32874

[mkoeppe]

Changed author from Matthias Koeppe to none

[kwankyu]

comment:35

Before we close this ticket, to clarify why this approach was abandoned, let me ask a couple of questions:

Would setup install work when optional distributions that contain namespace packages were installed?

Would legacy-install-cleaner work when optional distributions that contain namespace packages were installed?

If answers to both are yes, then the branch of this ticket could be an alternative to #32874?

[mkoeppe]

comment:36

Well, the branch here would have been a temporary solution, good enough until external packages arrive that fill the namespace sage.* and are maintained out of tree. Then #30152 would have been the next step, but I did not have an implementation strategy for it.

[kwankyu]

comment:37

Okay. Thanks.

[kwankyu]

Reviewer: Kwankyu Lee