sagelib: Generate pyproject.toml and setup.cfg [install_requires], requirements.txt, Pipfile, and src/Pipfile #30913

mkoeppe · 2020-11-14T07:21:10Z

After #29500, it is possible to install Python packages built by the Sage distribution into user-defined venvs.

As of Sage 9.3.beta8, sagelib is published as a source distribution on PyPI (https://pypi.org/project/sagemath-standard), but the package does not have proper dependency information yet.

As part of bootstrap, we now generate (1) the pyproject.toml (PEP 517 metadata) and install_requires information, (2) requirements.txt and (3) Pipfile using m4 from the information added in #30719. These files provide 3 distinct ways of setting up a user-defined venv.

(Adding pyproject.toml does not change how the Sage distribution installs sagelib because build/pkgs/sagelib/spkg-install uses setup.py install directly.)

To test - use ./bootstrap and then configure and build the Sage distribution. Then follow the instructions in build/pkgs/sagelib/src/tox.ini:

    #
    # SUPPORTED ENVIRONMENTS:
    #
    # Build dependencies according to requirements.txt (all versions fixed).
    # Use ONLY the wheels built and stored by the Sage distribution (no PyPI):
    #
    #   ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v -e python-sagewheels-nopypi)'
    #
    python-sagewheels-nopypi,
    #
    # Build and test without using the concrete dependencies specified by requirements.txt,
    # using the dependencies declared in pyproject.toml and setup.cfg (install-requires) only:
    # Still use ONLY the wheels built and stored by the Sage distribution (no PyPI).
    #
    #   ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v -e python-sagewheels-nopypi-norequirements)'
    #
    python-sagewheels-nopypi-norequirements,
    #

Related or follow-up tickets:

./configure --enable-editable #31377 ./configure --enable-editable
Developer's Guide: Add instructions for sagelib development in a conda environment #28746 Developer's Guide: Add instructions for sagelib development in a conda environment
Meta-ticket: Make sagelib pip-installable in a conda environment #28752 Make sagelib pip-installable in a conda environment
Use pipenv to create venv for development #30371 In-place (editable) installs of sagelib in a venv
sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}" #29041 sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"
add documentation on how to set up the virtual environment, using either plain python -m venv, virtualenv (20.x), or pipenv.

References:

https://snarky.ca/what-the-heck-is-pyproject-toml/

CC: @dimpase @tobiasdiez @jhpalmieri @videlec @kiwifb @malb

Component: build

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: 04da2c6

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30913

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-11-15T01:42:15Z

Changed dependencies from #30578, #30912 to #30578, #30912, #30719

mkoeppe · 2020-11-15T03:40:51Z

Branch: u/mkoeppe/sagelib__add_setup_cfg__install_requires_

mkoeppe · 2020-11-15T03:43:04Z

Author: Matthias Koeppe

mkoeppe · 2020-11-15T03:43:04Z

Commit: b0f5cd2

mkoeppe · 2020-11-15T03:43:04Z

Last 10 new commits:

`e61929d`	`build/pkgs/sphinx/install-requires.txt: Update from gentoo`
`8fe0e35`	`build/pkgs/python_openid: Unused, remove`
`ba1d913`	`build/pkgs/itsdangerous: Unused, remove`
`bb6c4ae`	`build/pkgs/tox/install-requires.txt: New`
`d507501`	`build/pkgs/networkx/install-requires.txt: Update from gentoo`
`c3a9352`	`build/pkgs/pathpy/install-requires.txt: Package removed in #30611, remove`
`ebd4610`	`build/pkgs/pynac/install-requires.txt: New`
`6ec00dd`	`build/pkgs/{numpy,pillow}/install-requires.txt: New`
`6bff345`	`Merge branch 't/30719/add_build_pkgs_spkg_install_requires_txt_for_all_python_packages' into t/30913/sagelib__add_setup_cfg__install_requires_`
`b0f5cd2`	`bootstrap: Generate build/pkgs/sagelib/src/setup.cfg with m4`

sagetrac-git · 2020-11-15T04:21:37Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6094c68`	`build/pkgs/cypari/install-requires.txt: cypari2, not cypari`

sagetrac-git · 2020-11-15T04:21:37Z

Changed commit from b0f5cd2 to 6094c68

sagetrac-git · 2020-11-20T00:44:38Z

Changed commit from 6094c68 to 411cbfb

sagetrac-git · 2020-11-20T00:44:38Z

Branch pushed to git repo; I updated commit sha1. New commits:

`411cbfb`	`build/bin/sage-get-system-packages: Add comment`

sagetrac-git · 2020-11-20T02:07:15Z

Changed commit from 411cbfb to 3b02c3d

sagetrac-git · 2020-11-20T02:07:15Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3b02c3d`	`bootstrap: Run bootstrap scripts in build/pkgs/*/, create build/pkgs/sagelib/bootstrap`

kiwifb · 2020-11-20T19:46:22Z

comment:10

Will this be able to deal with optional package declarations (in time)?

mkoeppe · 2020-11-20T19:50:55Z

comment:11

The plan for optional packages is that users would install a package such as sagemath-meataxe or sagemath-tdlib in addition to sagemath-standard.

sagetrac-git · 2020-11-27T04:13:14Z

Branch pushed to git repo; I updated commit sha1. New commits:

`95b74e8`	`Merge tag '9.3.beta2' into t/30913/sagelib__add_setup_cfg__install_requires_`

sagetrac-git · 2020-11-27T04:13:14Z

Changed commit from 3b02c3d to 95b74e8

mkoeppe · 2020-12-06T17:43:52Z

Changed keywords from none to sd111

mkoeppe · 2020-12-06T21:21:14Z

Changed dependencies from #30578, #30912, #30719 to #30578, #30912

sagetrac-git · 2020-12-10T19:25:55Z

Changed commit from 95b74e8 to 8b8b9f9

sagetrac-git · 2020-12-10T19:25:55Z

Branch pushed to git repo; I updated commit sha1. New commits:

`99b56f5`	`build/pkgs/sagelib/src/tox.ini: Add environment python-norequirements`
`8b8b9f9`	`Merge tag '9.3.beta3' into t/30913/sagelib__add_setup_cfg__install_requires_`

mkoeppe · 2020-12-10T19:28:43Z

comment:16

Added instructions for testing to the ticket description.
In particular, this reveals the missing dependencies of upstream Python packages tracked in #30922.

sagetrac-git · 2020-12-10T19:36:37Z

Changed commit from 8b8b9f9 to 3382a47

sagetrac-git · 2020-12-10T19:36:37Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3382a47`	`build/pkgs/pplpy/install-requires.txt: Switch to git+https link`

sagetrac-git · 2020-12-12T01:01:58Z

Changed commit from 3382a47 to bf61db4

dimpase · 2021-03-24T19:32:35Z

comment:140

Processing /home/scratch2/dimpase/sage/sage/local/var/lib/sage/wheels/nbformat-5.0.7-py3-none-any.whl
Processing /home/scratch2/dimpase/sage/sage/local/var/lib/sage/wheels/widgetsnbextension-3.5.1-py2.py3-none-any.whl
INFO: pip is looking at multiple versions of ipython to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of ipykernel to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of gmpy2 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of fpylll to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cython to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cysignals to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cypari2 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cvxopt to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement jupyterlab-widgets>=1.0.0; python_version >= "3.6" (from ipywidgets)
ERROR: No matching distribution found for jupyterlab-widgets>=1.0.0; python_version >= "3.6"
ERROR: invocation failed (exit code 1)
ERROR: could not install deps [-rrequirements.txt]; v = InvocationError('/home/scratch2/dimpase/sage/sage/build/pkgs/sagelib/src/.tox/python-sagewheels-nopypi/bin/python -m pip install -rrequirements.txt', 1)
python-sagewheels-nopypi finish: getenv /home/scratch2/dimpase/sage/sage/build/pkgs/sagelib/src/.tox/python-sagewheels-nopypi after 1.87 seconds
______________________________________________________________ summary ______________________________________________________________
ERROR:   python-sagewheels-nopypi: could not install deps [-rrequirements.txt]; v = InvocationError('/home/scratch2/dimpase/sage/sage/build/pkgs/sagelib/src/.tox/python-sagewheels-nopypi/bin/python -m pip install -rrequirements.txt', 1)
cleanup /home/scratch2/dimpase/sage/sage/build/pkgs/sagelib/src/.tox/.tmp/package/1/sagemath-standard-9.3rc0.zip

no idea about that jupyterlab-widgets, we don't even seem to have such a package in Sage.

PS. ./sage --pip install jupyterlab-widgets didn't fix it.

mkoeppe · 2021-03-24T19:39:16Z

comment:141

We do have the SPKGjupyterlab_widgets. Running make jupyterlab_widgets will fix the problem.

sagetrac-git · 2021-03-24T20:01:41Z

Changed commit from 6700223 to 04da2c6

sagetrac-git · 2021-03-24T20:01:41Z

Branch pushed to git repo; I updated commit sha1. New commits:

`04da2c6`	`build/pkgs/ipywidgets: Patch out declaring install-requires of nbformat and jupyterlab-widgets`

mkoeppe · 2021-03-24T20:03:39Z

comment:143

Or try with this fix (which makes no change to how we install things in the Sage distribution, but removes excessive dependencies from the wheels)

mkoeppe · 2021-03-24T20:06:30Z

comment:144

(Newer versions of jupyterlab-widgets have more disciplined dependencies via use of jupyter_packaging; after an upgrade in the Sage 9.4 series we will be able to make jupyterlab-widgets a standard package and drop the patch to ipywidgets again.)

dimpase · 2021-03-24T20:42:17Z

comment:145

ohh, nodejs getting installed as a sage package?!

dimpase · 2021-03-24T20:53:54Z

comment:146

Replying to @mkoeppe:

We do have the SPKGjupyterlab_widgets. Running make jupyterlab_widgets will fix the problem.

it did not. trying the latest commit now

dimpase · 2021-03-24T21:47:58Z

comment:147

the patch let it go further, but now this puzzling "no space left on device"

  copying sage/sets/finite_enumerated_set.py -> build/lib.linux-x86_64-3.7/sage/sets
  copying sage/sets/set.py -> build/lib.linux-x86_64-3.7/sage/sets
  error: could not write to 'build/lib.linux-x86_64-3.7/sage/sets/set.py': No space left on device
  Building wheel for sagemath-standard (PEP 517) ... error
  ERROR: Failed building wheel for sagemath-standard
Failed to build sagemath-standard
ERROR: Could not build wheels for sagemath-standard which use PEP 517 and cannot be installed directly
Exception information:
Traceback (most recent call last):
  File "/home/scratch2/dimpase/sage/sage/build/pkgs/sagelib/src/.tox/python-sagewheels-nopypi/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/home/scratch2/dimpase/sage/sage/build/pkgs/sagelib/src/.tox/python-sagewheels-nopypi/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/home/scratch2/dimpase/sage/sage/build/pkgs/sagelib/src/.tox/python-sagewheels-nopypi/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 361, in run
    ", ".join(pep517_build_failure_names)
pip._internal.exceptions.InstallationError: Could not build wheels for sagemath-standard which use PEP 517 and cannot be installed directly
Removed build tracker: '/tmp/pip-req-tracker-rj13d2c8'
ERROR: invocation failed (exit code 1)
python-sagewheels-nopypi finish: installpkg /home/scratch2/dimpase/sage/sage/build/pkgs/sagelib/src/.tox/.tmp/package/1/sagemath-standard-9.3rc0.zip after 308.24 seconds

I really don't get is, there is almost 300Gb left on /home/scratch2/
Maybe it tries to copy stuff via some silly place like /tmp or /var or ~/.sage/, and the error comes from this? No idea.

New commits:

`04da2c6`	`build/pkgs/ipywidgets: Patch out declaring install-requires of nbformat and jupyterlab-widgets`

mkoeppe · 2021-03-24T22:17:43Z

comment:148

pip likes to use temporary directories, probably in /tmp or /var/tmp

mkoeppe · 2021-03-24T22:18:32Z

comment:149

Try setting TMPDIR to something elese

dimpase · 2021-03-24T22:32:43Z

comment:150

is TMPDIR an env.var. I can export before running this job? Or is it something pythonic?

mkoeppe · 2021-03-24T22:45:37Z

comment:151

Just the standard environment variable

dimpase · 2021-03-25T10:07:09Z

comment:152

OK, this run finishes, with some doctest errors, mainly in sphinx/docs related:

$ ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v -e python-sagewheels-nopypi)'
...
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/all.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/misc/package.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/misc/sagedoc.py  # 4 doctests failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/docs/conf.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/interfaces/gap.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage_setup/find.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage_docbuild/__init__.py  # 35 doctests failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage_docbuild/sphinxbuild.py  # 13 doctests failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage_docbuild/utils.py  # 5 doctests failed

if this is expected, feel free to make it positive review.

dimpase · 2021-03-25T10:07:09Z

Reviewer: Dima Pasechnik

mkoeppe · 2021-03-25T16:11:46Z

comment:153

Thanks for testing! Yes, this looks as expected. The documentation is not built by this invocation. We should disable the corresponding doctests, for example using #30778 - but that's something for the 9.4 series.

mkoeppe · 2021-03-29T23:54:39Z

comment:154

Setting priority to blocker to bring this ticket to the attention of the release bot.

vbraun · 2021-05-27T20:30:52Z

Changed branch from u/mkoeppe/sagelib__add_setup_cfg__install_requires_ to 04da2c6

mkoeppe added this to the sage-9.3 milestone Nov 14, 2020

mkoeppe added c: build labels Nov 14, 2020

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Nov 15, 2020

This comment has been minimized.

Sign in to view

mkoeppe added s: positive review and removed s: needs review labels Mar 25, 2021

mkoeppe added p: blocker / 1 and removed p: critical / 2 labels Mar 29, 2021

mkoeppe added p: critical / 2 and removed p: blocker / 1 labels Apr 10, 2021

mkoeppe modified the milestones: sage-9.3, sage-9.4 Apr 18, 2021

vbraun removed the s: positive review label May 27, 2021

vbraun closed this as completed in 1c01281 May 27, 2021

slel mentioned this issue May 27, 2021

Drop Python 3.6 support #30551

Closed

vbraun mentioned this issue Jun 7, 2021

Failure when testing sage_conf version #31825

Closed

sagelib: Generate pyproject.toml and setup.cfg [install_requires], requirements.txt, Pipfile, and src/Pipfile #30913

sagelib: Generate pyproject.toml and setup.cfg [install_requires], requirements.txt, Pipfile, and src/Pipfile #30913

Comments

mkoeppe commented Nov 14, 2020

mkoeppe commented Nov 15, 2020

mkoeppe commented Nov 15, 2020

mkoeppe commented Nov 15, 2020

mkoeppe commented Nov 15, 2020

This comment has been minimized.

mkoeppe commented Nov 15, 2020

sagetrac-git mannequin commented Nov 15, 2020

sagetrac-git mannequin commented Nov 15, 2020

sagetrac-git mannequin commented Nov 20, 2020

sagetrac-git mannequin commented Nov 20, 2020

sagetrac-git mannequin commented Nov 20, 2020

sagetrac-git mannequin commented Nov 20, 2020

kiwifb commented Nov 20, 2020

mkoeppe commented Nov 20, 2020

sagetrac-git mannequin commented Nov 27, 2020

sagetrac-git mannequin commented Nov 27, 2020

mkoeppe commented Dec 6, 2020

mkoeppe commented Dec 6, 2020

sagetrac-git mannequin commented Dec 10, 2020

sagetrac-git mannequin commented Dec 10, 2020

This comment has been minimized.

mkoeppe commented Dec 10, 2020

sagetrac-git mannequin commented Dec 10, 2020

sagetrac-git mannequin commented Dec 10, 2020

sagetrac-git mannequin commented Dec 12, 2020

dimpase commented Mar 24, 2021

mkoeppe commented Mar 24, 2021

sagetrac-git mannequin commented Mar 24, 2021

sagetrac-git mannequin commented Mar 24, 2021

mkoeppe commented Mar 24, 2021

mkoeppe commented Mar 24, 2021

dimpase commented Mar 24, 2021

dimpase commented Mar 24, 2021

dimpase commented Mar 24, 2021

mkoeppe commented Mar 24, 2021

mkoeppe commented Mar 24, 2021

dimpase commented Mar 24, 2021

mkoeppe commented Mar 24, 2021

dimpase commented Mar 25, 2021

dimpase commented Mar 25, 2021

mkoeppe commented Mar 25, 2021

mkoeppe commented Mar 29, 2021

vbraun commented May 27, 2021