tox.ini: Update voidlinux, update installation guide #36957

mkoeppe · 2023-12-25T00:24:53Z

To test: Use export DOCKER_TARGETS="with-system-packages configured with-targets-pre with-targets", then one of:

tox -e docker-voidlinux-standard-python3.11
tox -e docker-voidlinux-musl-standard-python3.11

Support for tox -e docker-voidlinux-standard-sitepackages will have to wait for:

Python 3.12.x #36181

This

Fixes tox.ini: Fix voidlinux #33084

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

tornaria · 2023-12-27T02:51:15Z

For any void linux users out there:

$ sudo xbps-install sagemath
...

is in general quite up to date (10.2 right now). Feel free to report any issues in https://github.com/void-linux/void-packages/issues. We also have a draft PR tracking beta/rc sagemath prereleases (currently void-linux/void-packages#47712 tracking 10.3 prereleases)

Installing sagemath in any other way is unsupported. The main problem you could have is that, void being a rolling distro, the system packages will change and your locally compiled sage might break.

…commend Arch, Void downstream packages

… up-to-date versions of SageMath

mkoeppe · 2023-12-27T17:45:37Z

I'll defer other work on the installation guide to a follow up PR.

tornaria · 2023-12-31T14:17:36Z

Documentation lgtm.

As explained above I'm not really interested, but just to see how it works I tried running tox -e docker-voidlinux-standard: it failed with Error building package python3-3.11.1 (actually, it didn't fail, it hanged after printing that error, and it actually seems to be grabbing 800% cpu -- it's been trashing my cpus for fflas_ffpack tests for something like 20+ minutes).

This is futile, why are you trying to build python?

BTW, I think xbps-install -Su is incorrect, if there were any updates available it would stop to ask for confirmation. To avoid that, do xbps-install -Suy.

You are also missing a bunch of packages (of course everything required is available as packages; nothing is vendored in in our sagemath package). Most notable:

cython, cysignals, cypari
linbox, givaro, fflas_ffpack
gap
databases
(all?) python packages...
numpy, ipython, jupyter, scipy, sympy... come on.
psutil: that's funny

For the names of required packages, see: https://github.com/void-linux/void-packages/blob/master/srcpkgs/sagemath/template

mkoeppe · 2023-12-31T17:04:50Z

As explained above I'm not really interested, but just to see how it works I tried running tox -e docker-voidlinux-standard: it failed with Error building package python3-3.11.1 (actually, it didn't fail, it hanged after printing that error, and it actually seems to be grabbing 800% cpu -- it's been trashing my cpus for fflas_ffpack tests for something like 20+ minutes).

There's definitely something wrong with fflas_ffpack, we also see random hangs like you describe on other platforms in our CI. Hoping to get #35148 unstuck soon.

This is futile, why are you trying to build python?

It's probably rejecting python 3.12.

mkoeppe · 2023-12-31T17:14:58Z

You are also missing a bunch of packages (of course everything required is available as packages; nothing is vendored in in our sagemath package). Most notable:
[...]

(all?) python packages...

Yes, using system site packages is still only an experimental option @orlitzky. To test it with tox, one can use tox -e docker-voidlinux-standard-sitepackages. (But one needs a suitable system python first...)

For the names of required packages, see: https://github.com/void-linux/void-packages/blob/master/srcpkgs/sagemath/template

Thanks for the pointer! You added many of these packages in

Add more void packages to distros/void.txt #33098

I'll check what's new.

mkoeppe · 2023-12-31T18:15:02Z

#20 21.57 ## -------------------------------------------------------- ##
#20 21.57 ## Checking whether SageMath should install SPKG python3... ##
#20 21.57 ## -------------------------------------------------------- ##
#20 21.57 configure:39721: checking whether any of bzip2 liblzma libffi zlib is installed as or will be installed as SPKG
#20 21.57 configure:39731: result: no
#20 21.57 configure:39735: checking for python3 >= 3.9.0, < 3.12.0 with modules sqlite3, ctypes, math, hashlib, crypt, socket, zlib, distutils.core, ssl, ensurepip
#20 21.57 configure:39744: result: 
#20 21.57 configure:39748: checking for python3.11
#20 21.57 configure:39771: found /usr/sbin/python3.11
#20 21.57 configure:39783: result: /usr/sbin/python3.11
#20 21.57 configure:39800: checking ... whether /usr/sbin/python3.11 is good
#20 21.57 Traceback (most recent call last):
#20 21.57   File "/usr/lib/python3.11/crypt.py", line 6, in <module>
#20 21.57     import _crypt
#20 21.57 ModuleNotFoundError: No module named '_crypt'
#20 21.57 
#20 21.57 During handling of the above exception, another exception occurred:
#20 21.57 
#20 21.57 Traceback (most recent call last):
#20 21.57   File "<string>", line 1, in <module>
#20 21.57   File "/usr/lib/python3.11/crypt.py", line 11, in <module>
#20 21.57     raise ImportError("The required _crypt module was not built as part of CPython")
#20 21.57 ImportError: The required _crypt module was not built as part of CPython
#20 21.57 configure:40378: result: no, the version is in the supported range but cannot import one of the required modules: sqlite3, ctypes, math, hashlib, crypt, socket, zlib, distutils.core, ssl, ensurepip
#20 21.57 configure:40415: error: the python3 selected using --with-python=python3.11 is not suitable

mkoeppe · 2023-12-31T18:17:18Z

According to https://docs.python.org/3/library/crypt.html: Deprecated since version 3.11, will be removed in version 3.13

I'll remove the check for it

…when DOCKER_BUILDKIT=1

mkoeppe · 2023-12-31T18:31:48Z

tox is broken on voidlinux

@mkoeppe ➜ /workspaces/sage (tox_voidlinux) $ docker run -it sage-voidlinux-standard-python3.11-configured:8fdd9f0132 bash
bash-5.2# tox
Traceback (most recent call last):
  File "/usr/sbin/tox", line 33, in <module>
    sys.exit(load_entry_point('tox==3.28.0', 'console_scripts', 'tox')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/sbin/tox", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/lib/python3.12/site-packages/tox/__init__.py", line 32, in <module>
    from .session import cmdline  # isort:skip
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/tox/session/__init__.py", line 22, in <module>
    from tox.config import INTERRUPT_TIMEOUT, SUICIDE_TIMEOUT, TERMINATE_TIMEOUT, parseconfig
  File "/usr/lib/python3.12/site-packages/tox/config/__init__.py", line 22, in <module>
    import six
ModuleNotFoundError: No module named 'six'

(Also unusable because it is outdated)

tornaria · 2023-12-31T18:41:31Z

As explained above I'm not really interested, but just to see how it works I tried running tox -e docker-voidlinux-standard: it failed with Error building package python3-3.11.1 (actually, it didn't fail, it hanged after printing that error, and it actually seems to be grabbing 800% cpu -- it's been trashing my cpus for fflas_ffpack tests for something like 20+ minutes).

There's definitely something wrong with fflas_ffpack, we also see random hangs like you describe on other platforms in our CI. Hoping to get #35148 unstuck soon.

Isn't it just running a (very, very) long test suite?

Note that the new versions of givaro, fflas, and linbox are supported since 10.1 (#35612) so even without #35148 it should be possible to use them just fine.

All the upper bounds on unreleased package versions are really bad. Should just get rid of all of them.

This is futile, why are you trying to build python?

It's probably rejecting python 3.12.

For no reason. Same as above.

BTW, I changed build/pkgs/python3/spkg-configure.m4 but tox -e docker-voidlinux-standard doesn't seem to take the new version, even after I rerun bootstrap.

mkoeppe · 2023-12-31T22:29:49Z

Maybe it's distutils.core (that's deprecated!) and perhaps it would work if one adds python3-setuptools to the list of packages for python3, but each time I change something I have to download everything again so I'm getting bored....

I know -- I've done all that work already in the python 3.12 PR. It's ready, just needs review.

Cherry-picked for velocity in #36983, please review

mkoeppe · 2024-01-02T17:10:14Z

@tornaria Let's merge it?

mkoeppe · 2024-01-02T17:19:42Z

Thanks!

mkoeppe · 2024-01-02T18:03:53Z

By the way, I had trouble to get voidlinux to work in a devcontainer. I don't know who the target user base of voidlinux is, but I think that's a showstopper for adoption by developers who use VS Code or want to use Codespaces or anything like that.

tornaria · 2024-01-02T18:20:52Z

By the way, I had trouble to get voidlinux to work in a devcontainer.

I have no idea what is a "devcontainer".

I don't know who the target user base of voidlinux is,

Can scratch its own itch.

but I think that's a showstopper for adoption by developers who use VS Code or want to use Codespaces or anything like that.

Some might say this is a feature. But in fact there's actually a vscode package and it's up to date, so I don't see what is the problem.

mkoeppe · 2024-01-02T19:14:15Z

By the way, I had trouble to get voidlinux to work in a devcontainer.

I have no idea what is a "devcontainer".

Since you didn't say "... and I don't want to know": https://code.visualstudio.com/docs/devcontainers/containers

but I think that's a showstopper for adoption by developers who use VS Code or want to use Codespaces or anything like that.

Some might say this is a feature. But in fact there's actually a vscode package and it's up to date, so I don't see what is the problem.

It means for Sage that -- despite the competent packaging of Sage on voidlinux -- it's hard to recommend it as a deployment option for Sage for users who have to use Windows.

tornaria · 2024-01-02T19:35:39Z

By the way, I had trouble to get voidlinux to work in a devcontainer.

I have no idea what is a "devcontainer".

Since you didn't say "... and I don't want to know": https://code.visualstudio.com/docs/devcontainers/containers

Ok, but what is the trouble you had, and what can be done on void linux side to improve your experience?

but I think that's a showstopper for adoption by developers who use VS Code or want to use Codespaces or anything like that.

Some might say this is a feature. But in fact there's actually a vscode package and it's up to date, so I don't see what is the problem.

It means for Sage that -- despite the competent packaging of Sage on voidlinux -- it's hard to recommend it as a deployment option for Sage for users who have to use Windows.

Still no idea what went wrong.

mkoeppe · 2024-01-02T19:49:37Z

what is the trouble you had, and what can be done on void linux side to improve your experience?

Let's take this to #36995

mkoeppe · 2024-01-02T20:51:42Z

@tornaria Did you say you also package the current beta? What would I need to do to install it?

tornaria · 2024-01-02T21:26:24Z

@tornaria Did you say you also package the current beta? What would I need to do to install it?

I do have a draft PR void-linux/void-packages#47712

It works perfectly fine, it's a draft only because void policy is to stick to main releases, and I wouldn't want to be updating sagemath every week anyways.

However, there are no binary packages for the beta release: you'd have to build it from source using xbps-src. Roughly speaking: clone my branch of the void repo, run xbps-src to build the sagemath package; this will create a local repo with the packages needed, and you can install from there into the running system.

If you find this could be useful, I can do it from scratch and record step by steps (this would be 3-4 easy automatic steps). I guess one could also put the repo with the beta packages built in some public place, then just configure xbps to use that repo, but I don't do this.

mkoeppe · 2024-01-02T21:30:25Z

@tornaria Did you say you also package the current beta? What would I need to do to install it?

there are no binary packages for the beta release: you'd have to build it from source using xbps-src. Roughly speaking: clone my branch of the void repo, run xbps-src to build the sagemath package; this will create a local repo with the packages needed, and you can install from there into the running system.

If you find this could be useful, I can do it from scratch and record step by steps (this would be 3-4 easy automatic steps).

Yes, that would be great, thanks!

I guess one could also put the repo with the beta packages built in some public place, then just configure xbps to use that repo, but I don't do this.

No need for this.

tornaria · 2024-01-02T22:22:15Z

Something like this should work:

git clone --depth=200 -b sagemath https://github.com/tornaria/void-packages void-sage
cd void-sage
git fetch --depth=500 https://github.com/void-linux/void-packages
git rebase FETCH_HEAD
./xbps-src binary-bootstrap
./xbps-src pkg sagemath
sudo xbps-install --repository=hostdir/binpkgs/sagemath sagemath

It's important that you rebase on top of the main repo to avoid rebuilding packages that have changed.

The pkg step you can change for ./xbps-src pkg -Q sagemath for running tests before packaging, and ./xbps-src pkg -K sagemath for running tests in long mode.

Once the package is built, all you need to share with other boxes to install the pkg should be in the local repo in hostdir/binpkgs/sagemath. The --repository option can in principle be a url; however I think for remote urls this requires signing the packages (you can always wget the files and use it as a local repo).

mkoeppe · 2024-01-03T17:08:49Z

I'll do a PR for Merge givaro/fflas_ffpack/linbox detection into one spkg-configure.m4 #34359 and then we set relaxed version constraints

Done in

build/pkgs/{givaro,fflas_ffpack,linbox}: Accept matching set of new versions #36997

mkoeppe · 2024-01-03T21:30:03Z

Something like this should work:

git clone --depth=200 -b sagemath https://github.com/tornaria/void-packages void-sage
cd void-sage
...

@tornaria Thanks! I've added this to #36995 as .devcontainer/downstream-voidlinux-tornaria in d650e13

tornaria · 2024-01-03T21:33:55Z

Something like this should work:
git clone --depth=200 -b sagemath https://github.com/tornaria/void-packages void-sage
cd void-sage
...
@tornaria Thanks! I've added this to #36995 as .devcontainer/downstream-voidlinux-tornaria in d650e13

Is this usable from the command line, or only via vscode? If so, how? Does it work with podman or only with docker?

mkoeppe · 2024-01-03T21:35:44Z

You can probably use https://github.com/devcontainers/cli but I have no experience with it

To test: Use `export DOCKER_TARGETS="with-system-packages configured with-targets-pre with-targets"`, then one of: - `tox -e docker-voidlinux-standard-python3.11` - `tox -e docker-voidlinux-musl-standard-python3.11` Support for `tox -e docker-voidlinux-standard-sitepackages` will have to wait for: - sagemath#36181 This - Fixes sagemath#33084    ### 📝 Checklist     - [ ] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: sagemath#36957 Reported by: Matthias Köppe Reviewer(s):

…ing sphinx-inline-tabs   Preview: https://deploy-preview-37056-- sagemath.netlify.app/html/en/installation/source#linux-system-package- installation Same for the SPKG pages in the reference manual. https://deploy- preview-37056-- sagemath.netlify.app/html/en/reference/spkg/gfortran#equivalent-system- packages Same for the git instructions in the developer guide, using 2 tabs for HTTPS and SSH: https://deploy-preview-37056-- sagemath.netlify.app/html/en/developer/github#forking-the-sage- repository    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36957 (merged here)  URL: sagemath#37056 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

… (setuptools instead of distutils)   - Cherry-picked from sagemath#36181 This PR does not change what versions are accepted! To test with Python 3.12, still need to adjust the version bound set in `build/pkgs/python3/spkg-configure.m4`.    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36957 (merged here to resolve merge conflict)  URL: sagemath#36983 Reported by: Matthias Köppe Reviewer(s): Gonzalo Tornaría, Matthias Köppe

mkoeppe self-assigned this Dec 25, 2023

mkoeppe added s: needs review c: scripts labels Dec 26, 2023

mkoeppe requested a review from tornaria December 26, 2023 02:16

mkoeppe marked this pull request as ready for review December 26, 2023 02:23

Matthias Koeppe added 5 commits December 26, 2023 17:28

tox.ini (voidlinux): Update to use ghcr.io images

97071f8

build/pkgs/_prereq/distros/void.txt: Add diffutils

497d7d5

build/pkgs/_prereq/distros/void.txt: Add gzip

b2f488f

build/pkgs/_prereq/distros/void.txt: Add python3-devel and prereqs

7d52f9e

src/doc/bootstrap, src/doc/en/installation: Add voidlinux package lists

ad0531d

mkoeppe force-pushed the tox_voidlinux branch from 574d229 to ad0531d Compare December 27, 2023 02:05

src/doc/en/installation: Recommend to not use Sage older than 9.5; re…

2b3874f

…commend Arch, Void downstream packages

mkoeppe changed the title ~~tox.ini: Update voidlinux~~ tox.ini: Update voidlinux, update installation guide Dec 27, 2023

mkoeppe force-pushed the tox_voidlinux branch from 58be8bb to 2b3874f Compare December 27, 2023 17:42

src/doc/en/installation/index.rst: Only some Linux distributions have…

4e959a9

… up-to-date versions of SageMath

mkoeppe added the c: documentation label Dec 28, 2023

Matthias Koeppe added 2 commits December 31, 2023 09:17

build/bin/write-dockerfile.sh (void): Use 'xbps-install -Suy'

00da950

build/pkgs/_python3.11/distros/void.txt: New

f68a4c6

Matthias Koeppe added 2 commits December 31, 2023 10:19

build/pkgs/python3/spkg-configure.m4: Do not check for _crypt

1cae3d4

tox.ini (docker): Use BUILDKIT_PROGRESS=plain to restore full output …

8fdd9f0

…when DOCKER_BUILDKIT=1

build/pkgs/_prereq/distros/void.txt: Add bash

d4f514d

This was referenced Jan 2, 2024

Prepare python3 spkg-configure.m4 for Python 3.12 (setuptools instead of distutils) #36983

Merged

Make pyproject.toml the source for build dependencies #36982

Merged

tornaria added s: positive review and removed s: needs review labels Jan 2, 2024

vbraun merged commit 3251d98 into sagemath:develop Jan 14, 2024
25 checks passed

mkoeppe added this to the sage-10.3 milestone Jan 14, 2024

github-actions bot removed the s: positive review label Jan 14, 2024

mkoeppe deleted the tox_voidlinux branch February 14, 2024 02:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tox.ini: Update voidlinux, update installation guide #36957

tox.ini: Update voidlinux, update installation guide #36957

mkoeppe commented Dec 25, 2023 •

edited

Loading

tornaria commented Dec 27, 2023

mkoeppe commented Dec 27, 2023

tornaria commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023 •

edited

Loading

tornaria commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

tornaria commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

tornaria commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

tornaria commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

tornaria commented Jan 2, 2024

mkoeppe commented Jan 3, 2024

mkoeppe commented Jan 3, 2024

tornaria commented Jan 3, 2024

mkoeppe commented Jan 3, 2024

tox.ini: Update voidlinux, update installation guide #36957

tox.ini: Update voidlinux, update installation guide #36957

Conversation

mkoeppe commented Dec 25, 2023 • edited Loading

📝 Checklist

⌛ Dependencies

tornaria commented Dec 27, 2023

mkoeppe commented Dec 27, 2023

tornaria commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023 • edited Loading

tornaria commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

tornaria commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

tornaria commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

tornaria commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

tornaria commented Jan 2, 2024

mkoeppe commented Jan 3, 2024

mkoeppe commented Jan 3, 2024

tornaria commented Jan 3, 2024

mkoeppe commented Jan 3, 2024

mkoeppe commented Dec 25, 2023 •

edited

Loading

mkoeppe commented Dec 31, 2023 •

edited

Loading