Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use system python packages with --enable-system-site-packages #36141

Merged
merged 253 commits into from
Sep 10, 2023

Conversation

orlitzky
Copy link
Contributor

@orlitzky orlitzky commented Aug 26, 2023

Rebased branch of #29665

Platform-specific testing:

  • for example TARGETS_PRE="build-local" tox -e docker-ubuntu-focal-standard-sitepackages -- build ptest

Fixes #29665

Matthias Koeppe and others added 30 commits August 21, 2023 20:10
…nfigure.

This new option is experimental and disabled by default. When it is
enabled, the string "--system-site-packages" is appended to a new
environment variable SAGE_VENV_FLAGS that is then passed to the
sage-venv command during the build using autoconf substitution.
Thusly the correct setting finds its way into local/pyvenv.cfg.
This new macro encapsulates the check for system installations of
python packages. It uses setuptools.version's pkg_resources to
check if the contents of install-requires.txt are satisfied for
the given package; and if not, tells the build system to install
the SPKG instead.
This includes a SAGE_SPKG_DEPCHECK([gsl glpk suitesparse],... because
those C dependencies wind up linked into shared libraries like,

  /usr/.../cvxopt/glpk.cpython-39-x86_64-linux-gnu.so

that could potentially conflict with (for example) the glpk SPKG.
This includes a SAGE_SPKG_DEPCHECK([zeromq],... because libzmq winds
up linked into shared libraries like,

  /usr/.../zmq/backend/cython/message.cpython-39-x86_64-linux-gnu.so

that potentially conflict with the zeromq SPKG.
This includes a SAGE_SPKG_DEPCHECK([openblas],... because blas/lapack
wind up linked into shared libraries like,

  /usr/.../numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so

that potentially conflict with the openblas SPKG.
This includes a SAGE_SPKG_DEPCHECK([openblas],... because blas/lapack
wind up linked into shared libraries like,

  /usr/.../scipy/linalg/_flinalg.cpython-39-x86_64-linux-gnu.so

that potentially conflict with the openblas SPKG.
This includes a SAGE_SPKG_DEPCHECK([bzip2 freetype libpng qhull],...
because those dependencies wind up linked into shared libraries like,

  /usr/lib/.../matplotlib/ft2font.cpython-39-x86_64-linux-gnu.so

that potentially conflict with their SPKG counterparts.
@orlitzky
Copy link
Contributor Author

orlitzky commented Sep 1, 2023

it's a Python bug that the minor version is not available from pkg-config, IMHO

At least get_python_version() is documented to output something usable. You can rely on $PATH to execute it. There are far far worse situations.

Anyway, I pushed the cut change. Everything else LGTM. Thanks for the review and the help. This should keep me busy for a while.

@vbraun
Copy link
Member

vbraun commented Sep 1, 2023

merge conflict with #36112

@mkoeppe
Copy link
Contributor

mkoeppe commented Sep 1, 2023

Merged #36112

@mkoeppe
Copy link
Contributor

mkoeppe commented Sep 1, 2023

@mkoeppe
Copy link
Contributor

mkoeppe commented Sep 2, 2023

debian-bullseye (https://github.com/mkoeppe/sage/actions/runs/6054949711/job/16433159643#logs)

[categorie] updating environment: [new config] 223 added, 0 changed, 0 removed
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/sage/src/sage/repl/ipython_kernel/__main__.py", line 1, in <module>
    from ipykernel.kernelapp import IPKernelApp
  File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 33, in <module>
    from traitlets.utils import filefind
ImportError: cannot import name 'filefind' from 'traitlets.utils' (/usr/lib/python3/dist-packages/traitlets/utils/__init__.py)
[plot3d   ] Extension error:
[plot3d   ] Notebook execution failed (exception: Kernel died before replying to kernel_info)

@github-actions
Copy link

github-actions bot commented Sep 2, 2023

Documentation preview for this PR (built with commit de831a5; changes) is ready! 🎉

@mkoeppe
Copy link
Contributor

mkoeppe commented Sep 2, 2023

fedora-33:

pkg_resources.ContextualVersionConflict: (sphinxcontrib-htmlhelp 1.0.3 (/usr/lib/python3.9/site-packages), Requirement.parse('sphinxcontrib-htmlhelp>=2.0.0'), {'sphinx'})

@dimpase
Copy link
Member

dimpase commented Sep 2, 2023 via email

mkoeppe added a commit to mkoeppe/sage that referenced this pull request Sep 2, 2023
@mkoeppe
Copy link
Contributor

mkoeppe commented Sep 2, 2023

Once more I'll explain that old Fedora is proxy for RHEL.

@dimpase
Copy link
Member

dimpase commented Sep 2, 2023

We are not married to RHEL, and a proxy is just a proxy. It is very broken.

To Hell with RHEL, if they are not willing to provide any useful CI.

vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 4, 2023
…e-packages

    
Rebased branch of sagemath#29665

Platform-specific testing:
- for example TARGETS_PRE="build-local" tox -e docker-ubuntu-focal-
standard-sitepackages -- build ptest

Fixes sagemath#29665
    
URL: sagemath#36141
Reported by: Michael Orlitzky
Reviewer(s): Dima Pasechnik, Matthias Köppe, Michael Orlitzky
vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 5, 2023
…e-packages

    
Rebased branch of sagemath#29665

Platform-specific testing:
- for example TARGETS_PRE="build-local" tox -e docker-ubuntu-focal-
standard-sitepackages -- build ptest

Fixes sagemath#29665
    
URL: sagemath#36141
Reported by: Michael Orlitzky
Reviewer(s): Dima Pasechnik, Matthias Köppe, Michael Orlitzky
@vbraun vbraun merged commit 7bb1e6b into sagemath:develop Sep 10, 2023
8 of 9 checks passed
@mkoeppe mkoeppe added this to the sage-10.2 milestone Sep 10, 2023
@tornaria tornaria mentioned this pull request Sep 10, 2023
3 tasks
vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 11, 2023
    
This fixes a doctest with newer versions of debugpy, as seen in in
sagemath#36141 and
sagemath#36129. Explanation in the commit
message.
    
URL: sagemath#36218
Reported by: Michael Orlitzky
Reviewer(s): Matthias Köppe
vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 14, 2023
    
This fixes a doctest with newer versions of debugpy, as seen in in
sagemath#36141 and
sagemath#36129. Explanation in the commit
message.
    
URL: sagemath#36218
Reported by: Michael Orlitzky
Reviewer(s): Matthias Köppe
vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 16, 2023
    
This fixes a doctest with newer versions of debugpy, as seen in in
sagemath#36141 and
sagemath#36129. Explanation in the commit
message.
    
URL: sagemath#36218
Reported by: Michael Orlitzky
Reviewer(s): Matthias Köppe
vbraun pushed a commit to vbraun/sage that referenced this pull request Sep 16, 2023
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
After sagemath#36141, pyproject.toml and setup.cfg files are no longer generated
correctly.
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36240
Reported by: Matthias Köppe
Reviewer(s): Michael Orlitzky
@orlitzky orlitzky deleted the system-python-packages branch September 22, 2023 00:16
vbraun pushed a commit to vbraun/sage that referenced this pull request Oct 12, 2023
…ironment files

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
- Environment files are now versioned with the Python minor version, as
proposed in (and cherry-picked from) sagemath#35986 by @tobiasdiez
- CI for conda now uses `--enable-system-site-packages` from sagemath#36141 by
@orlitzky

Split out from

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36405
Reported by: Matthias Köppe
Reviewer(s): Tobias Diez
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

configure --enable-system-site-packages via spkg-configure.m4 for cvxopt and other python packages
4 participants