Restructure sage.*.all for modularization, replace relative by absolute imports #37900

saraedum · 2024-04-29T11:50:38Z

This PR recreates #36676 by @mkoeppe after it was reverted, see #37796.

We restructure the all.py files for modularization.

Guided by the technical constraints outlined in https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s, #35095 defines distribution packages (pip-installable packages) sagemath-{brial, combinat, eclib, flint, gap, giac, glpk, graphs, groups, homfly, lcalc, libbraiding, libecm, linbox, modules, mpmath, ntl, pari, plot, polyhedra, schemes, singular, standard-no-symbolics, symbolics}.

When a namespace package such as sage.misc is filled by modules from several distribution packages, we create modules named:

src/sage/misc/all__sagemath_environment.py
src/sage/misc/all__sagemath_objects.py
src/sage/misc/all__sagemath_repl.py

Import statements are moved from src/sage/misc/all.py to these files as appropriate, and src/sage/misc/all.py imports * from there.

Also some imports are replaced by lazy imports.

The new files provide the top level namespaces for the modularized distribution packages, thus enabling modularized testing.

This design was introduced in #29865 (merged in Jan 2022, early in the Sage 9.6 development cycle).

Moreover, applied a one-line command to replace relative by absolute imports, thus complementing #36666, which does not touch .all* files.

The changes to other files in sage.modules etc. come from the PRs #36597, #36572, #36588, #36589 merged here and do not need review.

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

Depends on Replace relative imports by absolute ones in modules #36597 (merged here to resolve merge conflict)
Depends on Replace relative imports by absolute ones in categories #36572 (merged here to resolve merge conflict)
Depends on Replace relative imports by absolute ones in rings #36588 (merged here to resolve merge conflict)
Depends on Replace relative imports by absolute ones in a few packages #36589 (merged here to resolve merge conflict)
Depends on Autogenerate flint headers #36449 (merged here to resolve merge conflict)
Depends on add simple function to count irreducible polynomials over 𝔽q #36390 (merged here to resolve merge conflict)
Depends on Relative to absolute imports, del lazy_import in .all files #38030 (split out from here)
Depends on all*.py files: Use 'del lazy_import', 'del install_doc' #38086 (split out from here)
Depends on src/sage/algebras/steenrod/all.py: Use lazy_import, remove deprecated global import #38095 (split out from here)
Depends on sage.combinat, sage.games: Use more lazy_import in all.py files #38096 (split out from here)

(cd src && for d in $(find sage -name __pycache__ -prune -o -type d -print); do sed -i.bak "/from *[.].*import/s/from /from $(echo $d | sed 's,/,.,g')/;s/[.] import / import /;" $d/all*.py; done)

…zy imports

…th#36566

…te-known-test-failures

…o if ! grep -q 'del lazy_import' $a; then echo 'del lazy_import' >> $a; fi; done

…l*.py

…*/all*.py'

…ctree

…o if ! grep -q 'del install_doc' $a; then echo 'del install_doc' >> $a; fi; done

The chain of immediate ancestors of this commit was created by cherry-picking the relevant commits that comprise sagemath#36676. Replaying all these commits (and the necessary conflict resolution) still leaves a non-empty diff when comparing 10.4.beta2 to the previous sagemath#36676 and 10.4.beta3 to that cherry-picked branch. This commit makes sure that the diff is trivial. The changes introduced here were likely introduced in conflict resolution when merging develop into sagemath#36676. I did not replay these conflict resolutions, so I do this here manually.

saraedum · 2024-04-29T11:56:12Z

Last count of votes from #36676:

👍 @mkoeppe @kwankyu @culler @jhpalmieri @kiwifb @NathanDunfield @GMS103 @roed314
👎 @tobiasdiez @dimpase @tornaria @mantepse @orlitzky

(Technically, I am the "author" of this PR but I just recreated it. I did not vote on this PR.)

SageMath version 10.4.beta5, Release Date: 2024-05-02

….all`       Cherry-picked from sagemath#36676/sagemath#37900. ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#37993 Reported by: Matthias Köppe Reviewer(s): John H. Palmieri, Matthias Köppe

…gemath#37900

….all`       Cherry-picked from sagemath#36676/sagemath#37900. ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#37993 Reported by: Matthias Köppe Reviewer(s): John H. Palmieri, Matthias Köppe

…`.all` files       - Split out from sagemath#37900 ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38030 Reported by: Matthias Köppe Reviewer(s):

…l_doc'       - Cherry-picked from sagemath#37900 Includes style fixes from there as well. ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38086 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

…t, remove deprecated global import       - The change to use `lazy_import` was cherry-picked from sagemath#37900 - The global import was deprecated in sagemath#32647 (2021) ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38095 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

…l_doc'       - Cherry-picked from sagemath#37900 Includes style fixes from there as well. ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38086 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

…t, remove deprecated global import       - The change to use `lazy_import` was cherry-picked from sagemath#37900 - The global import was deprecated in sagemath#32647 (2021) ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38095 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

…m global bindings

…gemath#37900

…t` in all.py files       This helps with the modularization, but is also desirable for reducing the start-up time of Sage. - Cherry-picked from sagemath#37900 Related: sagemath#35520 ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38096 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

…gemath#37900

as proposed in https://groups.google.com/g/sage-devel/c/ewHP20bmskQ, because (among other things): It has been dropped from Debian testing (`debian-trixie`) and `ubuntu-noble`, where it seems to block SageMath upgrades (Sage is stuck at 9.5 in Debian/Ubuntu). See https://repology.org/project/brial/versions - Doctests are already conditional on the dynamically detected feature `sage.rings.polynomial.pbori` - We create a distribution **sagemath-brial** that packages the corresponding extension modules (cherry-picked from sagemath/sage#35095). - Documentation is conditionalized by using Sphinx tags The new distribution builds on top of enlarged distributions **sagemath-objects**, **sagemath-categories**. This is a major update of these distributions, which makes them much more useful. Also includes style changes to `all*.py` files cherry-picked from sagemath/sage#37900. This PR is the same as sagemath/sage#36380

Matthias Koeppe and others added 24 commits April 14, 2024 21:56

Set up all__sage_categories.py

e4b376b

sage.*.all*: Update from sagemath#35095

a960393

relative to absolute imports

2a71fe1

(cd src && for d in $(find sage -name __pycache__ -prune -o -type d -print); do sed -i.bak "/from *[.].*import/s/from /from $(echo $d | sed 's,/,.,g')/;s/[.] import / import /;" $d/all*.py; done)

Fixups in .all*

0b7f892

Reverts

6a8f404

Restores

c791f78

src/sage/misc/dev_tools.py: Update doctest output

366035e

src/sage/misc/sageinspect.py: Update doctests for increased use of la…

a96f570

…zy imports

Defer some changes to all__sagemath_{objects,categories} until sagema…

c749f35

…th#36566

sage -fixdoctests --no-test --distribution sagemath_categories --upda…

563a3f2

…te-known-test-failures

Defer some changes to all__sagemath_repl until sagemath#36566

22395ae

src/sage/calculus/all.py: Move 'symbolic_expression' to new module .expr

7de8387

for a in $(git --no-pager grep -l lazy_import src/sage/**/all*.py); d…

e411118

…o if ! grep -q 'del lazy_import' $a; then echo 'del lazy_import' >> $a; fi; done

Manual fixes re 'del lazy_import'

7a07f69

autopep8 --in-place --ignore E402 --max-line-length=88 src/sage/**/al…

dfc6cca

…l*.py

sed -i.bak 's,http:,https:,g' src/sage/**/all*.py

a7fdfc9

Fix errors from 'sage -tox -e pycodestyle -- --ignore E402 src/sage/*…

e630e90

…*/all*.py'

src/doc/en/reference/calculus/index.rst: Add sage.calculus.expr to to…

bcc7137

…ctree

src/sage/calculus/expr.py: Move imports from method to module

80d33ee

for a in $(git --no-pager grep -l install_doc src/sage/**/all*.py); d…

49a508d

…o if ! grep -q 'del install_doc' $a; then echo 'del install_doc' >> $a; fi; done

src/sage/combinat/all.py: Also del install_dict

91ad68f

src/sage/rings/all__sagemath_categories.py: Add import of lazy_import

e9f86a0

Merge remote-tracking branch 'origin/develop' into 36676

f8cda25

saraedum added s: needs review disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ labels Apr 29, 2024

This was referenced Apr 29, 2024

Add # sage_setup: distribution directives to all files #37901

Open

Revert PR with disputed dependencies #37796

Merged

Restructure sage.*.all for modularization, replace relative by absolute imports #36676

Closed

This comment was marked as off-topic.

Sign in to view

mkoeppe added the s: needs work label May 3, 2024

Merge tag '10.4.beta5' into 36676

ac36cbc

SageMath version 10.4.beta5, Release Date: 2024-05-02

mkoeppe removed the s: needs work label May 12, 2024

github-actions bot added the v: large label May 12, 2024

Merge remote-tracking branch 'upstream/develop' into 36676

45cc4bd

mkoeppe added a commit to mkoeppe/sage that referenced this pull request May 22, 2024

Update all__sagemath_{objects,categories,repl,environment}.py from sa…

73e0feb

…gemath#37900

Merge remote-tracking branch 'upstream/develop' into 36676

0a27566

Matthias Koeppe added 4 commits June 1, 2024 16:44

sage.combinat, sage.games: Use more lazy_import in all.py files

20e6cda

src/sage/structure/parent.pyx: In doctest, use SymmetricFunctions fro…

a34bd71

…m global bindings

src/sage/categories/weyl_groups.py: Fix import

327fb5c

Merge branch 'lazy_import_combinat' into 36676

af1ff33

vbraun force-pushed the develop branch from eba5e19 to e5f42fa Compare June 3, 2024 22:15

Matthias Koeppe added 2 commits June 9, 2024 11:01

Merge remote-tracking branch 'upstream/develop' into 36676

fd5ba9a

src/sage/matrix/args.pyx: late-import MatrixSpace in all code paths

8865085

mkoeppe added a commit to mkoeppe/sage that referenced this pull request Jun 11, 2024

Update all__sagemath_{objects,categories,repl,environment}.py from sa…

d8f6e21

…gemath#37900

mkoeppe added a commit to mkoeppe/sage that referenced this pull request Jun 22, 2024

Update all__sagemath_{objects,categories,repl,environment}.py from sa…

60ea218

…gemath#37900

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

saraedum commented Apr 29, 2024 •

edited by mkoeppe

Loading

saraedum commented Apr 29, 2024

This comment was marked as off-topic.

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

Are you sure you want to change the base?

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

Conversation

saraedum commented Apr 29, 2024 • edited by mkoeppe Loading

📝 Checklist

⌛ Dependencies

saraedum commented Apr 29, 2024

This comment was marked as off-topic.

saraedum commented Apr 29, 2024 •

edited by mkoeppe

Loading