spkg-configure.m4 for most external Python pkgs #36777
Conversation
|
I don't want to install all that crap to test but the new |
|
@dimpase If you are doing this exercise as an approach for using the system Jupyter notebook / JupyterLab, I'd suggest to consider a different approach. The front-end does not need to run in the same Python instance as the kernel running the Sage library. |
|
Is the last comment meant to derail the effort to unvendor a bunch of packages? |
|
the frontend certainly does not need to run on the same Python, that is by the design of Jupyter. I don't quite know why in the 1st place it was decided to vendor the whole Jupyter in Sage, probably as a replacement of sagenb, which actually was at some point made a separate package with a separate git repo. So we are repeating the whole Jupyter packaging effort here 😭 |
|
SPKGs don't come with any metadata about how they're supposed to work with python, and the sage package manager doesn't do dependency resolution, so to do it right, the people interested in not using these packages would have to maintain the list by hand (like we already do with |
|
That's why people interested in sage-the-distro should get a separate repo where they can have their own fun with 400+ packages with built from source compilers, jupyter, whatever. I just want sagelib as a Python package, without any toolchain or frontend, scipy-style. Such a Python package can very well be installed in sage-the-distro. Or they can vendor it... |
Glad we can agree on this! The next step is to understand that this means that you can treat it like system packages providing executables instead of a Python site packages. The model of |
The problem with this is that there's no way to know what dependencies are unique to tox except to muck around in |
I have a robust solution for this in #36766 |
I'm surprised about this example. How would you describe scipy's approach to vendoring? |
|
scipy does not vendor compilers (or jupyter). It assumes that anyone wanting to build it from source has all the needed tools installed: python, cython, fortran, C, C++, etc... It surely provides instructions for all this. |
|
Is there something else about vendoring that you may be forgetting? |
|
Like? |
|
Libraries? |
|
sure they e.g. ship libgfortran in their binary wheels. So? |
|
Other libraries? |
|
we are talking about "toolchain", not maths stuff that needs to be delivered to the end user one way or another. yes, they vendor about a dozen things. |
i don't think we are.
They do! And in fact they even do that in the real meaning of "vendoring", namely including upstream source code in their source code distribution. So when you were saying ....
... do you mean this is what you would sagelib to look like? |
|
There are better examples than scipy I'm sure, but now that scipy has a meson build system the bundling will get resolved eventually: scipy/scipy#17751 For obscure math subprojects there's always a chicken and egg problem trying to convince upstream to unbundle things that no distro packages it because everyone bundles it because no distro packages it...
Aren't you still manually curating the jupyter package list? In any case, the question isn't whether or not it can be made robust, but whether or not it's a waste of time. |
Thanks for the pointer! |
|
anyhow, they don't vendor python, openblas, numpy. They have few header-only C++ libs as git submodules (boost/math), an LP solver Higs. They don't have 100 packages per release to update. Our technology for not vendoring is actually better, we can e.g. skip boost. |
The Sage library does not vendor anything at all. Edit: @dimpase replied by editing: I count shipping a tarfile along, and installing it as vendoring. It's merely a different wrapping of the same thing, we could have just as well checked that code into our git tree. |
|
So, Dima, once again: when you were saying ....
What is it that you mean? |
|
I use scipy as an example as I know it a bit. I got my hands dirty fixing few things for them (a side effect is that I ended up as an official contributor, in a paper, and Scopus counts about 15000 citations to this paper. Beancounters are happy :-)) |
now that's news to me ;) |
|
How come? What's the difference between having the package X untarred in advance, or untarred on the fly, if your building process builds/installs X? |
|
Matthias, years before you started actively contributing to Sage, we referred to creating spkg-configure.m4 file for an spkg X as unvendoring of X. |
|
Not that this matters, but no, Dima, the historical record does not agree with this: https://github.com/sagemath/sage/issues?q=unvendor |
dimpase
force-pushed
the
nb_7_packages_m4
branch
from
a41514e
to
5c3c332
Compare
No, that's a false claim. Please look again at the repos I've linked above and you will see that they the packages there are completely independent and self contained. In particular, the source code is living in the package and is not symlinked into it. |
That's perhaps a bit harsh for this discussion? |
|
When a project is making changes, it is important to take the developer community with you. One such route from monolithic Sage to multirepo Sage (if the community decides to do this) would go like this:
I have been working since 2020 on step 1 in #35095. This is where the real work of modularization is happening; and there's still nontrivial work to be done; and during this time, I do not have to bother other developers with the added complexity of changing source code locations. The other steps are trivial. |
|
This is largely orthogonal to the need to spin off Sage the distro into a number of separate repos, one or more. |
|
Sage the distro does not even consist of much code at all; but it provides the package metadata (versions in install-requires) that are needed for bootstrapping the pkgs/ subdirectories. |
|
If you count the code which ends up installed by the spkg-install and similar scripts, it is a lot of code (e.g. gcc/gfortran are a lot of code) |
|
Would build/pkgs still bother you if everything except for the patches was just one big yaml file? |
|
do you want to replace the tarballs in |
dimpase
added
the
disputed
Update: they are not going to be in 2.2.0a2 - aleaxit/gmpy#462 (comment) but there is an interesting development regarding an openblas wheel - aleaxit/gmpy#462 (comment) |
|
incidentally we can also reuse the related to get rid of gcc/gfortran |
…sions of the Sage distribution
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
Discussions of the complexity of the Sage distribution pop up
unexpectedly, as seen in
sagemath#36982 (comment),
sagemath#36982 (comment),
sagemath#36982 (comment),
sagemath#36982 (comment),
sagemath#36982 (comment),
sagemath#36982 (comment),
sagemath#36982 (comment)
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment),
sagemath#36726 (comment)
sagemath#36777 (comment),
sagemath#36777 (comment),
sagemath#36777 (comment),
sagemath#36777 (comment),
sagemath#36777 (comment),
sagemath#36777 (comment)
To help put such discussions on a solid factual basis, we introduce the
command `sage --package metrics`.
```
$ ./sage -package metrics :standard:
has_file_distros_arch_txt=131
has_file_distros_conda_txt=216
has_file_distros_debian_txt=125
has_file_distros_fedora_txt=138
has_file_distros_gentoo_txt=181
has_file_distros_homebrew_txt=61
has_file_distros_macports_txt=129
has_file_distros_nix_txt=51
has_file_distros_opensuse_txt=146
has_file_distros_slackware_txt=25
has_file_distros_void_txt=184
has_file_patches=35
has_file_spkg_check=59
has_file_spkg_configure_m4=222
has_file_spkg_install=198
has_tarball_upstream_url=231
line_count_file_patches=22561
line_count_file_spkg_check=402
line_count_file_spkg_configure_m4=2792
line_count_file_spkg_install=2960
packages=272
type_standard=272
```
Use `PATH=build/bin:$PATH SAGE_ROOT=some-other-worktree build/bin/sage-
package metrics :standard:` to obtain the metrics of another version of
Sage in some other worktree.
We add computation and before/after comparison of the metrics to the CI
Linux Incremental workflow.
As an illustration, we change one Python package from "normal" to
"wheel", removing an `spkg-install.in` file in the process. See https://
github.com/sagemath/sage/actions/runs/7342841283/job/19992606617?pr=3697
7#step:6:12
More metrics can be added after
- sagemath#36740
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->
### 📝 Checklist
<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->
- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.
### ⌛ Dependencies
<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
URL: sagemath#36977
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe
dimpase
added
s: needs review
and removed
s: needs info
disputed
|
Documentation preview for this PR (built with commit 016fc0e; changes) is ready! 🎉 |
After #36129 and #36776 we are still left with a bunch of external (non-Sage) Python packages
which need
spkg-configure.m4and distros info.Here we add these.
📝 Checklist
⌛ Dependencies