Add SAGE_ROOT/environment.yml for "conda env create -f".

[mkoeppe]

These files describes the conda environment (conda package dependencies):

environment.yml - describes conda packages that are useful for installing sage.

This would be a more idiomatic alternative to the conda install line that is shown at the end of ./configure

In combination with the spkg-configure mechanism, this will speed up installation of Sage, in particular for users stuck on outdated OS distributions.

Open:

• use GitLab CI to actually check that sage builds with environment.yml
• scripts to support keeping package versions in sync between environment.yml, sage-the-distribution (build/pkgs), and conda-forge packaging (https://github.com/conda-forge/sage-feedstock/tree/master/recipe, https://github.com/conda-forge/sagelib-feedstock/tree/master/recipe)

See also:

• Meta-ticket: Make sagelib pip-installable in a conda environment #28752 for a more ambitious conda ticket
• Add documentation for implementing spkg-configure.m4 and distros/ for SPKGs #26668 Add documentation for implementing spkg-configure.m4 for SPKGs
• Meta-ticket: spkg-configure: Try to use as many system packages as possible #27330 Try to use as many system packages as possible

Depends on #30839
Depends on #30719
Depends on #30611

CC: @isuruf @saraedum @dimpase @embray

Component: build

Keywords: conda

Author: Isuru Fernando

Branch/Commit: 736e006

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/28745

[mkoeppe]

Branch: u/mkoeppe/worktree-sagelib-conda

[mkoeppe]

New commits:

cf6e48a

First version of environment.yml for conda

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Commit: cf6e48a

[mkoeppe]

Changed keywords from none to conda

[saraedum]

comment:3

mkoeppe: I can only see one file in the diff here on trac. Did you forget to add the other?

[mkoeppe]

comment:4

This is how far I got!

[mkoeppe]

comment:5

Any help and discussion is welcome.

[saraedum]

comment:6

First of all, I think it's great to explore ways to provide an alternative to the SPKG system. I have been wanting to use conda for this for a long time.

I have not used conda-debug a lot but I think a combination of conda debug [sagelib-recipe], conda install sage, conda remove sagelib --force should essentially do the right thing here as well.

A proper environment.yml file, as you propose, probably makes more sense though. For me the question is how we can make sure that it is kept up to date. We might want to use GitLab CI to actually check that sagelib builds with that environment.yml.

In the long run, it would be great if doctests passed, see conda-forge/sagelib-feedstock#19.

[saraedum]

comment:7

I am not sure I understand your intentions for the sage-the-distribution environment.yml. What would the workflow look like? Install that environment.yml, then force-uninstall the package that I want to work on? How would the SPKG system know which packages it needs to build now?

[mkoeppe]

comment:8

Replying to @saraedum:

I am not sure I understand your intentions for the sage-the-distribution environment.yml. What would the workflow look like? Install that environment.yml, then force-uninstall the package that I want to work on? How would the SPKG system know which packages it needs to build now?

Through the spkg-configure mechanism that was introduced recently, sage-the-distribution decides whether to install its own version of the package; but this can be overridden using configure options such as --with-system-glpk=force. (Here "system", of course, from the viewpoint of sage, includes the conda packages.) Sage developers could use the conda environment to avoid compiling many packages by default, but still work on and install sage-the-distribution packages that override the packages provided by conda.

[mkoeppe]

comment:9

Replying to @saraedum:

We might want to use GitLab CI to actually check that sagelib builds with that environment.yml.

+1

[saraedum]

comment:10

So, to make this actually useful, we'd need the corresponding configure flags to use "system" for everything that is in environment.yml as well, right?

[saraedum]

comment:11

But maybe we should split this into two tickets. One to do the sagelib environment.yml, and then this one to do the more complicated sage-the-distribution environment.yml.

[mkoeppe]

comment:13

Replying to @saraedum:

So, to make this actually useful, we'd need the corresponding configure flags to use "system" for everything that is in environment.yml as well, right?

No, the default is already to use the "system" package when it is available and deemed suitable by the configure test.
It can be changed to using the sage distribution's package by using configure --with-system-glpk=no.

[mkoeppe]

comment:14

Replying to @saraedum:

But maybe we should split this into two tickets. One to do the sagelib environment.yml, and then this one to do the more complicated sage-the-distribution environment.yml.

Good idea. I have created #28752 for that and have narrowed down the present ticket.

[mkoeppe]

comment:16

Replying to @mkoeppe:

Replying to @saraedum:

But maybe we should split this into two tickets. One to do the sagelib environment.yml, and then this one to do the more complicated sage-the-distribution environment.yml.

Good idea. I have created #28752 for that and have narrowed down the present ticket.

(Actually, the sagelib one is more complicated - see #28752!)

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

3e95cab

environment.yml: Add patch, ninja

[sagetrac-git]

Changed commit from cf6e48a to 3e95cab

[mkoeppe]

comment:18

This works reasonably well already (this is on macOS Mojave, with current miniconda 3).
I had the problem that automake and aclocal installed by conda segfault, so I bootstrapped Sage using the homebrew binaries.
Then, running configure, most packages that have spkg-configure scripts recognize the installed conda packages, with the following exceptions:

SAGE_SPKG_CONFIGURE_PERL_TERM_READLINE_GNU
SAGE_SPKG_CONFIGURE_PARI_SEADATA_SMALL
SAGE_SPKG_CONFIGURE_GF2X
SAGE_SPKG_CONFIGURE_LIBSEMIGROUPS
SAGE_SPKG_CONFIGURE_PATCH
SAGE_SPKG_CONFIGURE_ISL
SAGE_SPKG_CONFIGURE_PARI*
SAGE_SPKG_CONFIGURE_ECLIB
SAGE_SPKG_CONFIGURE_LCALC

Then, a parallel make succesfully compiled 135 packages but got stuck on ecl-16.1.2.p5, openblas-0.3.6.p0, and tachyon-0.98.9.p7.

[mkoeppe]

Changed branch from u/isuruf/environment_yaml to u/mkoeppe/environment_yaml

[mkoeppe]

New commits:

025064e

Merge tag '9.3.beta0' into t/28745/environment_yaml

[mkoeppe]

Changed commit from c252e72 to 025064e

[isuruf]

Changed commit from 025064e to 7def5b9

[isuruf]

Changed branch from u/mkoeppe/environment_yaml to u/isuruf/environment_yaml

[isuruf]

New commits:

7def5b9

Merge remote-tracking branch 'trac/develop' into u/mkoeppe/environment_yaml

[vbraun]

comment:72

Merge conflict

[isuruf]

comment:73

Which branch should I fix the conflict against? develop doesn't seem to have a conflict with this branch

[dimpase]

comment:74

Replying to @isuruf:

Which branch should I fix the conflict against? develop doesn't seem to have a conflict with this branch

it's unfortunate feauture of our workflow that this is a bit of a moving target.
https://github.com/vbraun/sage/tree/develop

do not try to merge/rebase with it, as it's likely to be in vain.

[mkoeppe]

Changed dependencies from #30839 to #30839, #30719

[mkoeppe]

Changed branch from u/isuruf/environment_yaml to u/mkoeppe/environment_yaml

[mkoeppe]

Changed commit from 7def5b9 to e53623d

[mkoeppe]

Last 10 new commits:

e61929d	build/pkgs/sphinx/install-requires.txt: Update from gentoo
8fe0e35	build/pkgs/python_openid: Unused, remove
ba1d913	build/pkgs/itsdangerous: Unused, remove
bb6c4ae	build/pkgs/tox/install-requires.txt: New
d507501	build/pkgs/networkx/install-requires.txt: Update from gentoo
c3a9352	build/pkgs/pathpy/install-requires.txt: Package removed in #30611, remove
ebd4610	build/pkgs/pynac/install-requires.txt: New
6ec00dd	build/pkgs/{numpy,pillow}/install-requires.txt: New
77cc105	Merge commit '6ec00dd002b41c4fe9b91233f2cebe37c87144d9' of git://trac.sagemath.org/sage into t/28745/environment_yaml
e53623d	build/pkgs/itsdangerous/distros/conda.txt: Remove

[mkoeppe]

Changed dependencies from #30839, #30719 to #30839, #30719, #30611

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. Last 10 new commits:

d365707	build/pkgs/packaging: Update to 20.4
248c84a	build/pkgs/pathpy: No longer required by pickleshare, remove
be5bdf2	build/pkgs/pyparsing: Update to 2.4.7
88b8427	build/pkgs/pytz: Update to 2020.4
df7972c	build/pkgs/rpy2: Update to 3.3.6
7cd96b2	build/pkgs/scipy/patches/extern_decls.patch: Remove
84d778c	Fixing upstream url for networkx
13af6c6	fix tarball name
45b3408	Merge commit '13af6c6b8c1533ba9d1b45b127e1a7b7d30000c6' of git://trac.sagemath.org/sage into t/28745/environment_yaml
736e006	build/pkgs/pathpy/distros/conda.txt: Remove

[sagetrac-git]

Changed commit from e53623d to 736e006

[vbraun]

Changed branch from u/mkoeppe/environment_yaml to 736e006