Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

[mkoeppe]

Testing of source and binary distributions relies too much on manual testing by people.

We propose to set up Dockerfiles and CI scripts that document and test the expected capabilities of source and binary Sage distributions; and of downstream distribution packaging.

Testing of source trees and source distributions: (in progress)

With the important changes brought by #27330 (Meta-ticket: spkg-configure: Try to use as many system packages as possible), sage-the-distribution has become less monolithic; it now interacts in a more complex way with distribution packaging. We propose to add infrastructure for testing correct installation of sage source distributions on a variety of platforms.

1. We collect information about distribution packages systematically and store it in on a per-SPKG basis in build/pkgs/SPKG/distros/. (Right now this information is scattered - in downstream sage distribution packagers' build scripts, trac tickets, replies to bug reports in google groups, the sage installation manual, personal knowledge...) This is part of Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add tox.ini #29053, for debian/fedora/arch/conda (with follow-up tickets tox.ini, build/bin/write-dockerfile.sh: Add gentoo linux, add more gentoo packages #29105, Add cygwin package information #29106 for other platforms).

2. We generate well-defined test environments in the form of Dockerfiles. This makes it possible to test, on one's development computer, the correct installation of sage-the-distribution on a variety of platforms and configurations. Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add tox.ini #29053 provides build/bin/write-dockerfile.sh that generates the Dockerfile using the information in 1. The minimal configuration has just the packages that are needed for a build to succeed. The standard configuration installs all distribution packages that sage knows how to use.

3. Running tests on many test environments is automated using tox in Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add tox.ini #29053. The top-level tox.ini file defines the test environments. Using tox's factor conditions (https://tox.readthedocs.io/en/latest/config.html#complex-factor-conditions), there is a succinct description of many test environments. For example, we can run tox -e docker-debian-stretch-minimal,docker-arch-latest-standard.

4. Add GitHub Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin) #29087 automates running the tox tests for a selection of 34 platforms/configurations using a GitHub Actions workflow on every git push to a GitHub repository. An example run: https://github.com/mkoeppe/sage/actions/runs/32812271

Tickets:

• Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add tox.ini #29053: Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add tox.ini --- this helps test that all spkg-configure.m4 scripts work as expected
• Add Dockerfile that sets up sage prerequisites on Arch Linux #29066: Add Dockerfile that sets up sage prerequisites on Arch Linux
• Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages #29104: Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local)
• tox.ini update: Fix breakage caused by src/ext move, add 32-bit linux environments and other architectures supported by docker #29143: Add to tox.ini 32-bit linux environments and other architectures supported by docker
• Add more system packages to build/pkgs/SPKG/distros/ #29129: Add more system packages to build/pkgs/SPKG/distros/
• add system packages info for R and others #29273: add system packages info for R and others
• Add cygwin package information #29106: Add cygwin package information
• Add GitHub CI workflow for cygwin #29295: Add GitHub CI workflow for cygwin
• Add GitHub Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin) #29087: Add GitHub Actions workflow for testing spkg-configure / build on various Linux distributions
• tox.ini: Add tests for python2 #29367: tox.ini: Add tests for python2
• tox.ini: Add environment local-conda #29415: tox.ini: Add environment local-conda
• tox.ini, GitHub CI workflow - minor fixes, more homebrew configuration factors, improve docker interface #29417: tox.ini, GitHub CI workflow - minor fixes
• Add documentation of tox and GitHub actions workflow to developer's manual #29401: Add documentation of tox and GitHub actions workflow to developer's manual
• archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages #29366: archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages
• GitHub CI workflow: Add testing of optional packages #29341: GitHub CI workflow: Add testing of optional packages
• ci-cygwin.yml: Multi-stage build #29403: ci-cygwin.yml: Multi-stage build
• Make Docker images generated by GitHub CI workflows available; minor CI fixes #29530: Make Docker images generated by GitHub CI workflows available; minor CI fixes
• For Sage 9.2: Remove Python 2 support from the build system and CI scripts #29669: For Sage 9.2: Remove Python 2 support from the build system and CI scripts
• .github/workflows/tox-gcc_spkg.yml: Remove python2 #29757: .github/workflows/tox-gcc_spkg.yml: Remove python2
• .github/workflows/: Test all optional packages, add workflow for testing all experimental packages #29901: .github/workflows/: Test all optional packages, add workflow for testing all experimental packages
• .github/workflows/tox-gcc_spkg.yml: Disable tests on some platforms that fail --without-system-gcc #29815: .github/workflows/tox-gcc_spkg.yml: Disable tests on some platforms that fail --without-system-gcc
• tox.ini, build/bin/write-dockerfile.sh: Add gentoo linux, add more gentoo packages #29105: Add gentoo package information, extend dockerfile generator to gentoo
• tox.ini: debian-bullseye, -sid have Python3.8 instead of 3.7 #29851: tox.ini: debian-bullseye, -sid have Python3.8 instead of 3.7
• Fix tox-docker builds broken by #29884 #30064: Fix tox-docker builds broken by src/doc/bootstrap: Generate src/doc/en/reference/repl/options.txt #29884
• tox.ini: Add raspbian-buster #29091: tox.ini: Add raspbian-buster
• tox.ini: Various CI fixes #29929: tox.ini: Various CI fixes
• tox.ini: Add centos-6, docker-nobootstrap, manylinux-* #30195: tox.ini: Add centos-6, docker-nobootstrap, manylinux-*
• tox.ini: Test on voidlinux #30044: tox.ini: Test on voidlinux
• tox.ini / tox.yml: Move code for tagging/pushing docker images into tox.ini #30144: tox.ini / tox.yml: Move code for tagging/pushing docker images into tox.ini
• Add nix package information, extend dockerfile generator to nix #29130: Add nix package information, extend dockerfile generator to nix
• GH Actions: Test Windows with WSL #30216 GH Actions: Test Windows with WSL
• tox.ini: Add environments local-sudo-ubuntu-standard, etc. #30923: tox.ini: Add environments local-sudo-ubuntu-standard, etc.
• tox.ini: Add opensuse, add/update opensuse.txt across build/pkgs/ #30619: update opensuse.txt across build/pkgs/
• GH Actions: cygwin: Use incremental archives for the sage-local artifact #30154: cygwin: CI fixes
• Add SAGE_ROOT/environment.yml for "conda env create -f". #28745: Add SAGE_ROOT/environment.yml for "conda env create -f".
• GH Actions: Test optional/experimental packages with Xcode 12 #30726: GH Actions: Test optional/experimental packages with Xcode 12
• GH Actions: Add docker-opensuse #29527: tox.ini: Add docker-opensuse
• tox.ini: Add scientificlinux #30213: tox.ini: Add scientificlinux
• Meta-ticket: more system pkgs info for various linuxes #29356: Meta-ticket: more system pkgs info for various linuxes
• GH Actions: conda-forge tests broken #32113: GH Actions: conda-forge tests broken
• tox.ini, GH Actions: Update Linux platforms #32947: tox.ini, GH Actions: Update Linux platforms
• tox.ini: Add centos-stream-8, centos-stream-9, remove defunct centos-6 #33196: tox.ini: Add centos-stream-8, centos-stream-9, remove defunct centos-6
• tox.ini: Add -recommended package factor #33296: tox.ini: Add -recommended package factor
• GH Actions (macOS): Fix filtering of experimental packages #33140: GH Actions (macOS): Fix filtering of experimental packages
• GH Actions workflow that builds documentation on each push #31415: GH Actions workflow that builds documentation on each push
• tox.ini: Linux platform updates #33306: tox.ini: Linux platform updates
• tox / GH Actions: Add centos-7-devtoolset-gcc_{9,10,11} #32965: tox / GH Actions: Add centos-7-devtoolset-gcc_{9,10,11}
• tox / GH Actions: Add ubuntu variants using ppa:ubuntu-toolchain-r #32966: tox / GH Actions: Add ubuntu variants using ppa:ubuntu-toolchain-r
• Add github action running on each push #33263: Add github action running on each push
• Fixes for the conda-for-Sage-developers installation method, add GH Actions workflow #30845: tox.ini, GH Actions: Add test for conda without SPKG
• GH Actions (macOS): Run a job for "make build-local" first, cache image for job "make build" #32703: GH Actions: Run a job for make build-local first, cache image for job make build
• ci-cygwin*.yml: Fix packaging up of SAGE_LOCAL/var/tmp/sage/build/* #31211: ci-cygwin*.yml: Fix packaging up of SAGE_LOCAL/var/tmp/sage/build/*
• GH Actions: Repair upload of docker images to docker.pkg.github.com #30933: GH Actions: Repair upload of docker images
• GH Actions: Add workflow with "configure --enable-editable" #31413: GH Actions: Add workflow with configure --enable-editable
• tox -e docker-...-incremental #34228: tox -e docker-...-incremental
• GH Actions (docker): Run a job for "make build-local" first, cache image for job "make build" #33062: GH Actions (docker): Run a job for "make build-local" first, cache image for job "make build"
• Filter singular system package out of tox.ini for homebrew #32784: Filter singular system package out of tox.ini for homebrew
• GH Actions: WSL workflow fails with failed syscalls #31088: GH Actions: WSL workflow: No space left on device
• tox.ini, GH Actions: Update macOS environments to macOS 11, 12 #32570: tox.ini, GH Actions: Update macOS environments
• SAGE_ROOT/tox.ini: Add variants that provision a system python using pyenv #32531: SAGE_ROOT/tox.ini: Add variants that provision a system python using pyenv
• tox.ini: Add wsl as a technology, to complement local and docker #30505: tox.ini: Add wsl as a technology, to complement local and docker
• tox.ini: Add cygwin as a technology, to complement local and docker #30924: tox.ini: Add cygwin as a technology, to complement local and docker
• Make docker images from tox/GitHub CI workflow, regular Sage Docker images, gitpod, CoCalc interoperable #29536: Make docker images from GitHub CI workflow and regular Sage Docker images interoperable
• tox.ini: Add tests with Ubuntu + deadsnakes #30217: tox.ini: Add tests with Ubuntu + deadsnakes
• test gentoo "prefix" builds on top of some Linux distribution
• tox.ini: Add local-copy #29416: tox.ini: Add local-copy
• GH Actions: Use cancel-all-previous-workflow-runs #30506: GH Actions: Use cancel-all-previous-workflow-runs
• Manage disk space constraints in continuous integration #32280: Manage disk space constraints in continuous integration
• Enable VPATH builds (several independent build trees connected to one source tree) #21469: Enable VPATH builds (several independent build trees connected to one source tree)
• tox.ini: Add test environments using LXC #29283: tox.ini: Add test environments using LXC
• Add to tox.ini a toxenv docker-linuxbrew #29159: Add to tox.ini a toxenv docker-linuxbrew
• Customize CI workflows on GitHub Actions using specially structured tag names or workflow dispatch with inputs #29535: Customize CI workflows on GitHub Actions using specially structured tag names
• Meta-ticket: CI workflows for upstream projects using Sage reusable workflows #33338 Refactor GH Actions workflows using https://docs.github.com/en/actions/learn-github-actions/reusing-workflows, also use reusable workflows for the upstream deployments of ci-sage.yml (suggested in Add integration testing with python from Debian, Ubuntu, Fedora, Gentoo, Arch Linux, OpenSUSE, Slackware pypa/setuptools#2923 (comment))

Related meta-tickets:

• Meta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh #29146: Meta-ticket: Refactor/improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh
• Meta-ticket: CI workflows for upstream projects using Sage reusable workflows #33338 Meta-ticket: CI workflows for upstream projects using Sage reusable workflows
• Meta-ticket: Remove support for outdated distros #32074 Meta-ticket: Remove support for outdated distros

Ideas without tickets so far:

• Testing that it is possible to build without error when upgrading with git from some list of previous releases.
• Test that packages can be cleanly uninstalled

---

Testing of the sage binary distributions: moved to #31133

---

Testing of downstream packaging of sage: (planning stage)

• src/bin/sage: Delegate handling of options of sage-the-distribution, specifying a subset that is supported by sagelib proper #29111: Specify a subset of sage command line options that are supported by sagelib - rather than sage-the-distribution. Test that sage -c SAGECOMMAND, sage -t, sage -python -c COMMAND, ... work with the sage script provided by the distribution (see https://groups.google.com/d/msg/sage-packaging/BmkxIBdwbvE/fRMl2sjdBQAJ)
• Add devcontainer.json for development with VS Code in a Docker container #33671 Add devcontainer.json for development with VS Code in a Docker container

---

Symptoms:

• Installing any optional package on the binary distribution triggers full recompile of sagelib #29055: Installing any optional package on the binary distribution triggers full recompile of sagelib
• Repair use of system BLAS broken by #29025: remove numpy site.cfg [DEFAULT/ALL] sections #29051: Paths configured in installed numpy site.cfg [DEFAULT/ALL] do not affect scipy

CC: @vbraun @kiwifb @isuruf @dimpase @embray @saraedum @antonio-rojas @slel @sheerluck @tobiasdiez

Component: build

Keywords: ContinuousIntegration, sd111

Issue created by migration from https://trac.sagemath.org/ticket/29060

[saraedum]

Changed keywords from none to ContinuousIntegration

[saraedum]

comment:1

I like the general idea of doing more CI a lot. Probably we should split this ticket up though or turn it into a meta ticket that tracks similar CI improvements such as #28457, #24854, #25262.

For this to be actually noticed by people we probably would need some integration with Trac. If I understood embray correctly, adding something like that, similar to the patchbot status, would not be difficult. (Since #28457 has been ready for a while, it might be a good first candidate for such an integration.)

I would propose to use GitLab CI for probably all the things you describe. We already have some CI setup there that has been mostly stable recently (though most people don't know about it https://gitlab.com/sagemath/dev/trac/pipelines?page=1&scope=all since it's not visible in trac yet.) I had made some experiments with macOS & Windows there that were not completely disappointing. For macOS you cannot use docker but still GitLab CI works, see e.g. #25980. Windows can be run inside docker (on Windows) but two years ago, the performance was not sufficient, see #25805. Without docker it worked fine though.

For our Linux CI needs, we can mostly use the free GitLab runners but last time we tried we could also easily get plenty of free credits on Google Cloud. For macOS & Windows, I had at some point built a PoC runner that slelievre volunteered to host; unfortunately, I never finished that project.

[mkoeppe]

comment:2

Replying to @saraedum:

I like the general idea of doing more CI a lot. Probably we should split this ticket up though or turn it into a meta ticket that tracks similar CI improvements such as #28457, #24854, #25262.

For this to be actually noticed by people we probably would need some integration with Trac.

Yes, that would be nice. But it would already help if this were run at the time that betas and releases are prepared. I don't think it needs to be run on every ticket.

[dimpase]

comment:12

This needs a writeup to explain how all these cogs: tox, docker, github actions, etc. fit together and move.

[mkoeppe]

comment:14

Replying to @dimpase:

This needs a writeup to explain how all these cogs: tox, docker, github actions, etc. fit together and move.

Good idea, please take a look at the revised description and let me know what needs more detail.

[dimpase]

comment:19

On some branches there are also lists of <distro> packages in build/pkgs/<distro>.txt which appear to be used for some kind of bootstrapping.
What are these?

[mkoeppe]

comment:20

build/pkgs/DISTRO.txt contains the minimal requirements for building sage from a source distribution
and build/pkgs/DISTRO-bootstrap.txt are the additional requirements so that ./bootstrap works.
In #29124 I propose to create pseudo-spkgs to capture these, instead, to make this a bit more uniform.

[mkoeppe]

comment:21

Comments on the top of build/pkgs/debian.txt explain this.

[mkoeppe]

comment:41

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

[mkoeppe]

Changed keywords from ContinuousIntegration to ContinuousIntegration, sd111

[mkoeppe]

comment:44

Moving to 9.4, as 9.3 has been released.