actions: remote testing functionality

[oliver-sanders]

Use containers in our GitHub Actions testing workflow to act as remote hosts for testing purposes.

• Apparently we don't have an issue for this!
• Closely related to Use CI to run tests for batch systems #3372

The functional test battery currently supports two types of remote host:

• remote host
• remote host with shared filesystem

And there are currently two (soon to be three) task communication methods:

• tcp (currently called zmq)
• polling (poll)
• ssh tunnelling (ssh+tcp) - zmq: implement SSH tunnelling #3327

That means that to fully test Cylc we need six platforms. These can probably be implemented as two containers (📦) where 📦 2 is a subtle variant of 📦 1.

	tcp	poll	ssh+tcp
remote host	📦 1	📦 1	📦 1
remote host with shared filesystem	📦 2	📦 2	📦 2

We currently support 9 batch systems (🧮):

• at
• background
• loadleveler
• lsf
• moab
• pbs
• pbs_multi_cluster
• sge
• slurm

So that leaves us with 2 * 3 * 9 = 54 containers!

Ok, so this is getting a bit nuts, for testing we can probably test all of the batch systems (🧮) in the remote host + tcp image.

	tcp	poll	ssh+tcp
remote host	📦1 + 🧮	📦 1	📦 1
remote host with shared filesystem	📦 2	📦 2	📦 2

Questions

• How many containers do we need?
• Can we avoid duplication between containers?

[hjoliver]

How many containers do we need?

Seems to me two is enough as testing batch system support is orthogonal to testing remote host support (isn't it?) with separate or shared FS - i.e. no need for the cross combinations.

Can we avoid duplication between containers?

Presumably you mean duplication of test runs, not duplication of container content? If the tests are quick enough we could run all batch system tests in both contains, and to hell with the duplication. Or we could randomly select half the batch system tests for one container, half for the other (nice side effect, we would eventually pick up unforeseen cross-combination issues, at the cost of perfect reproducibility).

[oliver-sanders]

Presumably you mean duplication of test runs

Sorry, I actually meant duplication in docker files. The remote tests are few and fairly fast so that shouldn't be a major issue.

[oliver-sanders]

Ping @kinow who knows the most.

[kinow]

Ping @kinow who knows the most.

Far from it.

How many containers do we need?

I think it would be easier to have separate containers for each batch system, instead of a single container, or containers with more than 1 batch system installed.

Can we avoid duplication between containers?

There are a few ways of doing that.

Some projects use Shell to process a template file and generate the Dockerfile, e.g. php

• https://github.com/docker-library/php/blob/master/Dockerfile-alpine.template
• https://github.com/docker-library/php/blob/6f12272b33d7f85ea6e89dffcf2a6c211773de39/update.sh#L123

A few releases ago Docker added a way to use multiple containers together. So there could be some way to package cylc in one container A (using something different than pip, maybe conda, pyinstaller, or rpm), and combine with a container B that is simply using the image of a batch system container; https://docs.docker.com/develop/develop-images/multistage-build/

Something like:

FROM cylcbase:latest AS cylc
WORKDIR /opt
# package cylc and dependencies creating some file like cylc.zip with a conda env, etc

FROM some-image-with-pbs:latest
COPY --from=cylc /opt/cylc.zip .
# at this point we should have Cylc, and the PBS, so we would just need to figure out a way to run the Cylc test-battery for PBS here

Not sure what approach would be the best, but we can try either and see if that works well on GH Actions I think?

[oliver-sanders]

Removing the question label, time for investigation...

[oliver-sanders]

Putting myself on the ticket as I've made some progress on this.

• It's fairly straight-forward to setup a Conda docker image then install Cylc into it via Conda.
• Other images can use this is their base (avoiding duplication).
• SSH from the host into a container is pretty simple.
• SSH from the container into the host is likely to more interesting.

The platform matrix for tests takes the form:

(is_local_platform, batch_system, has_shared_fs, comms_method)

Caveats:

• Installing Conda (or Mamba) onto Alpine involves provisioning your own static libraries 🤮 so we will probs have to use a more heavy-weight image.
• Conda installations use a lot of space making the docker images somewhat unwieldy.

[oliver-sanders]

The four containers in the matrix (shared_fs, indep_fs) * (tcp, poll) can be reduced to a single image with four sets of run options.

The batch systems would require one container each but could be based on one of the four "base" images from the matrix above. In practice it may be easier to combine at and background into the "base" images.

So that leaves us with one image per batch system and one container per logically sensible combination of (is_local_platform, batch_system, has_shared_fs, comms_method).

So far indep_fs works fine but there is an issue with shared_fs containers where the host cannot issue commands to workflows running inside the container (which some of the tests require). This would presumably require meddling /etc/hosts on the host system which is not viable.