Command line job submission

This is a proposal for BOINC-CLI, a Linux command-line interface for submitting jobs to a BOINC server. BOINC-CLI is intended to be simple; it should be as easy to use as similar HTCondor commands. It should not require knowing anything about BOINC or Docker.

Applications

A job consists of an application, command-line arguments, and input files.

An application is a set of files that may include

• Shell (sh) scripts
• Python programs (which may import installed Python modules)
• Compiled programs (which may dynamically link installed libraries)

One of these files is designated as the "main program".

BOINC-CLI collects these files, puts them in a VM, and runs them in a Docker container in the VM.

Packaging an application

An application is described by a TOML-format config file:

app_name = "foo"

files = [
    {path = "foo.sh", "main=true"},
    {path = "prog"},
    {path = "/usr/lib/blah.sh"},
    {path = "foo.py"}
]

boinc_package app_config_file

This reads the config file, collects all needed files, builds Docker layers, and stores them in a directory ~/boinc_apps/name.

TODO: how do we collect Python dependencies? Info is here: https://docs.docker.com/language/python/build-images/

TODO: what Docker packages are required?

TODO: how do we collect the libraries needed for a binary executable? Use ldd?

Testing an application package

To test an application locally: make a directory containing input files. In that directory, run

boinc_run_local app_name [--cmdline x y z]

This runs the given application in a Docker container, copying the input files to its directory, passing the given args to the main program.

It shows all resulting error messages. Any resulting output files are put in the current directory.

Submitting jobs

Create a file ~/.boinc/project containing

# project URL and account authenticator

project_url = "https://foo.blah/"
auth = "fkjeifkjfkf"

To submit jobs, prepare a config file of the form

app_name = "foo"
jobs = [
    {
        id: "1",
        cmdline: "--foo",
        infiles = {"in1": "path1", "in2": "path2"},
        outfiles = {"out1", "path1"}
    },
    # ...
]

To check the validity of the config file:

boinc_submit --check job_config_file

To submit the jobs:

boinc_submit job_config_file

This creates a "batch" and shows the batch ID

To see the status of running jobs:

boinc_status [--batch id]

This shows the list of batches. If an ID is given, it shows the status of each job (queued, running, succeeded, failed).

To abort jobs:

boinc_abort --batch id

To fetch the output files of completed jobs:

boinc_fetch --batch id

Implementation

The implementation shouldn’t be that hard. It’s based on technology we already have: boinc2docker, and the remote job and file management mechanisms.

The boinc_submit command (a Python script) does the following:

1. Check ~/.boinc/apps to see whether we have a Docker layer for the app. If not, build one using boinc2docker.
2. Use the remote file management mechanism to copy files (app and input) to the Apache container.
3. Use the remote job submission mechanism to submit the jobs. Write its ID to a file.

boinc_status etc. use the remote job submission mechanism.