File system layout

This page contains information about how mistry data are laid out in the file system.

There are (2) main categories of data: build data and the project data. These two categories are placed inside their respective directories: the "data" directory and the "projects" directory.

Data directory

This directory contains:

• the build artifacts
• build logs
• build result metadata (exit code, status etc.)

Given a project named "foo" and data_path set to /var/lib/mistry/data, an example data directory looks like this:

/var/lib/mistry/data/: $ tree foo
`-- foo # project name
    |-- latest -> /var/lib/mistry/data/foo/ready/589e9... # symlink to the latest completed build
    |-- pending # currently running jobs
    |   |-- 48f96a905fa3de9dc150d28973233bd6d83cb3349812946b4013e2d82fce9504
    |   |   |-- data
    |   |   |   |-- artifacts
    |   |   |   |-- cache
    |   |   |   `-- params
    |   |   |-- out.log
    |   |   `-- build_info.json
    `-- ready # completed jobs
        |-- 589e9563457ed1cec9b1d33549fe1565eab51b0c839bb0c2d53ad476afa6503c # job ID
        |   |-- data
        |   |   |-- artifacts   # results of the build
        |   |   |-- cache       # data persisted across builds for incremental building
        |   |   `-- params      # job parameters (as files)
        |   |-- out.log         # the log of the build
        |   `-- build_info.json # metadata regarding the result of the build (exit code etc.)
        `-- 645d23ee67b17f091eeba36536898d3519dd89676077bfdb1f90b9be075ef665
            |-- data
            |   |-- artifacts
            |   |-- cache
            |   `-- params
            |-- out.log
            `-- build_info.json
`-- projectB
  # ...

The data directories in the host are mounted at /data inside job execution containers. For example, the container of the job 48f96a9... would have /var/lib/mistry/data/foo/pending/48f96a905fa3de9dc150d28973233bd6d83cb3349812946b4013e2d82fce9504/data mounted at /data.

This path is where all the magic happens and it contains (3) sub-directories:

1. /data/artifacts/: contains build results (aka. artifacts). These are the contents that are served when someone uses mistry-cli
2. /data/cache/: contains contents that are used for caching purposes, written from previous builds. Files that should be retained between builds and are not part of the artifacts should be placed here.
3. /data/params/: contains the job parameters

Projects directory

This is where the job execution environments exist, in the form of Docker images. For an example, take a look at the project directory used in the tests.

It is set via the projects_path setting.