Split workers image into job-specific images and configure Nomad to use the correct one for each job.

[kurtwheeler]

Context

As we continue to create different Processors the ccdl/dr_workers Docker image will continue to grow to contain all the dependencies and (at least some of the data) the different jobs need. This makes for a very bulky image that takes a long time to build, upload, and download along with requiring a non-trivial amount of disk space to store.

Problem or idea

We should have a Docker image for every type of job we have. We should also have different Nomad job specs for each type of job, rather than having a single processor.nomad job spec for all of them.

Solution or next step

Split the workers Docker image into separate images for each job type and create Nomad job specifications for each job type. Note that some jobs may still be able to an image, for example the SCAN.UPC image should be usable by both Affymetrix and Illumina specific jobs types.

Finally, change data_refinery_common.message_queue.send_job function to specify job types directly to Nomad rather than specifying the job type via the job parameters.

[jaclyn-taroni]

We just talked about what an "RNA-seq" Docker image might contain: RSEM, Salmon, salmontools, tximport and for QC purposes FastQC, MultiQC

Just recording this conversation here, which is to say this is not a substitute for looking at the code for the processor!