Processor cookiecutter

[pkcakeout]

Required features

• Loading of images must be automated
• Processor specifications must describe which input formats are expected
  • If files are not specified they should be ignored
  • Inputs should support MHD/MHA/TIFF image formats
  • Inputs should support CSV format
• Support for multi-case processing is desirable
• Call a standard function to execute processing
  • Similar to evalutils, this should be generated automatically from the cookiecutter project
• Results:
  • JSON output (as python data structures)
  • Later: Overlays
• Configurable required resources for running the container
  • Automatically switch GPU enabled docker image if req. gpu_count > 0
• Dockerfile generation
• Local test run support
• Write tests
  • processor template generation
  • processor CLI tests
  • processor template runs on dummy data
  • local test / build / export tools works
• Partly updated documentation (needs a more comprehensive guide in the future)

[pkcakeout]

Rough template for the Dockerfile is available here:

https://github.com/DIAGNijmegen/rse-unit-of-knowledge-template/blob/master/processor/Dockerfile.template

[pkcakeout]

• Setup Continuous Integration builds. Options:
  • CircleCI
  • GitHub Actions (in beta)

[silvandeleemput]

After studying the processor specifications I came up with some changes to the /output/results.json format specifications which I think are both simpler and an improvement over the current specification.

To accommodate most use cases, processors should support one-to-many, many-to-one, and many-to-many file mappings (input -> output). To be complete, processors should support zero-to-many and many-to-zero mappings as well.

This feature is currently supported by entity specifying a list of [inputs] and under metrics specifying an output "filepath:". However, specifying the output files under metrics seems arbitrary and a bit off.

My proposal is to specify both inputs and outputs (as a list of file names / strings) under entity and remove the filepath: specification altogether frommetrics. This will simplify the output, neatly combining the input / output relation under the entity section and disentangle the outputs from the unrelated metrics section. An additional benefit is that this would simplify the automatic generation of the entity section in the results file using processor templating.

An example proposal results.json file would look something like the following:

[
  {
    "entity": {       
        "inputs": ["filenames", ...],
        "outputs": ["filenames", ...],
        ...
    },
    "metrics": ...,
    "error_messages": [
      ...
    ]
  },
  ...
]

I have discussed this with Paul and he agrees that this change will probably improve the specification. However, there seem to be some processors which follow the current specification.

[silvandeleemput]

• Setup of continuous integrations builds can be done through the existing Travis CI setup, since we want to integrate the processor templating inside evalutils. Tests still need to be written.

• Required resources can now be set through the evalutils CLI and get inserted into the Dockerfile using cookiecutter. See the following list:

  • com.radboudumc.diag.processor.hardware.cpu.count (int +)
  • com.radboudumc.diag.processor.hardware.cpu.capabilities (list of string *)
  • com.radboudumc.diag.processor.hardware.memory (string)
  • com.radboudumc.diag.processor.hardware.gpu.count (int *)
  • com.radboudumc.diag.processor.hardware.gpu.cuda_compute_capability (string)
  • com.radboudumc.diag.processor.hardware.gpu.memory (string)

• Additionally, the following meta data will be required to set:

  • com.radboudumc.diag.processor.author (string)
  • com.radboudumc.diag.processor.ticket (string)
  • com.radboudumc.diag.processor.name (string)

[silvandeleemput]

I studied the EnvoyAI API, which do some interesting things:

• Specification of inputs and outputs through (json/yaml) schema in the DockerFile including specific datatypes for all
  • Input and output file paths are hardcoded.
• Apparently it usually expects to process only a single case / run, but it does support e.g. dicom-series datatypes
• Support for labels in the dockerfile for displaying certain datatypes

[jmsmkn]

Looks good!

• Additionally, the following meta data will be required to set:

  • com.radboudumc.diag.processor.author (string)
  • com.radboudumc.diag.processor.ticket (string)
  • com.radboudumc.diag.processor.name (string)

I wouldn't make these required, or at least have an option to turn this off. We will want to use this library elsewhere, eg, challenges on grand challenge that take in processors from users, rather than prediction files.

[silvandeleemput]

@jmsmkn I would opt to keep the com.radboudumc.diag.processor.name label, since this will be used by cookiecutter for project folder and class creation (and maybe roughly specify its purpose)

The author label might even be unnecessary since a processor/algorithm will be tagged upon upload to grand-challenge.

The ticket label is only for internal use and I think prompting might be strange for external users. We could keep it optional for internal use though and maybe let users specify it with a flag. (should be well documented)

[jmsmkn]

What is the status of this issue? The PR was merged but there are probably some things left over. Should we continue in a new issue for these?

[silvandeleemput]

@jmsmkn I think, the main things left to do are:

• Documentation for the processor template Documentation for the algorithm-templating #271
• Adding schemas for input/output formatting and better grand-challenge integration Using schemas for explicit input and output types results.json #246

Since these are addressed in other issues, I think we can close this issue.

[jmsmkn]

Continued in #246