Digital Curation Manager - Job Processor

The 'DCM Job Processor'-API provides functionality to process jobs, i.e. a sequence of processing steps, in the DCM. This repository contains the corresponding Flask app definition. For the associated OpenAPI-document, please refer to the sibling package dcm-job-processor-api.

The contents of this repository are part of the Digital Curation Manager.

Local install

Make sure to include the extra-index-url https://zivgitlab.uni-muenster.de/api/v4/projects/9020/packages/pypi/simple in your pip-configuration to enable an automated install of all dependencies. Using a virtual environment is recommended.

1. Install with

   pip install .
   

2. Configure service environment to fit your needs (see here).
3. Run app as

   flask run --port=8080
   

4. To manually use the API, either run command line tools like curl as, e.g.,

   curl -X 'POST' \
     'http://localhost:8080/process' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "process": {
       "from": "import_ies",
       "to": "import_ies",
       "args": {
         "import_ies": { ... },
         "import_ips": { ... },
         "build_ip": { ... },
         "validation_metadata": { ... },
         "validation_payload": { ... },
         "build_sip": { ... },
         "transfer": { ... },
         "ingest": { ... },
       }
     },
     "id": "dab3e1bf-f655-4e57-938d-d6953612552b"
   }'
   

   or run a gui-application, like Swagger UI, based on the OpenAPI-document provided in the sibling package dcm-job-processor-api.

Run with docker compose

Simply run

docker compose up

By default, the app listens on port 8080. To rebuild an already existing image, run docker compose build.

Additionally, a Swagger UI is hosted at

http://localhost/docs

Afterwards, stop the process and enter docker compose down.

Tests

Install additional dev-dependencies with

pip install -r dev-requirements.txt

Run unit-tests with

pytest -v -s

Environment/Configuration

Service-specific environment variables are

• PROCESS_TIMEOUT [DEFAULT 30] service timeout duration in seconds
• IMPORT_MODULE_HOST [DEFAULT http://localhost:8080] Import Module host address
• IP_BUILDER_HOST [DEFAULT http://localhost:8081] IP Builder host address
• OBJECT_VALIDATOR_HOST [DEFAULT http://localhost:8082] Object Validator host address
• SIP_BUILDER_HOST [DEFAULT http://localhost:8083] SIP Builder host address
• TRANSFER_MODULE_HOST [DEFAULT http://localhost:8084] Transfer Module host address
• BACKEND_HOST [DEFAULT http://localhost:8085] Backend host address

Additionally this service provides environment options for

• BaseConfig and
• OrchestratedAppConfig

as listed here.

Contributors

• Sven Haubold
• Orestis Kazasidis
• Stephan Lenartz
• Kayhan Ogan
• Michael Rahier
• Steffen Richters-Finger
• Malte Windrath
• Roman Kudinov