chore: decouple Docker image builder from substrapp

[SdgJlbl]

Description

How has this been tested?

Open questions

• which version for the helm chart?

Checklist

• changelog was updated with notable changes
• documentation was updated

[linear]

FL-1080 Isolate Docker image builder in its own service

Context

https://www.notion.so/owkin-fdn/Decoupling-Docker-image-build-from-Substra-task-execution-b7b8d1e8abb44a42b8571d758c95d2c6

First step is to isolate all the code used to build a Docker image from a Dockerfile + build context in a separate service.

Specification

In the long term, we probably want the builder to be an independent service, but in the interest of time, I propose to host it for now in a Django app in the backend.

API spec:

• Input: name of the function to build
• Output: Docker image stored in the registry + status
• The actual build context and Dockerfile is currently stored in MinIO during the function registration call, the builder should get it from there
• We don't want a "blocking call" to the builder (because it will moved to the registration part that is synchronous), two options:
  • A/ endpoint returning 202 Accepted and processing the build in the background; another endpoint could be developed to return current status of the build (pending, done or failed)
  • B/ event-based system using gPRC, similar to what is done to communicate with the orchestrator

Technical refinement

• go for option B (for easier integration with a future task queue)
• no DB for the builder, image build status to be stored by the orchestrator

Acceptance criteria

Docker image build is no longer done as part of task execution.

[SdgJlbl]

/e2e

[SdgJlbl]

/e2e

[SdgJlbl]

/e2e

[SdgJlbl]

/e2e

[Owlfred]

End to end tests: ❌ FAILURE

Jobs status:

• Camelyon: ⏭️
• Dispatch Jobs: ✔️
• Distributed / distributed-sdk: ❌
• MNIST: ⏭️
• Standalone / standalone-sdk: ❌

Oh well.

[SdgJlbl]

Closed since the POC now lives on https://github.com/Substra/substra-backend/tree/poc-decoupled-builder