Support image components to use Dockerfile as component spec

[l0rd]

Requirement

If the project includes a Dockerfile, the developer should be able to use it as a DevWorkspace component.

The specification

In the 2.2.0 Devfile API image components can reference a Dockerfile and a container can use that image:

components:
  - name: devtools-image
    image:
      imageNameSelector: devtools                    # <=== was `imageName` see https://github.com/devfile/api/issues/985
      autoBuild: true
      dockerfile:
        uri: Dockerfile                              # <=== path relative to the project root
  - name: devtools-container2
    container:
      image: quay.io/devfile/devtools:ubi8-latest    # <=== matches selector and replaced when reconciling
      memoryLimit: 512Mi
      command: ['sleep', 'infinity']

Implementation proposal

Similar to project-clone, the DevWorkspace controller could start an image-build init-container to build and publish the image (the project should be cloned first to get the Dockerfile and the build context).

The image registry, name and tag are inferred by the controller and are not exposed to the user:

• the registry should be a DWO config
• the name should be inferred by the devfile and component name
• the tag should be incremental

The container component image that matches the selector is replaced with the inferred image/name/tag

Additional context

Full support of the image components is out of scope. In particular the DevWorkspace controller should raise a warning and ignore an image component that has:

• autoBuild set to false
• the dockerfile referenced through git or the devfile registry

We should also clarify what are the scenarios for images with autobuild: false and that are not referenced by other components.

[amisevsk]

I spent some time working on this and have some proof-of-concept code for building containers in workspace initContainers. However, there are a few issues/open questions:

• [blocker] Using an init container for this purpose, as described, will not work -- at least not without complicated and potentially error-prone workarounds. To start a pod on the cluster, Kubernetes first needs to pull all images for that pod, leading to pods expecting to build their own images failing to start (need initContainer to build image -> need image to start initContainer)
• We could potentially instead use a job (or separate pod) to perform the container build, but then it becomes unclear when this job should be restarted and how cloning should happen (do we re-do the build every time the workspace stops + is started? How do we track this in the operator, which is stateless?)
• Including images to build can lead to significant delays in starting a workspace, with a container build generally taking around a minute for a moderately complex container (potentially more or less, depending on what the container is doing)
  • This could potentially be worked around by allocating PVC storage to cache layers between runs; I haven't investigated this very far but this could speed up re-builds at the cost of requiring storage and managing its cleanup (when do we clear old layers from this PVC?)
• Enabling automatic container build + push is complicated and requires a significant amount of additional configuration:
  • There needs to be some way to share image repository credential information with the builder pod (in my testing, I use a quay.io robot account and authfile mounted into the builder container)
  • The operator needs to have configuration for a repository it can push to, and the credentials provided above need to allow read/write access to that repository.

In addition to the above, we should consider other approaches to building containers -- for example, OpenShift provides BuildConfigs and ImageStreams that could potentially be used to achieve the same goal (with fewer or different drawbacks).