⚠️ CBI is no longer under active development, in favor of Tekton. See tektoncd/pipeline and tektoncd/catalog.

---

CBI: Container Builder Interface for Kubernetes

CBI provides a vendor-neutral interface for building (and pushing) container images on top of a Kubernetes cluster, with support for several backends such as Docker, BuildKit, Buildah, kaniko, img, Google Cloud Container Builder, Azure Container Registry Build and OpenShift Source-to-Image (S2I).

• Current status
  • Specification
  • Implementation
• Quick start
  • Installation
  • Run your first buildjob
• Advanced usage
  • Push to a registry
  • Build contexts
    • ConfigMap context
    • Git context
    • HTTP(S) context
    • Rclone context (S3, Dropbox, SFTP, and many)
  • Plugin
    • Specify the plugin explicitly
    • Google Cloud Container Builder plugin
    • Azure Container Registry Build plugin
    • Openshift Source-to-Image plugin
• Design (subject to change)
  • Components
  • Build context
    • BuildkitSession (Planned)
• Contribute to CBI
  • Testing
  • Local testing with DinD
• FAQs
  • Q: Does CBI standardize the Dockerfile specification?
  • Q: Does CBI replace BuildKit?
  • Q: Is CBI a part of Kubernetes, a Kubernetes incubator, or a CNCF project?

Current status

Specification

• CBI CRD: pre-alpha, see pkg/apis/cbi/v1alpha1/types.go.
• CBI plugin API: pre-alpha, see pkg/plugin/api/plugin.proto.

Implementation

• CBI controller daemon (cbid): pre-alpha, see cmd/cbid.

• Plugins (all of them are pre-alpha):

Plugin	Backend	Dockerfile	cloudbuild.yaml	OpenShift S2I	BuildKit LLB	ACB Pipeline
docker	Docker	Yes ✅
buildkit	BuildKit	Yes ✅			Planned
buildah	Buildah	Yes ✅
kaniko	kaniko	Yes ✅
img	img	Yes ✅
gcb	Google Cloud Container Builder	Yes ✅	Yes ✅
acb	Azure Container Registry Build	Yes ✅			Planned	Planned
s2i	OpenShift Source-to-Image (S2I)			Yes ✅

• Planned plugins (subject to change): Bazel, Singularity, OpenShift Image Builder, Orca, ...

• Context providers (available for all plugins)

  • ConfigMap
  • Git, with support for SSH secret
  • HTTP(S)
  • Rclone: Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, DigitalOcean Spaces, Dreamhost, Dropbox, FTP, Google Cloud Storage, Google Drive, HTTP, Hubic, IBM COS S3, Memset Memstore, Microsoft Azure Blob Storage, Microsoft OneDrive, Minio, Nextloud, OVH, Openstack Swift, Oracle Cloud Storage, Ownloud, pCloud, put.io, QingStor, Rackspace Cloud Files, SFTP, Wasabi, WebDAV, Yandex Disk

• Planned context providers: BuildKitSession

Please feel free to open PRs to add other plugins.

Quick start

Requires Kubernetes 1.9 or later.

Installation

$ kubectl apply -f https://raw.githubusercontent.com/containerbuilding/cbi/master/cbi-latest.yaml

Everything (except CustomResourceDefinition, ClusterRole, and ClusterRoleBinding) will be installed to the cbi-system namespace.

You may edit the YAML file to change the namespace or to add NetworkPolicy.

Hint for Google Kubernetes Engine (GKE) users

If you hit an error while creating clusterrolebindings, you need to execute kubectl command as follows:

$ pw=$(gcloud --format json container clusters describe ${YOUR_GKE_CLUSTER_NAME} | jq -r .masterAuth.password)
$ kubectl --username=admin --password=${pw} ...
 

Hint for OpenShift users

oc login -u system:admin -n default might be needed before running kubectl.

You would also need to enable privileged containers for most plugins. (how?)

The CBI controller daemon and the following plugins will be installed:

Plugin	Requirements
docker	Docker needs to be installed on the hosts
buildah	Privileged containers needs to be enabled
buildKit	Privileged containers needs to be enabled
kaniko	None (Google Cloud is not needed)
img	Privileged containers needs to be enabled (See kubernetes/community#1934 and Jess's blog for the ongoing work to remove this requirement)
gcb	Requires Google Cloud service account with IAM roles, see this section (Your cluster does not need to be GKE or on GCE)
acb	Requires Azure service principal with IAM roles, see this section (Your cluster does not need to be AKS or on Azure VMs)
s2i	Docker needs to be installed on the hosts (OpenShift is not needed)

The default plugin is docker.

You may edit the YAML file to remove unneeded plugins or change the priorities.

Run your first buildjob

Create a buildjob ex-git-nopush from examples/ex-git-nopush.yaml:

$ kubectl create -f https://raw.githubusercontent.com/containerbuilding/cbi/master/examples/ex-git-nopush.yaml
buildjob "ex-git-nopush" created

Make sure the buildjob is created:

$ kubectl get buildjobs
NAME      AGE
ex-git-nopush       3s

Inspect the underlying job and the result:

$ kubectl get job $(kubectl get buildjob ex-git-nopush --output=jsonpath={.status.job})
NAME      DESIRED   SUCCESSFUL   AGE
ex-git-nopush-job   1         1            30s
$ kubectl logs $(kubectl get pods --selector=job-name=ex-git-nopush-job --show-all --output=jsonpath={.items..metadata.name})
Sending build context to Docker daemon 79.87 kB
Step 1 : FROM alpine:latest
...
Successfully built bef4a548fb02

Delete the buildjob (and the underlying job)

$ kubectl delete buildjobs ex-git-nopush
buildjob "ex-git-nopush" deleted

Advanced usage

Push to a registry

First you need to create a credential using kubectl create secret docker-registry .... See Kubernetes's manual.

Hint for Amazon Elastic Container Registry (ECR) users

You can create the credential as follows (expires per 12 hours):

$ TOKEN=`aws ecr get-authorization-token --output text --query authorizationData[].authorizationToken | base64 -d | cut -d: -f2`
$ kubectl create secret docker-registry my-registry-secret \
 --docker-server=https://12345678.dkr.ecr.ap-northeast-1.amazonaws.com \
 --docker-username=AWS \
 --docker-password="${TOKEN}" \
 --docker-email="${EMAIL}"

Hint for Google Container Registry (GCR) users

See here for creating the credential.

Hint for Azure Container Registry (ACR) users

See here for creating the credential.

You can specify the registry credential via spec.registry.secretRef.name.

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-git-push
spec:
  registry:
    target: example.com/foo/bar:baz
    push: true
    secretRef:
      name: docker-registry-secret-name
  language:
    kind: Dockerfile
  context:
    kind: Git
    git:
      url: ssh://me@git.example.com/foo/bar.git

Note: for Google Cloud Container Builder plugin, please refer to the Google Cloud Container Builder plugin section.

Note: for Azure Container Registry Build plugin, please refer to the Azure Container Registry Build plugin section.

Build contexts

ConfigMap context

This is the easiest but only suitable for trivial images.

Example manifest:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ex-configmap-nopush-configmap
data:
  Dockerfile: |-
    FROM busybox
    ADD hello /
    RUN cat /hello
  hello: "hello, world"
---
apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-configmap-nopush
spec:
  registry:
    target: example.com/foo/ex-configmap-nopush
    push: false
  language:
    kind: Dockerfile
  context:
    kind: ConfigMap
    configMapRef:
      name: ex-configmap-nopush-configmap

Git context

Git context is suitable for most cases.

If the repo is private, you need to create a SSH secret as follows, and specify the secret via spec.context.git.sshSecretRef.name:

$ kubectl create secret generic ssh-secret-name --from-file=id_rsa=$HOME/.ssh/id_rsa --from-file=config=$HOME/.ssh/config --from-file=known_hosts=$HOME/.ssh/known_hosts

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-git-nopush
spec:
  registry:
    target: example.com/foo/bar:baz
    push: false
  language:
    kind: Dockerfile
  context:
    kind: Git
    git:
# only url is mandatory. See git(1) for url spec.
      url: ssh://me@git.example.com/foo/bar.git
      revision: master
      sshSecretRef:
        name: ssh-secret-name

HTTP(S) context

HTTP(S) context provider allows using tar(.gz) archive as a build context. This is useful for sending large contexts without interacting with a git repo.

You can create a temporary HTTP server in the Kubernetes cluster, and upload a context tarball as follows.

$ kubectl run nginx --image nginx:alpine --port 80
$ kubectl expose deployment nginx
$ tar cvf a.tar /path/to/your-context-directory
$ kubectl cp a.tar $(kubectl get pod -l run=nginx -o jsonpath={..metadata.name}):/usr/share/nginx/html

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex
spec:
  registry:
    target: example.com/foo/bar:baz
    push: false
  language:
    kind: Dockerfile
  context:
    kind: HTTP
    http:
      url: http://nginx/a.tar

Rclone context (S3, Dropbox, SFTP, and many)

Rclone supports fetching files and directories from various storage services: Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, DigitalOcean Spaces, Dreamhost, Dropbox, FTP, Google Cloud Storage, Google Drive, HTTP, Hubic, IBM COS S3, Memset Memstore, Microsoft Azure Blob Storage, Microsoft OneDrive, Minio, Nextloud, OVH, Openstack Swift, Oracle Cloud Storage, Ownloud, pCloud, put.io, QingStor, Rackspace Cloud Files, SFTP, Wasabi, WebDAV, Yandex Disk.

Any backend supported by Rclone should work with CBI, although only Amazon S3 is tested with CBI currently.

To use Rclone context provider, you need to create a secret from your ~/.config/rclone/rclone.conf, and specify as spec.context.rclone.secretRef.

$ kubectl create secret generic my-rclone-secret --from-file=$HOME/.config/rclone/rclone.conf

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-rclone-nopush
spec:
  registry:
    target: example.com/foo/ex-rclone-nopush
    push: false
  language:
    kind: Dockerfile
  context:
    kind: Rclone
    rclone:
      remote: s3
      path: my-s3-bucket/some-directory
      secretRef:
        name: my-rclone-secret

To use SFTP remote, you might need to specify spec.context.rclone.sshSecretRef as in Git context.

Plugin

Specify the plugin explicitly

Usually. the plugin is automatically selected by the CBI controller daemon.

However, if you prefer a specific plugin for some reason such as performance or stability, you can specify the plugin explicitly using spec.pluginSelector constraint.

e.g. for Buildah plugin (plugin.name=buildah),

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  ...
spec:
  pluginSelector: plugin.name=buildah
  ...

Google Cloud Container Builder plugin

You need to create a Google Cloud service account JSON with the following IAM roles in https://console.cloud.google.com/iam-admin/serviceaccounts :

• Cloud Container Builder Editor
• Project Viewer
• Storage Admin

And create a corresponding Kubernetes secret that contains json as follows:

$ kubectl create secret generic my-gcb --from-file=json=my-gcb-service-account.json

You don't need to use GKE (of course you can use though).

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-git-push
  annotations:
    cbi-gcb/secret: my-gcb
    cbi-gcb/project: my-gcp-project
spec:
  registry:
    target: gcr.io/example/foo
    push: true
  language:
    kind: Dockerfile
  context:
    kind: Git
    git:
      url: https://git.example.com/foo/bar.git
  pluginSelector: plugin.name=gcb

Note:

• metadata.annotations["cbi-gcb/secret"] needs to be set to the name of the secret
• metadata.annotations["cbi-gcb/project"] needs to be set to the name of the Google Cloud project
• spec.registry.target needs to be in the gcr.io/* or *.gcr.io/* namespace.
• spec.registry.push needs to be true
• spec.registry.secretRef must not be set

In addition to Dockerfile, gcb plugin also supports building images from cloudbuild.yaml.

See examples/ex-google-cloudbuild-push.yaml.sh.

Azure Container Registry Build plugin

You need to create a Azure service principal with a PEM/DER cert:

$ az ad sp create-for-rbac --name ServicePrincipalName --create-cert
{
  "appId": "...",
  "displayName": "..."",
  "fileWithCertAndPrivateKey": "my-acb.pem",
  "name": "http://...",
  "password": null,
  "tenant": "..."
}

And create a corresponding Kubernetes secret that contains cert as follows:

$ kubectl create secret generic my-acb --from-file=cert=my-acb.pem

You don't need to use AKS (of course you can use though).

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-git-push
  annotations:
    cbi-acb/secret: my-acb
    cbi-acb/app-id: APP_ID
    cbi-acb/tenant: TENANT
spec:
  registry:
    target: example.azurecr.io/example/foo
    push: true
  language:
    kind: Dockerfile
  context:
    kind: Git
    git:
      url: https://git.example.com/foo/bar.git
  pluginSelector: plugin.name=acb

Note:

• metadata.annotations["cbi-acb/secret"] needs to be set to the name of the secret
• metadata.annotations["cbi-acb/app-id"] needs to be set to the App ID.
• metadata.annotations["cbi-acb/tenant"] needs to be set to the tenant ID.
• spec.registry.target needs to be in the *.azurecr.io/* namespace.
• spec.registry.secretRef must not be set
• To prevent the password from being leaked via the az login command line strings in Kubernetes Job objects, we don't support password-based authentication for the service principal. You may need to create an additional password-based service principal for ImagePullSecret. See here.

Openshift Source-to-Image plugin

s2i plugin supports building images from S2I source but it does not support Dockerfile.

See examples/ex-s2i-nopush.yaml.

Design (subject to change)

Components

CBI is composed of the following specifications and implementations.

Specifications:

• CBI CRD: Kubernetes custom resource definition for buildjob objects.
• CBI plugin API: gRPC API used for connecting cbid to plugins.

Implementations:

• CBI controller daemon (cbid): a controller that watches creation of CBI CRD objects and creates Kubernetes Job objects correspondingly.
• CBI CLI (cbictl): a reference CLI implementation for cbid
• CBI plugins: the actual pods that build and push images.
• CBI session manager (cbism): pods that speak BuildKit session gRPC (or other similar protocols) for supporting large build context and diffcopy.

The concept of CBI session manager (cbism) is decoupled from cbid, so as to make cbid free from I/O overhead.

Build context

CBI defines the following values for context.kind:

• ConfigMap: Kubernetes config map. Only suitable for small contexts.
• Git: git repository, with support for Kubernetes secrets
• HTTP: HTTP(S) tar(.gz) ball
• Rclone: Rclone

Plugin implementations SHOULD implement ConfigMap, Git, and HTTP, but none of them is mandatory. Also, implementations MAY accept non-standard context.kind values.

For ease of implementation of these context providers, CBI provides cbipluginhelper image that contains CLI utilities for accessing these remote contexts. Plugin implementations may inject the cbipluginhelper image with an emptyDir volume into the initContainers of the job pods, so as to support contexts that are not natively supported by the backends.

e.g. CBI plugin for Docker (cbi-docker) supports Rclone context using cbipluginhelper, while Docker itself does not support Rclone.

BuildkitSession (Planned)

If BuildkitSession is specified as context.kind, the pod ID of a CBI session manager, TCP port number, and the session ID would be set to the status fields of the BuildJob object.

The client is expected to send the context to the specified session manager pod using BuildKit session gRPC (via the HTTP/1.1 gate). To connect to the pod, the client may use kubectl port-forward or kubectl exec ... socat.

Future version would also provide Ingress for exposing the CBI session manager in more efficient ways.

Contribute to CBI

• Vendoring is managed via dep.
• To update CRD definition, please edit pkg/apis/cbi/v1alpha1/types.go and run hack/codegen/update-codegen.sh. Please do not edit autogenerated files manually.

Testing

$ ./hack/build/build-push-apply.sh your-registry.example.com:5000/cbi test20180501

This command performs:

• Build and push CBI images as your-registry.example.com:5000/cbi/{cbid,cbi-docker,...}:test20180501
• Generate /tmp/cbi.generated.yaml so that the manifest uses the images on your-registry.example.com:5000/cbi/{cbid,cbi-docker,...}:test20180501.
• Execute kubectl apply -f /tmp/cbi.generated.yaml.

Local testing with DinD

You may use hack/dind/up.sh for setting up a local Kubernetes cluster and a local registry using Docker-in-Docker.

$ ./hack/dind/up.sh
$ DOCKER_HOST=localhost:62375 ./hack/build/build-push-apply.sh cbi-registry:5000/cbi test20180501
$ ./hack/dind/down.sh

The Kubernetes cluster and the "bootstrap" Docker listening at localhost:62375 can connect to cbi-registry:5000 without auth.

FAQs

Q: Does CBI standardize the Dockerfile specification?

A: No, the Dockerfile specification has been maintained by Docker, Inc.

CBI itself is neutral to any image building instruction language (e.g. Dockerfile).

However, most backend implementations would accept Dockerfile.

Q: Does CBI replace BuildKit?

A: No, CBI just provides an abstract interface for several backends such as BuildKit.

Q: Is CBI a part of Kubernetes, a Kubernetes incubator, or a CNCF project?

A: Currently no, unlike CRI/CNI/CSI.

But it'd be good to donate CBI to such a vendor-neutral organization if CBI becomes popular.