diff --git a/README.md b/README.md index 991690612726..649835a9ebd2 100644 --- a/README.md +++ b/README.md @@ -27,9 +27,11 @@ Read the proposal from https://github.com/moby/moby/issues/32925 Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056cc5317 +:information_source: If you are visiting this repo for the usage of experimental Dockerfile features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)`, please refer to [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md). + ### Used by -[Moby](https://github.com/moby/moby/pull/37151) +[Moby & Docker](https://github.com/moby/moby/pull/37151) [img](https://github.com/genuinetools/img) @@ -37,6 +39,12 @@ Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056c [container build interface](https://github.com/containerbuilding/cbi) +[Knative Build Templates](https://github.com/knative/build-templates) + +[boss](https://github.com/crosbymichael/boss) + +[Rio](https://github.com/rancher/rio) (on roadmap) + ### Quick start Dependencies: @@ -79,6 +87,7 @@ See [`solver/pb/ops.proto`](./solver/pb/ops.proto) for the format definition. Currently, following high-level languages has been implemented for LLB: - Dockerfile (See [Exploring Dockerfiles](#exploring-dockerfiles)) +- [Buildpacks](https://github.com/tonistiigi/buildkit-pack) - (open a PR to add your own language) For understanding the basics of LLB, `examples/buildkit*` directory contains scripts that define how to build different configurations of BuildKit itself and its dependencies using the `client` package. Running one of these scripts generates a protobuf definition of a build graph. Note that the script itself does not execute any steps of the build. @@ -145,6 +154,10 @@ buildctl build --frontend=gateway.v0 --frontend-opt=source=tonistiigi/dockerfile buildctl build --frontend gateway.v0 --frontend-opt=source=tonistiigi/dockerfile --frontend-opt=context=git://github.com/moby/moby --frontend-opt build-arg:APT_MIRROR=cdn-fastly.deb.debian.org ```` +##### Building a Dockerfile with experimental features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)` + +See [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md). + ### Exporters By default, the build result and intermediate cache will only remain internally in BuildKit. Exporter needs to be specified to retrieve the result. @@ -207,15 +220,22 @@ buildctl debug workers -v BuildKit can also be used by running the `buildkitd` daemon inside a Docker container and accessing it remotely. The client tool `buildctl` is also available for Mac and Windows. +We provide `buildkitd` container images as [`moby/buildkit`](https://hub.docker.com/r/moby/buildkit/tags/): + +* `moby/buildkit:latest`: built from the latest regular [release](https://github.com/moby/buildkit/releases) +* `moby/buildkit:rootless`: same as `latest` but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md) +* `moby/buildkit:master`: built from the master branch +* `moby/buildkit:master-rootless`: same as master but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md) + To run daemon in a container: ``` -docker run -d --privileged -p 1234:1234 tonistiigi/buildkit --addr tcp://0.0.0.0:1234 +docker run -d --privileged -p 1234:1234 moby/buildkit:latest --addr tcp://0.0.0.0:1234 export BUILDKIT_HOST=tcp://0.0.0.0:1234 buildctl build --help ``` -The `tonistiigi/buildkit` image can be built locally using the Dockerfile in `./hack/dockerfiles/test.Dockerfile`. +The images can be also built locally using `./hack/dockerfiles/test.Dockerfile` (or `./hack/dockerfiles/test.buildkit.Dockerfile` if you already have BuildKit). ### Opentracing support diff --git a/docs/rootless.md b/docs/rootless.md index 8c4cff1cf429..ef42886a5c21 100644 --- a/docs/rootless.md +++ b/docs/rootless.md @@ -2,8 +2,9 @@ Requirements: - runc `a00bf0190895aa465a5fbed0268888e2c8ddfe85` (Oct 15, 2018) or later -- Some distros such as Debian (excluding Ubuntu) and Arch Linux require `echo 1 > /proc/sys/kernel/unprivileged_userns_clone` -- `newuidmap` and `newgidmap` need to be installed on the host. These commands are provided by the `uidmap` package. +- Some distros such as Debian (excluding Ubuntu) and Arch Linux require `sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone"`. +- RHEL/CentOS 7 requires `sudo sh -c "echo 28633 > /proc/sys/user/max_user_namespaces"`. You may also need `sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"`. +- `newuidmap` and `newgidmap` need to be installed on the host. These commands are provided by the `uidmap` package. For RHEL/CentOS 7, RPM is not officially provided but available at https://copr.fedorainfracloud.org/coprs/vbatts/shadow-utils-newxidmap/ . - `/etc/subuid` and `/etc/subgid` should contain >= 65536 sub-IDs. e.g. `penguin:231072:65536`. - To run in a Docker container with non-root `USER`, `docker run --privileged` is still required. See also Jessie's blog: https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/ @@ -56,9 +57,10 @@ $ build-using-dockerfile --buildkit-addr unix:///run/user/1001/buildkit/buildkit ## Set up (using a container) +Docker image is available as [`moby/buildkit:rootless`](https://hub.docker.com/r/moby/buildkit/tags/). + ``` -$ docker build -t buildkit-rootless --target rootless -f hack/dockerfiles/test.Dockerfile . -$ docker run --name buildkitd -d --privileged -p 1234:1234 buildkit-rootless --addr tcp://0.0.0.0:1234 +$ docker run --name buildkitd -d --privileged -p 1234:1234 moby/buildkit:rootless --addr tcp://0.0.0.0:1234 ``` `docker run` requires `--privileged` but the BuildKit daemon is executed as a normal user. diff --git a/frontend/dockerfile/docs/experimental.md b/frontend/dockerfile/docs/experimental.md new file mode 100644 index 000000000000..47dc6ae77f91 --- /dev/null +++ b/frontend/dockerfile/docs/experimental.md @@ -0,0 +1,129 @@ +# Dockerfile frontend experimental syntaxes + +## Note for Docker users + +If you are using Docker v18.06 or later, BuildKit mode can be enabled by setting `export DOCKER_BUILDKIT=1` on the client side. +Docker v18.06 also requires the daemon to be [running in experimental mode](https://docs.docker.com/engine/reference/commandline/dockerd/#description). + +You need to use `docker build` CLI instead of `buildctl` CLI mentioned in this document. +See [the `docker build` document](https://docs.docker.com/engine/reference/commandline/build/) for the usage. + +## Use experimental Dockerfile frontend +The features mentioned in this document are experimentally available as [`docker/dockerfile-upstream:experimental`](https://hub.docker.com/r/docker/dockerfile-upstream/tags/) image. + +To use the experimental features, the first line of your Dockerfile needs to be `# syntax=docker/dockerfile-upstream:experimental`. +As the experimental syntaxes may change in future revisions, you may want to pin the image to a specific revision. + +See also [#528](https://github.com/moby/buildkit/issues/528) for further information about planned `docker/dockerfile` releases. + +## Experimental syntaxes + +### `RUN --mount=type=bind` (the default mount type) + +This mount type allows binding directories (read-only) in the context or in an image to the build container. + +|Option |Description| +|---------------------|-----------| +|`target` (required) | Mount path.| +|`source` | Source path in the `from`. Defaults to the root of the `from`.| +|`from` | Build stage or image name for the root of the source. Defaults to the build context.| + + +### `RUN --mount=type=cache` + +This mount type allows the build container to cache directories for compilers and package managers. + +|Option |Description| +|---------------------|-----------| +|`id` | Optional ID to identify separate/different caches| +|`target` (required) | Mount path.| +|`ro`,`readonly` | Read-only if set.| +|`sharing` | One of `shared`, `private`, or `locked`. Defaults to `shared`. A `shared` cache mount can be used concurrently by multiple writers. `private` creates a new mount if there are multiple writers. `locked` pauses the second writer until the first one releases the mount.| + + +#### Example: cache Go packages + +```dockerfile +# syntax = docker/dockerfile-upstream:experimental +FROM golang +... +RUN --mount=type=cache,target=/root/.cache/go-build go build ... +``` + +#### Example: cache apt packages + +```dockerfile +# syntax = docker/dockerfile-upstream:experimental +FROM ubuntu +RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache +RUN --mount=type=cache,target=/var/cache/apt --mount=type=cache,target=/var/lib/apt \ + apt update && apt install -y gcc +``` + +### `RUN --mount=type=tmpfs` + +This mount type allows mounting tmpfs in the build container. + +|Option |Description| +|---------------------|-----------| +|`target` (required) | Mount path.| + + +### `RUN --mount=type=secret` + +This mount type allows the build container to access secure files such as private keys without baking them into the image. + +|Option |Description| +|---------------------|-----------| +|`id` | ID of the secret. Defaults to basename of the target path.| +|`target` | Mount path. Defaults to `/run/secrets/` + `id`.| +|`required` | If set to `true`, the instruction errors out when the secret is unavailable. Defaults to `false`.| + + +#### Example: access to S3 + +```dockerfile +# syntax = docker/dockerfile-upstream:experimental +FROM python:3 +RUN pip install awscli +RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ... +``` + +```console +$ buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. \ + --secret id=aws,src=$HOME/.aws/credentials +``` + +### `RUN --mount=type=ssh` + +This mount type allows the build container to access SSH keys via SSH agents, with support for passphrases. + +|Option |Description| +|---------------------|-----------| +|`id` | ID of SSH agent socket or key. Defaults to "default".| +|`target` | SSH agent socket path. Defaults to `/run/buildkit/ssh_agent.${N}`.| +|`required` | If set to `true`, the instruction errors out when the key is unavailable. Defaults to `false`.| + + +#### Example: access to Gitlab + +```dockerfile +# syntax = docker/dockerfile-upstream:experimental +FROM alpine +RUN apk add --no-cache openssh-client +RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts +RUN --mount=type=ssh ssh git@gitlab.com | tee /hello +# "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here +``` + +```console +$ eval $(ssh-agent) +$ ssh-add ~/.ssh/id_rsa +(Input your passphrase here) +$ buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. \ + --ssh default=$SSH_AUTH_SOCK +``` + +You can also specify a path to `*.pem` file on the host directly instead of `$SSH_AUTH_SOCK`. +However, pem files with passphrases are not supported. +