Skip to content

Commit

Permalink
Merge pull request #710 from AkihiroSuda/dfdoc
Browse files Browse the repository at this point in the history
update docs
  • Loading branch information
tiborvass authored Nov 1, 2018
2 parents fba893e + b5003d5 commit 5aab829
Show file tree
Hide file tree
Showing 3 changed files with 158 additions and 7 deletions.
26 changes: 23 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,24 @@ Read the proposal from https://github.com/moby/moby/issues/32925

Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056cc5317

:information_source: If you are visiting this repo for the usage of experimental Dockerfile features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)`, please refer to [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md).

### Used by

[Moby](https://github.com/moby/moby/pull/37151)
[Moby & Docker](https://github.com/moby/moby/pull/37151)

[img](https://github.com/genuinetools/img)

[OpenFaaS Cloud](https://github.com/openfaas/openfaas-cloud)

[container build interface](https://github.com/containerbuilding/cbi)

[Knative Build Templates](https://github.com/knative/build-templates)

[boss](https://github.com/crosbymichael/boss)

[Rio](https://github.com/rancher/rio) (on roadmap)

### Quick start

Dependencies:
Expand Down Expand Up @@ -79,6 +87,7 @@ See [`solver/pb/ops.proto`](./solver/pb/ops.proto) for the format definition.
Currently, following high-level languages has been implemented for LLB:

- Dockerfile (See [Exploring Dockerfiles](#exploring-dockerfiles))
- [Buildpacks](https://github.com/tonistiigi/buildkit-pack)
- (open a PR to add your own language)

For understanding the basics of LLB, `examples/buildkit*` directory contains scripts that define how to build different configurations of BuildKit itself and its dependencies using the `client` package. Running one of these scripts generates a protobuf definition of a build graph. Note that the script itself does not execute any steps of the build.
Expand Down Expand Up @@ -145,6 +154,10 @@ buildctl build --frontend=gateway.v0 --frontend-opt=source=tonistiigi/dockerfile
buildctl build --frontend gateway.v0 --frontend-opt=source=tonistiigi/dockerfile --frontend-opt=context=git://github.com/moby/moby --frontend-opt build-arg:APT_MIRROR=cdn-fastly.deb.debian.org
````

##### Building a Dockerfile with experimental features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)`

See [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md).

### Exporters

By default, the build result and intermediate cache will only remain internally in BuildKit. Exporter needs to be specified to retrieve the result.
Expand Down Expand Up @@ -207,15 +220,22 @@ buildctl debug workers -v

BuildKit can also be used by running the `buildkitd` daemon inside a Docker container and accessing it remotely. The client tool `buildctl` is also available for Mac and Windows.

We provide `buildkitd` container images as [`moby/buildkit`](https://hub.docker.com/r/moby/buildkit/tags/):

* `moby/buildkit:latest`: built from the latest regular [release](https://github.com/moby/buildkit/releases)
* `moby/buildkit:rootless`: same as `latest` but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md)
* `moby/buildkit:master`: built from the master branch
* `moby/buildkit:master-rootless`: same as master but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md)

To run daemon in a container:

```
docker run -d --privileged -p 1234:1234 tonistiigi/buildkit --addr tcp://0.0.0.0:1234
docker run -d --privileged -p 1234:1234 moby/buildkit:latest --addr tcp://0.0.0.0:1234
export BUILDKIT_HOST=tcp://0.0.0.0:1234
buildctl build --help
```

The `tonistiigi/buildkit` image can be built locally using the Dockerfile in `./hack/dockerfiles/test.Dockerfile`.
The images can be also built locally using `./hack/dockerfiles/test.Dockerfile` (or `./hack/dockerfiles/test.buildkit.Dockerfile` if you already have BuildKit).

### Opentracing support

Expand Down
10 changes: 6 additions & 4 deletions docs/rootless.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

Requirements:
- runc `a00bf0190895aa465a5fbed0268888e2c8ddfe85` (Oct 15, 2018) or later
- Some distros such as Debian (excluding Ubuntu) and Arch Linux require `echo 1 > /proc/sys/kernel/unprivileged_userns_clone`
- `newuidmap` and `newgidmap` need to be installed on the host. These commands are provided by the `uidmap` package.
- Some distros such as Debian (excluding Ubuntu) and Arch Linux require `sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone"`.
- RHEL/CentOS 7 requires `sudo sh -c "echo 28633 > /proc/sys/user/max_user_namespaces"`. You may also need `sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"`.
- `newuidmap` and `newgidmap` need to be installed on the host. These commands are provided by the `uidmap` package. For RHEL/CentOS 7, RPM is not officially provided but available at https://copr.fedorainfracloud.org/coprs/vbatts/shadow-utils-newxidmap/ .
- `/etc/subuid` and `/etc/subgid` should contain >= 65536 sub-IDs. e.g. `penguin:231072:65536`.
- To run in a Docker container with non-root `USER`, `docker run --privileged` is still required. See also Jessie's blog: https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/

Expand Down Expand Up @@ -56,9 +57,10 @@ $ build-using-dockerfile --buildkit-addr unix:///run/user/1001/buildkit/buildkit

## Set up (using a container)

Docker image is available as [`moby/buildkit:rootless`](https://hub.docker.com/r/moby/buildkit/tags/).

```
$ docker build -t buildkit-rootless --target rootless -f hack/dockerfiles/test.Dockerfile .
$ docker run --name buildkitd -d --privileged -p 1234:1234 buildkit-rootless --addr tcp://0.0.0.0:1234
$ docker run --name buildkitd -d --privileged -p 1234:1234 moby/buildkit:rootless --addr tcp://0.0.0.0:1234
```

`docker run` requires `--privileged` but the BuildKit daemon is executed as a normal user.
Expand Down
129 changes: 129 additions & 0 deletions frontend/dockerfile/docs/experimental.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Dockerfile frontend experimental syntaxes

## Note for Docker users

If you are using Docker v18.06 or later, BuildKit mode can be enabled by setting `export DOCKER_BUILDKIT=1` on the client side.
Docker v18.06 also requires the daemon to be [running in experimental mode](https://docs.docker.com/engine/reference/commandline/dockerd/#description).

You need to use `docker build` CLI instead of `buildctl` CLI mentioned in this document.
See [the `docker build` document](https://docs.docker.com/engine/reference/commandline/build/) for the usage.

## Use experimental Dockerfile frontend
The features mentioned in this document are experimentally available as [`docker/dockerfile-upstream:experimental`](https://hub.docker.com/r/docker/dockerfile-upstream/tags/) image.

To use the experimental features, the first line of your Dockerfile needs to be `# syntax=docker/dockerfile-upstream:experimental`.
As the experimental syntaxes may change in future revisions, you may want to pin the image to a specific revision.

See also [#528](https://github.com/moby/buildkit/issues/528) for further information about planned `docker/dockerfile` releases.

## Experimental syntaxes

### `RUN --mount=type=bind` (the default mount type)

This mount type allows binding directories (read-only) in the context or in an image to the build container.

|Option |Description|
|---------------------|-----------|
|`target` (required) | Mount path.|
|`source` | Source path in the `from`. Defaults to the root of the `from`.|
|`from` | Build stage or image name for the root of the source. Defaults to the build context.|


### `RUN --mount=type=cache`

This mount type allows the build container to cache directories for compilers and package managers.

|Option |Description|
|---------------------|-----------|
|`id` | Optional ID to identify separate/different caches|
|`target` (required) | Mount path.|
|`ro`,`readonly` | Read-only if set.|
|`sharing` | One of `shared`, `private`, or `locked`. Defaults to `shared`. A `shared` cache mount can be used concurrently by multiple writers. `private` creates a new mount if there are multiple writers. `locked` pauses the second writer until the first one releases the mount.|


#### Example: cache Go packages

```dockerfile
# syntax = docker/dockerfile-upstream:experimental
FROM golang
...
RUN --mount=type=cache,target=/root/.cache/go-build go build ...
```

#### Example: cache apt packages

```dockerfile
# syntax = docker/dockerfile-upstream:experimental
FROM ubuntu
RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
RUN --mount=type=cache,target=/var/cache/apt --mount=type=cache,target=/var/lib/apt \
apt update && apt install -y gcc
```

### `RUN --mount=type=tmpfs`

This mount type allows mounting tmpfs in the build container.

|Option |Description|
|---------------------|-----------|
|`target` (required) | Mount path.|


### `RUN --mount=type=secret`

This mount type allows the build container to access secure files such as private keys without baking them into the image.

|Option |Description|
|---------------------|-----------|
|`id` | ID of the secret. Defaults to basename of the target path.|
|`target` | Mount path. Defaults to `/run/secrets/` + `id`.|
|`required` | If set to `true`, the instruction errors out when the secret is unavailable. Defaults to `false`.|


#### Example: access to S3

```dockerfile
# syntax = docker/dockerfile-upstream:experimental
FROM python:3
RUN pip install awscli
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ...
```

```console
$ buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. \
--secret id=aws,src=$HOME/.aws/credentials
```

### `RUN --mount=type=ssh`

This mount type allows the build container to access SSH keys via SSH agents, with support for passphrases.

|Option |Description|
|---------------------|-----------|
|`id` | ID of SSH agent socket or key. Defaults to "default".|
|`target` | SSH agent socket path. Defaults to `/run/buildkit/ssh_agent.${N}`.|
|`required` | If set to `true`, the instruction errors out when the key is unavailable. Defaults to `false`.|


#### Example: access to Gitlab

```dockerfile
# syntax = docker/dockerfile-upstream:experimental
FROM alpine
RUN apk add --no-cache openssh-client
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
# "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here
```

```console
$ eval $(ssh-agent)
$ ssh-add ~/.ssh/id_rsa
(Input your passphrase here)
$ buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. \
--ssh default=$SSH_AUTH_SOCK
```

You can also specify a path to `*.pem` file on the host directly instead of `$SSH_AUTH_SOCK`.
However, pem files with passphrases are not supported.

0 comments on commit 5aab829

Please sign in to comment.