Skip to content

Commit

Permalink
docs: update docs for linux capabilities in exec/java/docker drivers
Browse files Browse the repository at this point in the history
Update docs for allow_caps, cap_add, cap_drop in exec/java/docker driver
pages. Also update upgrade guide with guidance on new default linux
capabilities for exec and java drivers.
  • Loading branch information
shoenig committed May 15, 2021
1 parent 7984182 commit a71081c
Show file tree
Hide file tree
Showing 5 changed files with 158 additions and 30 deletions.
1 change: 1 addition & 0 deletions drivers/shared/capabilities/set.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ type Set struct {
data map[string]nothing
}

// New creates a new Set setting caps as the initial elements.
func New(caps []string) *Set {
m := make(map[string]nothing, len(caps))
for _, c := range caps {
Expand Down
61 changes: 34 additions & 27 deletions website/content/docs/drivers/docker.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -452,30 +452,26 @@ config {
- `cap_add` - (Optional) A list of Linux capabilities as strings to pass directly to
[`--cap-add`](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities).
Effective capabilities (computed from `cap_add` and `cap_drop`) have to match the configured allowlist.
The allowlist can be customized using the [`allow_caps`](#plugin_caps) plugin option key in the client node's configuration.
The allowlist can be customized using the [`allow_caps`][allow_caps] plugin option key in the client node's configuration.
For example:

```hcl
config {
cap_add = [
"SYS_TIME",
]
}
```
```hcl
config {
cap_add = ["net_raw", sys_time"]
}
```

- `cap_drop` - (Optional) A list of Linux capabilities as strings to pass directly to
[`--cap-drop`](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities).
Effective capabilities (computed from `cap_add` and `cap_drop`) have to match the configured allowlist.
The allowlist can be customized using the [`allow_caps`](#plugin_caps) plugin option key in the client node's configuration.
The allowlist can be customized using the [`allow_caps`][allow_caps] plugin option key in the client node's configuration.
For example:

```hcl
config {
cap_drop = [
"MKNOD",
]
}
```
```hcl
config {
cap_drop = ["mknod"]
}
```

- `cpu_hard_limit` - (Optional) `true` or `false` (default). Use hard CPU
limiting instead of soft limiting. By default this is `false` which means
Expand Down Expand Up @@ -797,10 +793,7 @@ plugin "docker" {
}
allow_privileged = false
allow_caps = ["CHOWN", "NET_RAW"]
# allow_caps can also be set to "ALL"
# allow_caps = ["ALL"]
allow_caps = ["chown", "net_raw"]
}
}
```
Expand All @@ -823,13 +816,22 @@ plugin "docker" {
from the Docker engine during an image pull within this timeframe, Nomad will
timeout the request that initiated the pull command. (Minimum of `1m`)

- `allow_caps`<a id="plugin_caps"></a> - A list of allowed Linux capabilities.
Defaults to
`CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP, NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE` which is the list of
capabilities allowed by docker by default, as defined here. Allows the
operator to control which capabilities can be obtained by tasks using cap_add
and cap_drop options. Supports the value "ALL" as a shortcut for allowlisting
all capabilities.
- `allow_caps` - A list of allowed Linux capabilities. Defaults to

```hcl
["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod",
"net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"]
```

which is the same list of capabilities allowed by [docker by default][docker_caps]
(sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities can be obtained
by tasks using [`cap_add`][cap_add] and [`cap_drop`][cap_drop] options. Supports
the value `"all"` as a shortcut for allow-listing all capabilities supported by
the operating system.

!> **Warning:** Allowing more capabilities beyond the default may lead to
undesirable consequences, including untrusted tasks being able to compromise the
host system.

- `allow_runtimes` - defaults to `["runc", "nvidia"]` - A list of the allowed
docker runtimes a task may use.
Expand Down Expand Up @@ -1136,3 +1138,8 @@ Windows is relatively new and rapidly evolving you may want to consult the
[plugin-stanza]: /docs/configuration/plugin
[allocation working directory]: /docs/runtime/environment#task-directories 'Task Directories'
[`auth_soft_fail=true`]: #auth_soft_fail
[cap_add]: /docs/drivers/docker#cap_add
[cap_drop]: /docs/drivers/docker#cap_drop
[no_net_raw]: /docs/upgrade/upgrade-specific#nomad-1-1-0-rc1-1-0-5-0-12-12
[docker_caps]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
[allow_caps]: /docs/drivers/docker#allow_caps
43 changes: 43 additions & 0 deletions website/content/docs/drivers/exec.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,27 @@ be able to access sensitive process information like environment variables.
!> **Warning:** If set to `"host"`, other processes running as the same user will be
able to make use of IPC features, like sending unexpected POSIX signals.

- `cap_add` - (Optional) A list of Linux capabilities to enable for the task.
Effective capabilities (computed from `cap_add` and `cap_drop`) must be a subset
of the allowed capabilities configured with [`allow_caps`][allow_caps].

```hcl
config {
cap_add = ["net_raw", "sys_time"]
}
```

- `cap_drop` - (Optional) A list of Linux capabilities to disable for the task.
Effective capabilities (computed from `cap_add` and `cap_drop`) must be a subset
of the allowed capabilities configured with [`allow_caps`][allow_caps].

```hcl
config {
cap_drop = ["all"]
cap_add = ["chown", "sys_chroot", "mknod"]
}
```

## Examples

To run a binary present on the Node:
Expand Down Expand Up @@ -138,6 +159,23 @@ able to make use of IPC features, like sending unexpected POSIX signals.
for file system isolation without `pivot_root`. This is useful for systems
where the root is on a ramdisk.

- `allow_caps` - A list of allowed Linux capabilities. Defaults to

```hcl
["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod",
"net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"]
```

which is modeled after the capabilities allowed by [docker by default][docker_caps]
(sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities
can be obtained by tasks using [`cap_add`][cap_add] and [`cap_drop`][cap_drop] options.
Supports the value `"all"` as a shortcut for allow-listing all capabilities supported
by the operating system.

!> **Warning:** Allowing more capabilities beyond the default may lead to
undesirable consequences, including untrusted tasks being able to compromise the
host system.

## Client Attributes

The `exec` driver will set the following client attributes:
Expand Down Expand Up @@ -200,3 +238,8 @@ This list is configurable through the agent client

[default_pid_mode]: /docs/drivers/exec#default_pid_mode
[default_ipc_mode]: /docs/drivers/exec#default_ipc_mode
[cap_add]: /docs/drivers/exec#cap_add
[cap_drop]: /docs/drivers/exec#cap_drop
[no_net_raw]: /docs/upgrade/upgrade-specific#nomad-1-1-0-rc1-1-0-5-0-12-12
[allow_caps]: /docs/drivers/exec#allow_caps
[docker_caps]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
43 changes: 43 additions & 0 deletions website/content/docs/drivers/java.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,27 @@ be able to access sensitive process information like environment variables.
!> **Warning:** If set to `"host"`, other processes running as the same user will be
able to make use of IPC features, like sending unexpected POSIX signals.

- `cap_add` - (Optional) A list of Linux capabilities to enable for the task.
Effective capabilities (computed from `cap_add` and `cap_drop`) must be a subset
of the allowed capabilities configured with [`allow_caps`][allow_caps].

```hcl
config {
cap_add = ["net_raw", "sys_time"]
}
```

- `cap_drop` - (Optional) A list of Linux capabilities to disable for the task.
Effective capabilities (computed from `cap_add` and `cap_drop`) must be a subset
of the allowed capabilities configured with [`allow_caps`][allow_caps].

```hcl
config {
cap_drop = ["all"]
cap_add = ["chown", "sys_chroot", "mknod"]
}
```

## Examples

A simple config block to run a Java Jar:
Expand Down Expand Up @@ -138,6 +159,23 @@ be able to access sensitive process information like environment variables.
!> **Warning:** If set to `"host"`, other processes running as the same user will be
able to make use of IPC features, like sending unexpected POSIX signals.

- `allow_caps` - A list of allowed Linux capabilities. Defaults to

```hcl
["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod",
"net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"]
```

which is modeled after the capabilities allowed by [docker by default][docker_caps]
(sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities
can be obtained by tasks using [`cap_add`][cap_add] and [`cap_drop`][cap_drop] options.
Supports the value `"all"` as a shortcut for allow-listing all capabilities supported
by the operating system.

!> **Warning:** Allowing more capabilities beyond the default may lead to
undesirable consequences, including untrusted tasks being able to compromise the
host system.

## Client Requirements

The `java` driver requires Java to be installed and in your system's `$PATH`. On
Expand Down Expand Up @@ -208,3 +246,8 @@ This list is configurable through the agent client

[default_pid_mode]: /docs/drivers/java#default_pid_mode
[default_ipc_mode]: /docs/drivers/java#default_ipc_mode
[cap_add]: /docs/drivers/java#cap_add
[cap_drop]: /docs/drivers/java#cap_drop
[no_net_raw]: /docs/upgrade/upgrade-specific#nomad-1-1-0-rc1-1-0-5-0-12-12
[allow_caps]: /docs/drivers/java#allow_caps
[docker_caps]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
40 changes: 37 additions & 3 deletions website/content/docs/upgrade/upgrade-specific.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,36 @@ these fields.

Connect native tasks running in host networking mode will now have `CONSUL_HTTP_ADDR`
set automatically. Before this was only the case for bridge networking. If an operator
already explicitly set `CONSUL_HTTP_ADDR` then it will not get overriden.
already explicitly set `CONSUL_HTTP_ADDR` then it will not get overridden.

#### Linux capabilities in exec/java

Following the security [remediation][no_net_raw] in Nomad versions 0.12.12, 1.0.5,
and 1.1.0-rc1, the `exec` and `java` task drivers will additionally no longer enable
the following linux capabilities by default:

```
AUDIT_CONTROL AUDIT_READ BLOCK_SUSPEND DAC_READ_SEARCH IPC_LOCK IPC_OWNER LEASE
LINUX_IMMUTABLE MAC_ADMIN MAC_OVERRIDE NET_ADMIN NET_BROADCAST SYS_ADMIN
SYS_BOOT SYSLOG SYS_MODULE SYS_NICE SYS_PACCT SYS_PTRACE SYS_RAWIO SYS_RESOURCE
SYS_TIME SYS_TTY_CONFIG WAKE_ALARM
```

The capabilities now enabled by default are modeled after Docker default [`linux capabilities`]:

```
AUDIT_WRITE CHOWN DAC_OVERRIDE FOWNER FSETID KILL MKNOD NET_BIND_SERVICE
NET_RAW SETFCAP SETGID SETPCAP SETUID SYS_CHROOT
```

A new `allow_caps` plugin configuration parameter for [`exec`][allow_caps_exec]
and [`java`][allow_caps_java] task drivers can be used to restrict the set of
capabilities allowed for use by tasks.

Tasks using the `exec` or `java` task drivers can add or remove desired linux
capabilities using the [`cap_add`][cap_add_exec] and [`cap_drop`][cap_drop_exec]
task configuration options.


#### iptables

Expand All @@ -63,9 +92,9 @@ inserting them as the first rule. This allows better control for user-defined
iptables rules but users who append rules currently should verify that their
rules are being appended in the correct order.

## Nomad 1.1.0, 1.0.5, 0.12.12
## Nomad 1.1.0-rc1, 1.0.5, 0.12.12

Nomad versions 1.1.0, 1.0.5 and 0.12.12 change the behavior of the `docker`, `exec`,
Nomad versions 1.1.0-rc1, 1.0.5 and 0.12.12 change the behavior of the `docker`, `exec`,
and `java` task drivers so that the [`CAP_NET_RAW`] linux capability is disabled
by default. This is one of the [`linux capabilities`] that Docker itself enables
by default, as this capability enables the generation of ICMP packets - used by
Expand Down Expand Up @@ -1111,3 +1140,8 @@ deleted and then Nomad 0.3.0 can be launched.
[`CAP_NET_RAW`]: https://security.stackexchange.com/a/128988
[`linux capabilities`]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
[`allow_caps`]: /docs/drivers/docker#allow_caps
[no_net_raw]: /docs/upgrade/upgrade-specific#nomad-1-1-0-rc1-1-0-5-0-12-12
[allow_caps_exec]: /docs/drivers/exec#allow_caps
[allow_caps_java]: /docs/drivers/java#allow_caps
[cap_add_exec]: /docs/drivers/exec#cap_add
[cap_drop_exec]: /docs/drivers/exec#cap_drop

0 comments on commit a71081c

Please sign in to comment.