proposal: reduced default Linux capabilities

[tgross]

In #10572 we mitigated CVE-2021-32575 by removing the CAP_NET_RAW Linux capability from the docker, exec, and java (on Linux) drivers. We provided some knobs for users in #10600 to give Nomad users fine-grained control over Linux capabilities.

The Nomad developers propose tightening the default capabilities further, and would like to gather feedback from the Nomad user community. Many workloads can get away with a narrow subset of these capabilities. For example, a simple web server with bridge networking can likely get away without any capabilities set. We'd like to hear from you, Nomad community, about capabilities you definitely want for most of your workloads or definitely can do without.

A couple of things to keep in mind in this discussion:

• All capabilities will always be available by configuration. This proposal is only considering changing the default allowed capabilities.
• Because any change may not be backwards compatible for existing workloads, nothing discussed here will ship before Nomad 1.2.0.
• Only changes to the docker driver will impact macOS or Windows, as we don't currently support exec for Windows and the java driver doesn't have exec-style isolation on Windows or macOS yet.

Linux capabilities currently allowed by default:

• AUDIT_WRITE
• CHOWN
• DAC_OVERRIDE
• FOWNER
• FSETID
• KILL
• MKNOD
• NET_BIND_SERVICE
• SETFCAP
• SETGID
• SETPCAP
• SETUID
• SYS_CHROOT

Capabilities disallowed by default

• AUDIT_CONTROL
• AUDIT_READ
• BLOCK_SUSPEND
• DAC_READ_SEARCH
• IPC_LOCK
• IPC_OWNER
• LEASE
• LINUX_IMMUTABLE
• MAC_ADMIN
• MAC_OVERRIDE
• NET_ADMIN
• NET_BROADCAST
• SYS_ADMIN
• SYS_BOOT
• SYSLOG
• SYS_MODULE
• SYS_NICE
• SYS_PACCT
• SYS_PTRACE
• SYS_RAWIO
• SYS_RESOURCE
• SYS_TIME
• SYS_TTY_CONFIG
• WAKE_ALARM

[shoenig]

NCCGroup has a great whitepaper on container security, describing in depth how capabilities / namespaces / access controls can work together.

Just to clarify, "capabilities disallowed by default" by Nomad are all of the capabilities not in the default allow-list; Linux is adding new capabilities from time to time - we don't manage an explicit list of which ones to block.

[frederikbosch]

My guess is that most people using Nomad have limited knowledge on capabilities. So disabling all capabilities might lead to more Github issues. Suppose you run GitlabCE via Docker in Nomad, and by default all caps disabled, I guess it would be quite hard to figure out which capabilities are required. I think running a job with a security profile would be more helpful, like the idea below.

task "task" {
  security {
    profile = "strict" | "lax" | "none" | "custom"
    cap_add = [""]
    cap_remove = [""]
  }
}