user attribute disregard membership in other groups

[dvusboy]

Nomad version

Nomad v0.4.1

I'm aware this version is very old, but if my suspicion is correct, the implementation hasn't changed since.

Operating system and Environment details

kernel: 3.10.0-514.2.2.el7.x86_64
os-release:
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Issue

I'm using the raw_exec driver, and if I specify a user attribute for a task I run into problem performing docker operations. From the executor.out file, this was reported the task command was run with the specified user, but we see error messages like this:

Cannot connect to the Docker daemon. Is the docker daemon running on this host? docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. See 'docker run --help'.

The user in the job specification does belong to the group docker, but it is not the primary group the user belongs to, that'd be users. If it matters, that user is not part of /etc/passwd, but registered in a directory service.

Reproduction steps

Create a user, say foo with primary group anything but docker. Add said user to the docker group. Launch a simple batch job that does not use the docker driver and perform a simple docker operation like docker images. In the example belong, the user foo would have attributes like:

[root@test /]# foo
uid=1000(foo) gid=100(users) groups=100(users),992(docker)

Nomad Server logs (if appropriate)

N/A

Nomad Client logs (if appropriate)

N/A

Job file (if appropriate)

This is a simple job file that can reproduce the problem, given the user foo exists on the client node and has the above group membership.

job "user-bug" {
    region = "global"
    datacenters = [ "rnd" ]
    type = "batch"

    constraint {
        attribute = "${node.class}"
        value = "test"
    }

    group "bug" {
        count = 1

        task "docker-op" {
            driver = "raw_exec"
            user = "foo"
            config {
                command = "docker"
                args = [ "images" ]
            }
            resources {
                cpu = 100
                memory = 100
                disk = 200
            }
        }
    }
}

The job will repeatedly fail and the follow messages can be seen in the docker-op.stderr.0 file:

[root@test logs]# cat docker-op.stderr.0
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
...

While the docker-op-executor.out file looks like this:

[root@test docker-op]# cat docker-op-executor.out
2017/01/06 09:54:47 [DEBUG] executor: launching command docker images
2017/01/06 09:54:47 [DEBUG] executor: running command as kcwong
2017/01/06 09:54:47 [INFO] executor: registering services
2017/01/06 09:54:47 [INFO] executor: de-registering services and shutting down consul service
...

[dvusboy]

My suspicion is the implementation of UniversalExecutor.runAs() for Linux sets the gid to the user's primary group. Would not setting it fix the problem?

[dvusboy]

Any thought on this?

[dadgar]

@dvusboy Let me know if you want a build to test that PR or if you just want to wait till 0.6.0 RC

[dvusboy]

@dadgar I can wait until 0.6.0 RC. Thank you for addressing this.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.