[question] - executor can't create pid

[OferE]

Nomad version

Nomad v0.5.0

Operating system and Environment details

I am using raw_exec and sucessfully running docker without isolation. I am getting weired logs in the executor

Question

Can some one explain what am i doing wrong/what is the meaning of these lines in the executor logs?
my executor logs are filled with thousands of these:
016/12/14 14:48:39 [DEBUG] executor: unable to create new process with pid: 30390
2016/12/14 14:48:40 [DEBUG] executor: unable to create new process with pid: 30390
2016/12/14 14:48:41 [DEBUG] executor: unable to create new process with pid: 30390
2016/12/14 14:48:42 [DEBUG] executor: unable to create new process with pid: 30390
2016/12/14 14:48:44 [DEBUG] executor: unable to create new process with pid: 30405
2016/12/14 14:48:45 [DEBUG] executor: unable to create new process with pid: 30405
2016/12/14 14:48:46 [DEBUG] executor: unable to create new process with pid: 30405
2016/12/14 14:48:47 [DEBUG] executor: unable to create new process with pid: 30405

Thanks!

[diptanu]

@OferE These are the logs from the code which scans pids and gets their resource utilization for the raw exec driver. And these are thread ids for which we can't get the resource utilization. We can look into this and find a way to not log them.

[OferE]

thank u so much!
Do u need my assistance on reproducing this? If so - tomorrow i'll post here how to reproduce.

[OferE]

Instructions to reproduce:
The nomad version is 0.5
nomad is running as root.

This is the executable i use to run docker in a non isolated way:
(I place it under /opt/nomad_docker_run.sh in my clients)

#!/bin/bash
# handler for the signals sent from nomad to stop the container
my_exit() 
{
   echo "killing $CID"
   docker stop --time=5 $CID # try to stop it gracefully
   docker rm -f $CID # remove the stopped container 
}

trap 'my_exit; exit' SIGHUP SIGTERM SIGINT
echo `env`
# Building docker run command
CMD="docker run -d --name ${NOMAD_TASK_NAME}-${NOMAD_ALLOC_ID}"
for a in "$@"; do
   CMD="$CMD $a"
done

echo docker wrapper: the docker command that will run is: $CMD
echo from here on it is the docker output:
echo 
# actually running the command
CID=`$CMD`

# docker logs is printed in the background
docker logs -f $CID &

# allows the process to listen to signals every 3 seconds
while : 
   do 
      sleep 1
   done

This is an example of a job file that creates the problem:

# dummy image
job "dummy" {
  datacenters = ["dc1"]
  type = "system"
  constraint {
    operator  = "distinct_hosts"
    value     = "true"
  }
  update {
    stagger = "10s"
    max_parallel = 1
  }
  group "dummy" {
    restart {
      attempts = 10
      interval = "5m"
      delay = "25s"
      mode = "delay"
    }

    task "nginx" {
      driver = "raw_exec"
      kill_timeout = "45s"
      config {
        command = "/opt/nomad_docker_run.sh"
        args = ["--network=host",
                "nginx:latest",
                "sleep 100"]
      }
    }
  }
}

This is the head of the executor log in one of the clients (again the debug message appears thousands of time):

2016/12/15 05:09:45 [DEBUG] executor: launching command /opt/nomad_docker_run.sh --network=host nginx:latest sleep 100
2016/12/15 05:09:45 [INFO] executor: registering services
2016/12/15 05:10:03 [DEBUG] executor: unable to create new process with pid: 3453
2016/12/15 05:10:04 [DEBUG] executor: unable to create new process with pid: 3453
2016/12/15 05:10:06 [DEBUG] executor: unable to create new process with pid: 3574
2016/12/15 05:10:07 [DEBUG] executor: unable to create new process with pid: 3574
2016/12/15 05:10

I hope this helps,
Thanks!

[dadgar]

@OferE To be clear there isn't a bug. These are just verbose debug messages.

[OferE]

I know - the raw_exec works great for me.
I do care about these messages since this file get such line every second per container on the host which might fill my volume.

[OferE]

now that v.0.5.1isout ill close it. ill reopen if reproduced.
Thanks!

[OferE]

Hi - i just verified that this problem is still happening in v0.5.1 as well.

[OferE]

thank u so much for solving this. highly appreciated!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.