Restart unless stopped not working

[orion110217]

I noticed restart: unless-stopped is not working in docker-gen. Do I need to use always instead? Why is it not working?

[m-a-v]

I have the same problem. But also "always" does not work. After restart of the server I have to startup docker-gen manually. How can I find the reason for the exited docker-image? Why does it not restart automatically when I use always?

[jebw]

I'm hitting the same issue - docker-gen container + nginx container - both restart=always - if I restart the docker service, nginx comes back up but docker-gen remains stopped with an exit code of 2

Nothing obvious in the logs

Docker start will get the container running again

[obi12341]

Yep, exactly the same for me. Possible to fix this issue? It is very annoying in a production environment.

[danielkhalilov]

docker update --restart=unless-stopped 7e99ccfb8139 DOES NOT WORK....

[sarukuku]

We’re also experiencing the same issue.

[gghg1989]

Same issue here, every time the server rebooted or after ran my auto backup program. All other containers would be up except docker-gen.

[ghost]

Same here, is it something with nginx.tmpl or latest version of docker ce?

[m-a-v]

Due to problems with letsencrypt-nginx-proxy-companion I removed this component. Since then I didn't have any problems after reboot. Is it possible that letsencrypt-nginx-proxy-companion causes this trouble?

[roopemerikukka]

Has anyone found any solutions to this? I'm facing the same problem. If I reboot the server everything gets up and running except docker-gen which is exited with exit status 2. Related issue: #201

[Michiel-2]

I have the same problem. I assume the docker-letsencrypt-nginx-proxy-companion container most people will use with it, causes the problem. And only when another app/container triggers the letsencrypt container to do it's work. I Have done some tests with different combinations of containers and only when there was a container that triggerd the letsencrypt container, i got a exited (2) after a server-reboot or docker-machine restart. And not always, but around ~75% of the times.
It happens with 0.7.0, 0.7.3 and 0.7.4. Nginx offical image 1.14 and docker-letsencrypt-nginx-proxy-companion:v1.8
Can anybody tell what is outputting the exit code of 2?

[copierrj]

We worked around this problem by wrapping docker-gen in a shell script that executes docker-gen in a loop and traps SIGHUP and SIGTERM signals:

#!/bin/sh

echo container started
echo parameters: $@

hup() {
        echo SIGHUP received

        if [ -z "$docker_gen_pid" ]
        then
                echo no pid: signal ignored
        else
                echo sending signal to pid: $docker_gen_pid
                kill -HUP $docker_gen_pid
        fi

        wait
}

trap hup SIGHUP

term() {
        echo SIGTERM received

        if [ -z "$docker_gen_pid" ]
        then
                exit 0
        else
                echo killing pid: $docker_gen_pid
                kill $docker_gen_pid

                wait
                exit_code=$?
                echo docker-gen terminated with exit code: $exit_code
                exit $exit_code
        fi
}

trap term SIGTERM

while :
do
        echo starting docker-gen...
        /usr/local/bin/docker-gen $@ &
        docker_gen_pid=$!

        wait
        docker_gen_pid=

        echo docker-gen terminated with exit code: $?
        sleep 10
done

[MaksimKlepikov]

I'm facing the same problem. I use https://github.com/evertramos/docker-compose-letsencrypt-nginx-proxy-companion and after server reboot docker-gen doesn't run. At the first reboot nginx-proxy dont work. After docker-compose restart all next reboot doesn't break docker-proxy, but docker-gen anyway doesn't start and i don't know how bad this is.
My last ~20 logs row

2018/07/15 08:20:21 Received event die for container 63026b3a3882
2018/07/15 08:20:26 Debounce minTimer fired
2018/07/15 08:20:26 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2018/07/15 08:21:17 Received event start for container b9e8146e41d2
2018/07/15 08:21:22 Debounce minTimer fired
2018/07/15 08:21:23 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2018/07/15 08:21:44 Received event die for container b9e8146e41d2
2018/07/15 08:21:49 Debounce minTimer fired
2018/07/15 08:21:50 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2018/07/15 09:20:37 Received signal: terminated
2018/07/15 09:20:37 Received signal: terminated
2018/07/15 09:24:24 Generated '/etc/nginx/conf.d/default.conf' from 8 containers
2018/07/15 09:24:24 Sending container 'nginx-web' signal '1'
2018/07/15 09:24:24 Watching docker events
2018/07/15 09:24:24 Received signal: hangup
2018/07/15 09:24:24 Generated '/etc/nginx/conf.d/default.conf' from 8 containers
2018/07/15 09:24:24 Sending container 'nginx-web' signal '1'
2018/07/15 09:24:24 Received signal: hangup
2018/07/15 09:24:25 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2018/07/15 09:31:30 Received signal: terminated
2018/07/15 09:31:30 Received signal: terminated

Now i just set this cronjob:
@reboot sleep 30 && /usr/local/bin/docker-compose -f /root/docker-compose-letsencrypt-nginx-proxy-companion/docker-compose-multiple-networks.yml restart >> /var/log/docker-compose-nginx-after-reboot.log 2>&1
But i think it is need to be doing by docker-compose

[tbpoetke]

Any Progress with this Issue? I have same issue on Ubuntu 18.04 with Docker-CE 18.09.1. I can see it for months now :(

[jowi24]

I'm using CoreOS with Docker version 18.06.1-ce. Also hit by this issue.

[gaswirth]

Same here. Ubuntu 18.04.

[hactar]

We're having the same issue: nginx-gen randomly goes into the "stopped" state, even though we have a restart directive...

[jrd]

Thank you @copierrj . Your wrapper works very good.

I put it in a file called docker-gen-wrapper, made it executable (chmod +x) and my compose service is defined as:

entrypoint: ["/usr/local/bin/docker-gen-wrapper"]
command: ["-notify-sighup", "nginx-proxy", "-watch", "-wait", "5s:30s", "/etc/docker-gen/templates/nginx.tmpl", "/etc/nginx/conf.d/default.conf"]
volumes:
      - ./docker-gen-wrapper:/usr/local/bin/docker-gen-wrapper:ro
…

command is emptied when you define a new entrypoint, so you have to specify both.

[pgassmann]

the restart policy of docker only has effect after 10 seconds.

https://docs.docker.com/config/containers/start-containers-automatically/#restart-policy-details

A restart policy only takes effect after a container starts successfully. In this case, starting successfully means that the container is up for at least 10 seconds and Docker has started monitoring it. This prevents a container which does not start at all from going into a restart loop.

If nginx-gen is faster at startup than nginx, it has an error when connecting to the nginx container and crashes with "Exit 2"
Even with a depends_on policy in docker-compose.yml the problem occurs.

[JoelLinn]

Cause

SIGHUP is used to trigger generation. If the signal however is received before docker-gen is fully initialized the default golang handler will call os.Exit(2).
I fixed this by ignoring SIGHUP until docker-gen is ready.

Fix

You can use the docker image based on my fork: joellinn/docker-gen. It's also build with up to date dependencies.

(I don't fancy advertising forks but this project unfortunately seems to be unmaintained...)

[You-Honey]

Cause

SIGHUP is used to trigger generation. If the signal however is received before docker-gen is fully initialized the default golang handler will call os.Exit(2).
I fixed this by ignoring SIGHUP until docker-gen is ready.

Fix

You can use the docker image based on my fork: joellinn/docker-gen. It's also build with up to date dependencies.

(I don't fancy advertising forks but this project unfortunately seems to be unmaintained...)

Still happens with your image. :(

[JoelLinn]

Under which environment are you running the image? How does it "stop" (Status in docker ps). Is there any hint in the logs?

[wmundev]

Thanks @JoelLinn , your docker image works great, now when i do docker-compose down and docker-compose up -d , all my containers start without any error