nginx -s reload not working as expected on alpine versions

[keevan]

Hi there,

Hopefully I'm reporting this in the appropriate location, but I'm having an issue where I am experiencing some 'downtime' whenever I reload the nginx configurations using nginx -s reload on alpine images.

I've tested this on nginx:alpine-mainline and it seems

The basic setup is to use an nginx image as a reverse proxy, pointing to another web service (e.g. another nginx with a static page).
When the nginx acting as the reverse proxy's configuration is reloaded via the nginx -s reload command, the page fails to load (presumably because the process is no longer running?), and starts up again after a few seconds.

This behaviour is NOT an issue when I use the full nginx image (non-alpine), and running nginx -s reload causes no downtime as expected.

Here is a snippet of what the commands return when I run it using the full image (first block), and the alpine one (the one in the coloured box)

I'm not 100% sure what these numbers mean (process id??) but on the alpine one, it's always the same, and on the full nginx image, it's increasing as one might expect.

This is unfortunate because I am not able to utilize the alpine image for my needs, but would be good to understand what might be the cause of this issue. I've tried searching but couldn't find a similar issue.

Environment:

• I'm running this on armv6 architecture ( e.g. raspberry pi )
• nginx:mainline-alpine is used to test this feature, and it is NOT working. This is the image I would like to have this fixed for.
• nginx:mainline is used to test this feature, and it IS working as expected.
• Reverse proxy has a single server block with a proxy_pass to the other nginx service - let's call it the app
• The nginx serving the "app" is serving a single static html page, e.g.:

COPY ./index.html /usr/share/nginx/html/index.html

[thresheek]

I can't reproduce this on amd64 or aarch64, and I don't have an armv6 device to test:

$ docker run -d nginx:mainline-alpine
5ed2dd72c4249359d9bf6bc26b591111f0d7e3bf2b3063f7058cafd3dcc6b759
$ docker exec 5ed2dd nginx -s reload
2021/09/21 08:30:18 [notice] 41#41: signal process started
$ docker exec 5ed2dd nginx -s reload
2021/09/21 08:30:19 [notice] 55#55: signal process started
$ docker exec 5ed2dd nginx -s reload
2021/09/21 08:30:19 [notice] 69#69: signal process started

Can you post the full steps on how to reproduce it? e.g. if you're creating another image on top, etc.

[keevan]

Ah, so I dug into the issue a bit more. It's very strange why the armv6 image behaves like this.

I took this one from dockerhub (https://hub.docker.com/layers/nginx/library/nginx/mainline-alpine/images/sha256-826624c15f5e49e591d80f3e0c696f92a2d5967b989017572fe241edac294a2a?context=explore) and made sure I used the latest available for my architecture.

pi@home:~ $ docker run -d nginx@sha256:826624c15f5e49e591d80f3e0c696f92a2d5967b989017572fe241edac294a2a
97f6b66778b41781764f1bf0fea28a0e80b4de7dd3a79caec4c724333b8f0e35
pi@home:~ $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
97f6b66778b4        nginx               "/docker-entrypoint.…"   13 seconds ago      Up 7 seconds        80/tcp              trusting_bhabha
pi@home:~ $ docker exec 97f6b66778b4 nginx -s reload
2071/04/18 03:36:32 [notice] 32#32: signal process started
pi@home:~ $ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
pi@home:~ $ 

I think the previous attempts were via docker-compose, specifically the line

restart: unless stopped

was why I thought it wasn't working properly. In actuality, it seems like the process dies and the container is stopped (which is why the original example showed the same number on the next run)

My docker-compose.yml file only has an nginx service running with the nginx:mainline-alpine image. The only special bit is the restart: unless-stopped which in a way helps reproduce the issue.

Hope that helps.

Also tried again without it showing in the background to see the output (also noticing my clock is out of sync..)

[thresheek]

That behaviour seems strangely similar to #517 - can you test it the same way as in #517 (comment) for instance?

Are you using the latest docker engine version available?

[keevan]

That behaviour seems strangely similar to #517 - can you test it the same way as in #517 (comment) for instance?

Sure thing. I'll give that a read and test

Are you using the latest docker engine version available?

Ah good mention. I just checked, it's running Docker version 19.03.6, build 369ce74 which I believe is out out of date (apt upgrade shows docker-ce can be updated).

I'll give this a go since it seems like the path of least resistance. Thanks for the feedback @thresheek

[keevan]

This is the results after a docker update (sudo apt upgrade docker)

pi@sierra:~ $ docker --version
Docker version 20.10.8, build 3967b7d
pi@sierra:~ $ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
pi@sierra:~ $ docker run -d nginx@sha256:826624c15f5e49e591d80f3e0c696f92a2d5967b989017572fe241edac294a2a
f46441164d61fd541378de52f96c66a91b75a2ed2a7bdda4b081b7029bd855d5
pi@sierra:~ $ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS         PORTS     NAMES
f46441164d61   nginx     "/docker-entrypoint.…"   17 seconds ago   Up 4 seconds   80/tcp    trusting_shirley
pi@sierra:~ $ docker exec f46441164d61 nginx -s reload
2071/06/18 09:37:04 [notice] 33#33: signal process started
pi@sierra:~ $ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
pi@sierra:~ $

So it appears to be the same as before.

[keevan]

I randomly chose an older image (that actually downloaded), and it seems the issue is no more! Unfortunately, it does mean it's running on a older version of software in general

pi@sierra:~ $ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
pi@sierra:~ $ docker run -d nginx@sha256:4e502d6dd0a9e0f0e0c6d3604387c8a8c24d3cb25811aebca4fb466c785bf840
Unable to find image 'nginx@sha256:4e502d6dd0a9e0f0e0c6d3604387c8a8c24d3cb25811aebca4fb466c785bf840' locally
docker: Error response from daemon: received unexpected HTTP status: 500 Internal Server Error.
See 'docker run --help'.
pi@sierra:~ $ docker run -d nginx@sha256:9ca8cc0d585996e474d680b3bd8e9b48156815538ea3b37a98293f3039115ead
Unable to find image 'nginx@sha256:9ca8cc0d585996e474d680b3bd8e9b48156815538ea3b37a98293f3039115ead' locally
docker: Error response from daemon: received unexpected HTTP status: 500 Internal Server Error.
See 'docker run --help'.
pi@sierra:~ $ docker run -d nginx@sha256:2ba87319e603f89471a23b95f90d1635df8e5b7366c65afdda9228b3ffb4528d
Unable to find image 'nginx@sha256:2ba87319e603f89471a23b95f90d1635df8e5b7366c65afdda9228b3ffb4528d' locally
docker.io/library/nginx@sha256:2ba87319e603f89471a23b95f90d1635df8e5b7366c65afdda9228b3ffb4528d: Pulling from library/nginx
ecf664be551d: Pull complete
e68b1c905652: Pull complete
Digest: sha256:2ba87319e603f89471a23b95f90d1635df8e5b7366c65afdda9228b3ffb4528d
Status: Downloaded newer image for nginx@sha256:2ba87319e603f89471a23b95f90d1635df8e5b7366c65afdda9228b3ffb4528d
90ff196b551991294d10244b99ad9c66210c893ce5dd6a55f9f1f247a82bd1a9
pi@sierra:~ $ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED              STATUS          PORTS     NAMES
90ff196b5519   nginx     "nginx -g 'daemon of…"   About a minute ago   Up 58 seconds   80/tcp    hopeful_saha
pi@sierra:~ $ docker exec 90ff196b5519 nginx -s reload
2021/09/21 12:16:43 [notice] 8#8: signal process started
pi@sierra:~ $ docker exec 90ff196b5519 nginx -s reload
2021/09/21 12:16:50 [notice] 15#15: signal process started
pi@sierra:~ $ docker exec 90ff196b5519 nginx -s reload
2021/09/21 12:16:53 [notice] 22#22: signal process started
pi@sierra:~ $

The one I landed on that worked was this one:
https://hub.docker.com/layers/nginx/library/nginx/1.17.6-alpine/images/sha256-2ba87319e603f89471a23b95f90d1635df8e5b7366c65afdda9228b3ffb4528d?context=explore

[thresheek]

That just confirms that the issue is similar to #517, and older alpine runtime is not affected by the lack of support in docker/libseccomp on the host.

[keevan]

That just confirms that the issue is similar to #517, and older alpine runtime is not affected by the lack of support in docker/libseccomp on the host.

Tyvm, your reply in particular with regards to libseccomp helped me resolve my issue, particularly finding my way to this comment:
alpinelinux/docker-alpine#135

So to sum up, libseccomp2 needs an update on the rasp pi (running raspbian), and on raspbian it has outdated apt sources:

sudo apt install libseccomp2
...
libseccomp2 is already the newest version (2.3.3-4). # which doesn't seem to play nice with newer alpine images

so @RouxAntoine summed it up nicely by providing the instructions, in edit 2

Solution for future readers - of course read up supporting information before installing this

wget http://raspbian.raspberrypi.org/raspbian/pool/main/libs/libseccomp/libseccomp2_2.5.1-1+rpi1_armhf.deb 
sudo dpkg -i libseccomp2_2.5.1-1+rpi1_armhf.deb

Honorable mentions
@nickwest and @mysystem32 also alluded to where the updated packages could be found, however installing those on my pi did not work happy days. Causing things like this to appear after using apt, so I eventually went with one more tailored for the rasp pi.

Processing triggers for man-db (2.8.5-2) ...
Illegal instruction

and this to appear after running an affected docker image:

$ docker run -d nginx:1.19-alpine
ecf7bc27b48fd57a978acb8556272fe00fba68690b73ba02b07f89bc31ae3f2e
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/ecf7bc27b48fd57a978acb8556272fe00fba68690b73ba02b07f89bc31ae3f2e/log.json: no such file or directory): runc did not terminate successfully: unknown.

As this issue is on my end resolved, and nothing to do with this repository in particular, I'm happy to close this one off.

Thanks again @thresheek