[nats] Adding alpine3.14-nr "non root user" [wip]

[kozlovic]

This is test pointing to a work branch that adds a directory for
alpine3.14 with a different Dockerfile that has a "nats" user
and will run the server as non-root.

Signed-off-by: Ivan Kozlovic ivan@synadia.com

[github-actions]

Diff for 9aedbf3:

diff --git a/_bashbrew-cat b/_bashbrew-cat
index 866289e..2d8a54d 100644
--- a/_bashbrew-cat
+++ b/_bashbrew-cat
@@ -1,12 +1,17 @@
 Maintainers: Derek Collison <derek@synadia.com> (@derekcollison), Ivan Kozlovic <ivan@synadia.com> (@kozlovic), Waldemar Salinas <wally@synadia.com> (@wallyqs), Jaime Piña <jaime@synadia.com> (@variadico)
 GitRepo: https://github.com/nats-io/nats-docker.git
-GitFetch: refs/heads/main
-GitCommit: edcf3e2b3b3b909cca78069f1790bb3e9edc0851
+GitFetch: refs/heads/test_nr
+GitCommit: f620eaee77912568b490832647a943ec886af60e
 
 Tags: 2.6.4-alpine3.14, 2.6-alpine3.14, 2-alpine3.14, alpine3.14, 2.6.4-alpine, 2.6-alpine, 2-alpine, alpine
 Architectures: amd64, arm32v6, arm32v7, arm64v8
 Directory: 2.6.4/alpine3.14
 
+Tags: 2.6.4-alpine3.14-nr, 2.6-alpine3.14-nr, 2-alpine3.14-nr, alpine3.14-nr, 2.6.4-alpine-nr, 2.6-alpine-nr, 2-alpine-nr, alpine-nr
+SharedTags: 2.6.4, 2.6, 2, latest
+Architectures: amd64, arm32v6, arm32v7, arm64v8
+Directory: 2.6.4/alpine3.14-nr
+
 Tags: 2.6.4-nanoserver-1809, 2.6-nanoserver-1809, 2-nanoserver-1809, nanoserver-1809
 SharedTags: 2.6.4-nanoserver, 2.6-nanoserver, 2-nanoserver, nanoserver, 2.6.4, 2.6, 2, latest
 Architectures: windows-amd64
diff --git a/_bashbrew-list b/_bashbrew-list
index 073c68d..1b743fb 100644
--- a/_bashbrew-list
+++ b/_bashbrew-list
@@ -1,6 +1,8 @@
 nats:2
 nats:2-alpine
 nats:2-alpine3.14
+nats:2-alpine3.14-nr
+nats:2-alpine-nr
 nats:2-linux
 nats:2-nanoserver
 nats:2-nanoserver-1809
@@ -11,6 +13,8 @@ nats:2-windowsservercore-ltsc2016
 nats:2.6
 nats:2.6-alpine
 nats:2.6-alpine3.14
+nats:2.6-alpine3.14-nr
+nats:2.6-alpine-nr
 nats:2.6-linux
 nats:2.6-nanoserver
 nats:2.6-nanoserver-1809
@@ -21,6 +25,8 @@ nats:2.6-windowsservercore-ltsc2016
 nats:2.6.4
 nats:2.6.4-alpine
 nats:2.6.4-alpine3.14
+nats:2.6.4-alpine3.14-nr
+nats:2.6.4-alpine-nr
 nats:2.6.4-linux
 nats:2.6.4-nanoserver
 nats:2.6.4-nanoserver-1809
@@ -30,6 +36,8 @@ nats:2.6.4-windowsservercore-1809
 nats:2.6.4-windowsservercore-ltsc2016
 nats:alpine
 nats:alpine3.14
+nats:alpine3.14-nr
+nats:alpine-nr
 nats:latest
 nats:linux
 nats:nanoserver
diff --git a/nats_alpine/Dockerfile b/nats_alpine-nr/Dockerfile
similarity index 86%
copy from nats_alpine/Dockerfile
copy to nats_alpine-nr/Dockerfile
index 25b3929..89abf9a 100644
--- a/nats_alpine/Dockerfile
+++ b/nats_alpine-nr/Dockerfile
@@ -29,5 +29,10 @@ RUN set -eux; \
 COPY nats-server.conf /etc/nats/nats-server.conf
 COPY docker-entrypoint.sh /usr/local/bin
 EXPOSE 4222 8222 6222
+RUN adduser -g '' -h / -s /bin/ash -H -D nats
+RUN chown root:nats /etc/nats/nats-server.conf /usr/local/bin/docker-entrypoint.sh
+RUN chmod 640 /etc/nats/nats-server.conf
+RUN chmod 750 /usr/local/bin/docker-entrypoint.sh
+USER nats
 ENTRYPOINT ["docker-entrypoint.sh"]
 CMD ["nats-server", "--config", "/etc/nats/nats-server.conf"]
diff --git a/nats_alpine/docker-entrypoint.sh b/nats_alpine-nr/docker-entrypoint.sh
similarity index 100%
copy from nats_alpine/docker-entrypoint.sh
copy to nats_alpine-nr/docker-entrypoint.sh
diff --git a/nats_alpine/nats-server.conf b/nats_alpine-nr/nats-server.conf
similarity index 100%
copy from nats_alpine/nats-server.conf
copy to nats_alpine-nr/nats-server.conf

Relevant Maintainers:

• nats: @derekcollison @kozlovic @wallyqs @variadico

[yosifkit]

Each separate RUN that modifies the files from the COPY is creating another copy of the file in the image layers; although it is small, it is best practice to not cause duplication.

For running as a non-root user we usually solve that by adjusting the regular image to allow running as any arbitrary user (like docker-library/cassandra#48) and default to stepping down from root "as soon as possible". The main part being wider permissions to allow any user to run the process and read config (and write where necessary). We often use gosu to step down after initial filesystem setup; I don't think nats has that requirement and could just USER always?

This ensures that even if running on a platform like open shift, that randomizes the user ID for every container, it will still be able to run

[kozlovic]

@yosifkit I am sorry, I don't think I fully understand. You wrote: "I don't think nats has that requirement and could just USER always?".

Are you saying that we could have the original alpine-3.14/Dockerfile with simply this section added?:

--- a/2.6.4/alpine3.14/Dockerfile
+++ b/2.6.4/alpine3.14/Dockerfile
@@ -29,5 +29,10 @@ RUN set -eux; \
 COPY nats-server.conf /etc/nats/nats-server.conf
 COPY docker-entrypoint.sh /usr/local/bin
 EXPOSE 4222 8222 6222
+RUN adduser -g '' -h / -s /bin/ash -H -D nats
+RUN chown root:nats /etc/nats/nats-server.conf /usr/local/bin/docker-entrypoint.sh
+RUN chmod 640 /etc/nats/nats-server.conf
+RUN chmod 750 /usr/local/bin/docker-entrypoint.sh
+USER nats
 ENTRYPOINT ["docker-entrypoint.sh"]
 CMD ["nats-server", "--config", "/etc/nats/nats-server.conf"]

Note that NATS Server can now be run as a persistence layer (JetStream) and so I am not sure what would happen if the previous docker image was used and assets on files created and now starting the new image with "nats" user.
Also, what would happen for TLS certificates and so forth?

[yosifkit]

Are you saying that we could have the original alpine-3.14/Dockerfile with simply this section added?

Not that exact bit because of the file duplication. I was hoping something similar could be achieved but it seems that nats has more persistence than I remembered. For persistence it is more complicated if you want to change the user to non-root, but it should still be possible to adjust the image to let it run as any user without breaking current deployments.

So I think the minimum would be to widen permissions in the image as needed to allow an arbitrary user ID to execute/write/read what is necessary. For example, a default persistence directory would allow any user to write there but limit a user id to only able edit/delete their own files, like /tmp (I think 1777). A user account in the image (adduser) is nice for users to know the "default" (docker run can use basically any id number). Then users would add --user to their docker run when they are ready and have adjusted permissions on any docker volume or mounted directory or config as necessary.

You could stop there and just let users choose when to add --user to their deployments. For further backwards compatible improvement, the docker-entrypoint.sh could be used to upgrade users to run nats-server as a non-root user. When the container is run as root (and attempting to run nats-server), the entrypoint script would fix any permissions and ownership and then step down (via gosu or su-exec) to the non-root user before continuing to start nats-server. This is what we do in many database images, like cassandra to minimize what is run as root. This change makes it "impossible" to run nats-server as root without bypassing (--entrypoint) or tricking (docker run nats sh -c 'nats-server') the entrypoint script.

[kozlovic]

@yosifkit Thank you for the detailed answer, yet, I have no idea what that means that I would have to do :-). I am cc'ing some colleagues to see if they have a better understanding of what would need to be done specifically in our Dockerfile: @wallyqs @variadico @philpennock.
Note that the original user request to run without root is this: nats-io/nats-docker#43, and this was changing the existing alpine image. Since I was afraid that we may break existing deployments, this is why I went wit the idea of using a different image altogether in this PR: nats-io/nats-docker#71.

[philpennock]

I think the real issue here is what is the official-images stance on privileged ports and rootless containers? Should we be using setcap to add a privilege to the binary inside the container, or expecting that privilege elevation will be blocked?

At present, the nats image is not binding :443, but we might want to do so for the websockets functionality already in the codebase; if we make the official image run rootless and don't have a good solution for privileged ports, we risk ending up in an awkward situation.

I'm not arguing against rootless: my own stuff runs rootless, but always with more awareness of and integration with the deployment framework so that I can setup ambient capabilities. The issue is what is the docker-library stance on how to manage that.

My suggestion:

1. apk add setpriv
2. Have docker-entrypoint.sh check if the current user is 0; if not, just exec the nats binary
3. If we are root, use a command prefix of: setpriv --ambient-caps +net_bind_service --init-groups --no-new-privs --reuid nats

Since setpriv is already in Alpine (it's part of util-linux), they handle all security updates so it's one less thing to worry about.

[philpennock]

TIL: modern Docker lowers net.ipv4.ip_unprivileged_port_start inside the network namespace used for containers.

So the policy question becomes: are official-images allowed to rely upon that?

[yosifkit]

modern Docker lowers net.ipv4.ip_unprivileged_port_start inside the network namespace used for containers.
are official-images allowed to rely upon that?

Sure. Users on older Docker daemons can always add --sysctl net.ipv4.ip_unprivileged_port_start=0 to their docker run (or maybe custom config to have nats use custom higher ports?). We rely on these for the haproxy image.

As far as using setcap on the binary, from what I recall extended attributes aren't always preserved on some docker storage backends.

[tianon]

(Going to close this now as it seems the conversation has died out -- if it's still useful, please feel free to ask us to reopen 👍)