Error uploading file: mkstemp(/opt/unit/tmp//req-XXXXXXXX) failed (13: Permission denied)

[johnnyplaydrums]

Hi folks! I'm running netboxcommunity/netbox:v2.10.8 in a Nomad cluster. When I try to create a Device Type with an image attachment, I get a 500. Here are the log lines of interest (I obfuscated the url):

172.31.147.89 - - [15/Apr/2021:16:47:15 +0000] "GET /dcim/device-types/add/ HTTP/1.1" 200 49134 "https://internal.url/dcim/device-types/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
172.31.133.58 - - [15/Apr/2021:16:47:15 +0000] "GET /static/img/netbox.ico HTTP/1.1" 200 1174 "https://internal.url/dcim/device-types/add/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
172.31.126.36 - - [15/Apr/2021:16:47:16 +0000] "GET /api/dcim/manufacturers/?q=&limit=50&offset=0&brief=true HTTP/1.1" 200 193 "https://internal.url/dcim/device-types/add/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
2021/04/15 16:47:24 [alert] 18#24 *11 mkstemp(/opt/unit/tmp//req-XXXXXXXX) failed (13: Permission denied)
172.31.147.89 - - [15/Apr/2021:16:47:24 +0000] "POST /dcim/device-types/add/ HTTP/1.1" 500 54 "https://internal.url/dcim/device-types/add/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
172.31.147.89 - - [15/Apr/2021:16:48:50 +0000] "GET /api/dcim/manufacturers/?q=&limit=50&offset=0&brief=true HTTP/1.1" 200 193 "https://internal.url/dcim/device-types/add/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"

Netbox is erroring when trying to create the temp file: mkstemp(/opt/unit/tmp//req-XXXXXXXX) failed (13: Permission denied). Here are the permissions of /opt/unit/tmp/ as they exist by default:

bash-5.1# ls -al /opt/unit/tmp/
total 12
drwxrwxr-x    2 root     root          4096 Apr  1 06:07 .
drwxrwxr-x    1 root     root          4096 Apr 15 17:00 ..
bash-5.1#

I am able to create the device type if I don't include an image. Interestingly, this issue doesn't happen locally but does happen in the container deployed in Nomad. I can solve this by running chmod a+w /opt/unit/tmp/ in the container, but I need a permanent solution (b/c the chmod change will get wiped out on deploy, or if the container is restarted or changes nodes).

Is there some configuration I should change to fix this? Or any other ideas? Thank you 🙏

[networkhorse]

I am not sure about the solution for docker-compose, however I got this working in my bootc/netbox-chart deployment of Netbox (Kubernetes) by adding the following extraInitContainer:

  - name: opt-unit-directory-bug-workaround
    image: alpine:3
    command:
    - "/bin/sh"
    - "-c"
    - "mkdir -p /opt/unit/tmp/ && chown -R 1000:1000 /opt/unit/"
    volumeMounts:
    - name: optunit
      mountPath: /opt/unit/
    securityContext:
      runAsUser: 0
      runAsGroup: 0
      runAsNonRoot: false

Far from perfect, but it works.

[Random6554]

Its the same issue as defined here but for file path /opt/unit/tmp which is used for external storage options.

The issue is that using the default example in the ReadMe the Entrypoint runs as root where as the other processes are running as user unit(101).

This is what it should look like:

PID   USER     TIME  COMMAND
    1 101       0:00 {unitd} unit: main v1.22.0 [unitd --no-daemon --control unix:/opt/unit/unit.sock --pid /opt/unit/unit.pid --log /dev/stdout --state /opt/unit/state/ --tmp /opt/unit/tmp/]
    9 101       0:00 /bin/sh
   28 101       0:00 {unitd} unit: controller
   29 101       0:00 {unitd} unit: router
   33 101       0:02 {unitd} unit: "netbox" application

/opt/unit # chown -R 100:101 /opt/unit/tmp resolves the issue but is more of a work around.

I have added USER unit to the end of my Dockerfile as I re-build anyway but ideally this would be handled in the upstream Dockerfile.

This can also be fixed by using docker-compose to set the user to unit

[johnnyplaydrums]

Thanks for chiming in with some suggestions! For any Nomad users that come across this, I was able to "solve" this by running chmod when the container starts up. I did this by adding this template stanza to the nomad job file for netbox:

      template {
        data = <<EOH
#!/usr/bin/env bash
echo 'Running custom startup script...'
chmod a+w /opt/unit/tmp/
exec /opt/netbox/launch-netbox.sh
EOH
        destination = "local/custom-startup.sh"
        perms = 100
      }

and then in the task > config stanza, I changed the startup command to command = "/custom-startup.sh". This runs the custom script on startup and then runs the Netbox CMD exec /opt/netbox/launch-netbox.sh. This does mean that if we bump up the container version and the Netbox CMD has changed, we need to update the last line of that script to the new CMD. I've added that warning as a comment in our Nomad file and recommend you do the same if you use this solution.

If any Nomad users have a cleaner way to handle this, please reach out!