How to create docker media volume shared with host under production mode?

[xjlin0]

Hi team, thanks for creating Django cookie cutter, such an amazong tool!

My model has FileField for end users to upload stuff. And I also created some custome django-admin commands to import files. Everything works as expected under development mode. But uploaded/imported files end up in the docker instead of host filesystem under production mode.

ubuntu@panel:/$ sudo find . -name ham.jpg
./var/lib/docker/overlay2/8a497f5dac3e3eac557e84b9916df53e84385338ee4e3e09773d9def7db0d2d9/diff/app/attendees/media/private-media/attendee_portrait/ham.jpg
# (not in host filesystem such as /home/adam/public_html/attendees32/attendees/media...)

In this way shipping a new image will remove all uploaded files. So I copied the volume settings from local.yml to production.yml under django service, hoping to seperate files from docker image:

    volumes:
      - .:/app:z

But this break the system, because imported files are under a strange username systemd-resolve which does not have permission, and it cannot create the static folder /app/staticfiles when django start.

$ ls -l /home/adam/public_html/attendees32
-rw-r--r-- 1 systemd-resolve systemd-journal  693904 May  1 16:52 ham.jpg

django     | PermissionError: [Errno 13] Permission denied: '/app/staticfiles'

There must be something I completely messed up, how can I correctly create docker volume so that the files are created by ordinary web username? here is my altered production.yml. Thanks for any advise!

version: '3'

volumes:
  production_postgres_data: {}
  production_postgres_data_backups: {}
#  production_traefik: {}

services:
  django: &django
    build:
      context: .
      dockerfile: ./compose/production/django/Dockerfile
    image: attendees_production_django
    container_name: django
    depends_on:
      - postgres
      - redis
    volumes:
      - .:/app:z
    env_file:
      - ./.envs/.production/.django
      - ./.envs/.production/.postgres
    ports:
      - "8008:5000"
    command: /start

  postgres:
    build:
      context: .
      dockerfile: ./compose/production/postgres/Dockerfile
    image: attendees_production_postgres
    container_name: postgres
    volumes:
      - production_postgres_data:/var/lib/postgresql/data:Z
      - production_postgres_data_backups:/backups:z
    env_file:
      - ./.envs/.production/.postgres

#  traefik:
#    build:
#      context: .
#      dockerfile: ./compose/production/traefik/Dockerfile
#    image: attendees_production_traefik
#    depends_on:
#      - django
#    volumes:
#      - production_traefik:/etc/traefik/acme:z
#    ports:
#      - "0.0.0.0:80:80"
#      - "0.0.0.0:443:443"
#      - "0.0.0.0:5555:5555"

  redis:
    image: redis:6
    container_name: redis

  celeryworker:
    <<: *django
    image: attendees_production_celeryworker
    container_name: celeryworker
    depends_on:
      - redis
      - postgres
    ports: []
    command: /start-celeryworker

  celerybeat:
    <<: *django
    image: attendees_production_celerybeat
    container_name: celerybeat
    depends_on:
      - redis
      - postgres
    ports: []
    command: /start-celerybeat

  flower:
    <<: *django
    image: attendees_production_flower
    container_name: flower
    ports:
      - "5555:5555"
    command: /start-flower

[luzfcb]

@xjlin0 what are the exact options you chose when creating the project?
After creating the project, did you make any changes regarding static files and media files?

[xjlin0]

Hi Fábio thanks for checking, Here are the option based on my memory, since I have multiple cookie cutter repositories.

# ~/.cookiecutter_replay/cookiecutter-django.json
    "timezone": "UTC",
    "windows": "n",
    "use_pycharm": "y",
    "use_docker": "y",
    "postgresql_version": "13.5",
    "cloud_provider": "AWS",   # but I disabled later manually
    "mail_service": "Sendgrid",
    "use_async": "y",
    "use_drf": "y",
    "frontend_pipeline": "None",
    "use_celery": "y",
    "use_mailhog": "y",
    "use_sentry": "n",
    "use_whitenoise": "y",
    "use_heroku": "n",
    "ci_tool": "Github",
    "keep_local_envs_in_vcs": "y",
    "debug": "n",
    "_template": "https://github.com/cookiecutter/cookiecutter-django"

I also disabled traefik, and my apache for proxy hosting is

ProxyPass / http://127.0.0.1:8008/
ProxyPassReverse / http://127.0.0.1:8008/

I did modified a lot of settings, which ones should we check? thanks!

STATIC_ROOT = str(ROOT_DIR / "staticfiles")
STATIC_URL = "/static/"
STATICFILES_DIRS = [
    str(APPS_DIR / "static"),
    str(ROOT_DIR / "libraries"),
]

MEDIA_ROOT = str(APPS_DIR / "media")
MEDIA_URL = "/media/"
PRIVATE_STORAGE_ROOT = str(APPS_DIR / "media/private-media")
PRIVATE_STORAGE_AUTH_FUNCTION = "private_storage.permissions.allow_staff"

[xjlin0]

Just found away to answer my own question.

Since my system will allow users to upload their media files, it need to mount the volume read+write with the correct owner. Previously the user was wrongly as systemd-resolve, which can be corrected by the following user setting in production.yml:

services:
  django: &django
    build:
      context: .
      dockerfile: ./compose/production/django/Dockerfile
    image: attendees_production_django
    container_name: django
    depends_on:
      - postgres
      - redis
    user: "1001"      #### web user id ####
    volumes:
      - .:/app:z

Everything works now as the uploaded media are in the host's volume with the correct owner, rather than in the container land.

[ashermatsh]

how will we get the user id

[xjlin0]

For me, it's the id of the linux user started the server by docker-compose -f production.yml up -d