immich_server docker doesn't start after upgrading containerd.io

[acrobatmaxx]

The bug

First of all, thank you very much for this image, I LOVE Immich and definitely considering to donate to the project.

I have a home server running Ubuntu 22.04.5 with Kernel 6.8, where I run a bunch of Docker containers (Immich is one of them)

Yesterday I did a few system upgrades, and a couple of my docker containers stopped working (Immich is also one of them). After looking at the github Issues thread of one of them, we came in the conclusion that cause was the upgrade of containerd.io. The owner of that specific image told us to add a line in our docker compose, and it totally fixed the issue. The line was specific to the issue with the specific image, so I don't think we can replicate that exact fix to Immich.

Everything was working properly and fine until I did that upgrade yesterday.

My logs and compose are bellow. Thanks for the support!! And all of your work :)

The OS that Immich Server is running on

Ubuntu 22.04.5

Version of Immich Server

v1.121.0

Version of Immich Mobile App

1.121.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#version: '3.8'

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    logging:
      driver: "syslog"
      options:
        tag: "immich-server"
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
#    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /mnt/md0/maxime-data/Images/:/mnt/md0/maxime-data/Images/:ro
      - /mnt/md0/jessica-data/Photos/:/mnt/md0/jessica-data/Photos/:ro
    env_file:
      - .env
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    restart: always

immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: openvino # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    volumes:
      - ./redis-data:/data
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:
  
networks:
  default:
    external: true
    name: npm

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/md0/immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=n&eC5@AuX#ue3A

# Users (PUID) and Group (PGID)
PUID=1002
PGID=1003 

#Time Zone 
TZ=America/Montreal

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

The container simply doesn't start after running "docker compose up -d"

Relevant log output

[...]
Initializing Immich v1.121.0
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
(node:7) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:7) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:17) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
[Nest] 7  - 12/03/2024, 9:15:54 AM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 12/03/2024, 9:15:54 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)...
microservices worker exited with code 1
Killing api process
[...]

Additional information

No response

[bo0tzz]

[Nest] 7 - 12/03/2024, 9:15:54 AM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)...

Implies your database container isn't running, so you'll want to look at what its logs say.

Since you're still using the docker volume for the postgres data, my (unfortunate) guess is that the containerd upgrade corrupted that somehow.

[alextran1502]

If corruption is the case, you can find the database dump in UPLOAD_LOCATION/backups then perform the backup-restore steps would bring you back your instance

[acrobatmaxx]

Sorry for the delay. I updated my docker compose to the latest version. I usually check it before updating, but I must have completely missed the major modifications with the postgresql DB. I was then able to restore a previous backup and now everything is running smoothly. Thanks for the support, even though the issue was on my end! You guys rock!