Immich container seems to restart after upgrading to v1.121.0

[batcsg1]

The bug

Whenever I turn my PC on, all my Docker containers startup normally as they should. But my Immich docker container restarts every 5 or so seconds according to Docker Desktop

And after 20 seconds, the status of my Immich container shows as 3/4 running on Docker Desktop as shown in the screenshot below:

The OS that Immich Server is running on

Windows 11 Pro 24H2

Version of Immich Server

v1.121.0

Version of Immich Mobile App

v1.121.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Immich conatiner starts up
2. Immich container goes into an endless restart loop
3. Immich container status in Docker Desktop shows as 3/4 running
   ...

Relevant log output

Initializing Immich v1.121.0
Detected CPU Cores: 8
Starting api worker
Starting microservices worker
(node:7) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:7) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:19) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [NestFactory] Starting Nest application...
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] OpenTelemetryModule dependencies initialized +0ms
[Nest] 7  - 11/27/2024, 6:33:31 AM     LOG [NestFactory] Starting Nest application...
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] OpenTelemetryCoreModule dependencies initialized +0ms
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] ClsModule dependencies initialized +0ms
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +1ms
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 19  - 11/27/2024, 6:33:31 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)...
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)...
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (2)...
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (3)...
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (4)...
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (5)...
error: the database system is starting up
    at Parser.parseErrorMessage (/usr/src/app/node_modules/pg-protocol/dist/parser.js:283:98)
    at Parser.handlePacket (/usr/src/app/node_modules/pg-protocol/dist/parser.js:122:29)
    at Parser.parse (/usr/src/app/node_modules/pg-protocol/dist/parser.js:35:38)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (6)...
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (7)...
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (8)...
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (9)...
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
[Nest] 19  - 11/27/2024, 6:33:31 AM   ERROR [ExceptionHandler] getaddrinfo ENOTFOUND database
Error: getaddrinfo ENOTFOUND database
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
microservices worker exited with code 1
Killing api process
Initializing Immich v1.121.0
Detected CPU Cores: 8
Starting api worker
Starting microservices worker
(node:7) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:19) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:7) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)

Additional information

My Immich container is tunneled through a Cloudflare tunnel that points to my external subdomain I use for my Immich container: immich.batchelornz.com

Also my Immich server lives on a dynamic 4TB hard drive partition that consists of 2 2TB Seagate hard drives. The used space on the server is 216 GB

Status of my docker containers after running "docker ps":

CONTAINER ID   IMAGE                                                COMMAND                  CREATED       STATUS                             PORTS
738dd1d56d68   ghcr.io/immich-app/immich-server:release             "tini -- /bin/bash sâ^`¦"   5 hours ago   Up 57 seconds (health: starting)   0.0.0.0:2283->2283/tcp
27550983d8b6   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     5 hours ago   Up 7 minutes (healthy)             
493c98d7866b   portainer/portainer-ce:2.21.4                        "/portainer"             2 weeks ago   Up 7 minutes                       0.0.0.0:8000->8000/tcp, 0.0.0.0:9443->9443/tcp
3ca590c3d384   cloudflare/cloudflared:latest                        "cloudflared --no-auâ^`¦"   2 weeks ago   Up 7 minutes                        
e2c2e4e38c15   jc21/nginx-proxy-manager:latest                      "/init"                  2 weeks ago   Up 7 minutes                       0.0.0.0:80-81->80-81/tcp, 0.0.0.0:443->443/tcp
0099613b43ef   jc21/mariadb-aria:latest                             "/scripts/run.sh"        2 weeks ago   Up 7 minutes                       3306/tcp
ae6ea4c4e976   redis:6.2-alpine                                     "docker-entrypoint.sâ^`¦"   7 weeks ago   Up 7 minutes (healthy)             6379/tcp
59e1e5f33c04   tensorchord/pgvecto-rs:pg14-v0.2.0                   "docker-entrypoint.sâ^`¦"   7 weeks ago   Restarting (1) 32 seconds ago

[bo0tzz]

Your database container is in a crash loop.

[batcsg1]

Is there anything I can do to resolve that crash loop in my db container.

[bo0tzz]

Start by checking the logs to see why it's happening.

[alextran1502]

How much RAM are you allocating for the server?

[batcsg1]

32GB

[lucianosrp]

I am having the same issue, running on proxmox CT

[batcsg1]

Here is the most recent log file from the postgres database container:

2024-11-27 19:35:15.821 UTC [1] LOG:  starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-11-27 19:35:15.821 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-11-27 19:35:15.821 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-11-27 19:35:15.909 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-11-27 19:35:16.089 UTC [28] LOG:  database system was interrupted while in recovery at 2024-11-27 19:34:22 UTC
2024-11-27 19:35:16.089 UTC [28] HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
[2024-11-27T19:35:16Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17331".
[2024-11-27T19:35:16Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17575".
[2024-11-27T19:35:16Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17575/segments/a7bdf69f-9f85-48da-a288-91891695d2ce".
[2024-11-27T19:35:17Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17331/segments/ca568f38-5612-44ab-9b76-df141e2936bd".
2024-11-27 19:35:19.291 UTC [28] LOG:  database system was not properly shut down; automatic recovery in progress
2024-11-27 19:35:19.366 UTC [28] LOG:  redo starts at 0/E0104368
2024-11-27 19:35:19.465 UTC [28] LOG:  invalid record length at 0/E0191C18: wanted 24, got 0
2024-11-27 19:35:19.465 UTC [28] LOG:  redo done at 0/E0191BE0 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.09 s
2024-11-27 19:35:19.571 UTC [28] FATAL:  could not create file "base/16384/99013": File exists
2024-11-27 19:35:19.587 UTC [1] LOG:  startup process (PID 28) exited with exit code 1
2024-11-27 19:35:19.587 UTC [1] LOG:  aborting startup due to startup process failure
2024-11-27 19:35:19.589 UTC [1] LOG:  database system is shut down

[batcsg1]

And in Docker desktop it seems like the "immich_postgres" container seems to be the one continuously restarting on closer inspection:

[howimboe]

Hi there,

same behavior here : following the update, everything was fine. At some point, I had to restart my machine, and after that, immich_server never came back (exact same log as you). In the postgres container, only one strange recent log:

FATAL: role "${DB_USERNAME}" does not exist

everything else seems fine (so not same log as you on the database, no corruption detected). Note : i'm complete newbie on docker/immich/postgres, but more than happy to search/help/learn/contribute :)

so far my only step was to do a database backup, but probably too late :'(

[batcsg1]

Yeah I ran docker compose down to stop and delete the containers, and then ran docker compose up -d to start the containers again in detached mode. And yet the restarting of the PostgreSQL database still persists.

[bo0tzz]

@lucianosrp @howimboe you have different issues, please open your own threads.

@batcsg1 the logs from the database indicate that your database is corrupted. You'll want to figure out what went wrong (power loss? hardware problems? disk corruption?) and restore from a backup.

[batcsg1]

I managed to find a backup in the backups folder in the library directory:

[howimboe]

@bo0tzz : noted, will try.

Just to share : i did go back to v1.20.2 and redeploy => working fine. Then back to v1.21.0 => fine also. Maybe a mistake on my side, i'll keep an eye, and definitly open a ticket/discussion if I see more things.

@batcsg1 many thanks for sharing the backup folder, could be useful one day !