Assets not being tagged via PUT /api/tags/:id/assets

[benjamonnguyen]

The bug

Uploading assets and then trying to tag uploaded assets via the immich API is flaky.

Seeing this error when tagging fails: [Nest] 7 - 10/21/2024, 4:44:32 AM WARN [Microservices:Error: File not found - /usr/src/app/upload/upload/c9def4a9-d01b-4409-859f-749d5d98052f/e9/15/e915d50b-7b86-492a-9e49-2b1b17f0ac94.jpg.

The OS that Immich Server is running on

Fedora

Version of Immich Server

v1.118.2

Version of Immich Mobile App

v1.118.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${IMMICH_DIR}:/mnt/media:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always

  backup:
    container_name: immich_db_dumper
    image: docker.io/prodrigestivill/postgres-backup-local:14
    profiles:
      - backup
    env_file:
      - .env
    environment:
      POSTGRES_HOST: database
      POSTGRES_CLUSTER: 'TRUE'
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      SCHEDULE: "@daily"
      POSTGRES_EXTRA_OPTS: '--clean --if-exists'
      BACKUP_NUM_KEEP: 28
      BACKUP_DIR: /db_dumps
    volumes:
      - ${IMMICH_DIR}/db_dumps:/db_dumps
    depends_on:
      - database

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  model-cache:
  pgdata:

Your .env content

IMMICH_DIR=/mnt/lilnas/Docker/Immich

IMMICH_VERSION=v1.118.2
UPLOAD_LOCATION=$IMMICH_DIR/uploads
DB_PASSWORD=xxxx
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Upload images via immich-go
2. Call immich server API: PUT /tags/:id/assets (url, request bodies, and responses are in the "Additional information" panel)

Relevant log output

[Nest] 7  - 10/21/2024, 4:44:31 AM    WARN [Microservices:MetadataService] Asset 8ce9ec74-b471-4088-ac23-e332e22df754 has no time zone information
[Nest] 7  - 10/21/2024, 4:44:32 AM    WARN [Microservices:MetadataService] Asset d9c3d551-b15f-4961-b345-73a1b518caa7 has no time zone information
[Nest] 7  - 10/21/2024, 4:44:32 AM    WARN [Microservices:MetadataService] Asset d9c3d551-b15f-4961-b345-73a1b518caa7 has no time zone information
[Nest] 7  - 10/21/2024, 4:44:32 AM    WARN [Microservices:MetadataService] Asset 8ce9ec74-b471-4088-ac23-e332e22df754 has no time zone information
[Nest] 7  - 10/21/2024, 4:44:32 AM    WARN [Microservices:MetadataService] Asset 8ce9ec74-b471-4088-ac23-e332e22df754 has no time zone information
[Nest] 7  - 10/21/2024, 4:44:32 AM    WARN [Microservices:Error: File not found - /usr/src/app/upload/upload/c9def4a9-d01b-4409-859f-749d5d98052f/e9/15/e915d50b-7b86-492a-9e49-2b1b17f0ac94.jpg
    at ReadTask._ExifToolTask_parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:76:71)
    at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:47:167)
    at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)] Error reading exif data (upload/upload/c9def4a9-d01b-4409-859f-749d5d98052f/e9/15/e915d50b-7b86-492a-9e49-2b1b17f0ac94.jpg): Error: File not found - /usr/src/app/upload/upload/c9def4a9-d01b-4409-859f-749d5d98052f/e9/15/e915d50b-7b86-492a-9e49-2b1b17f0ac94.jpg
[Nest] 7  - 10/21/2024, 4:44:32 AM     LOG [Microservices:PersonService] Detected 1 new faces in asset 197c4427-b473-4ac1-b1b7-0f95507277b8
[Nest] 7  - 10/21/2024, 4:44:32 AM     LOG [Microservices:PersonService] Detected 1 new faces in asset 8ce9ec74-b471-4088-ac23-e332e22df754
[Nest] 7  - 10/21/2024, 4:44:32 AM    WARN [Microservices:Error: File not found - /usr/src/app/upload/upload/c9def4a9-d01b-4409-859f-749d5d98052f/54/a7/54a7365e-4c4e-42aa-bc87-aec2c7cdc214.xmp
    at ReadTask._ExifToolTask_parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:76:71)
    at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:47:167)
    at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)] Error reading exif data (upload/upload/c9def4a9-d01b-4409-859f-749d5d98052f/54/a7/54a7365e-4c4e-42aa-bc87-aec2c7cdc214.xmp): Error: File not found - /usr/src/app/upload/upload/c9def4a9-d01b-4409-859f-749d5d98052f/54/a7/54a7365e-4c4e-42aa-bc87-aec2c7cdc214.xmp
[Nest] 7  - 10/21/2024, 4:44:32 AM    WARN [Microservices:MetadataService] Asset d9c3d551-b15f-4961-b345-73a1b518caa7 has no time zone information
[Nest] 7  - 10/21/2024, 4:44:32 AM    WARN [Microservices:MetadataService] Asset d9c3d551-b15f-4961-b345-73a1b518caa7 has no valid date, falling back to asset.fileCreatedAt

Additional information

The API response reports all successful but tags are empty

[bo0tzz]

The issue isn't with the tagging api, but rather a file that is missing.
Potentially related to that, it looks like you have your UPLOAD_LOCATION inside of an external library mount. That's likely to cause problems.

[benjamonnguyen]

I'll test again after changing my docker-compose but any chance it's some kind of race condition where the file is being moved out of the upload directory into the library directory while the tagging is happening or perhaps just a stale path being fetched?

Either way I think the response shouldn't return success ideally.

[benjamonnguyen]

Updated docker-compose accordingly:

and still running into the same issue. Please let me know if there are any other configuration changes I should test.

[benjamonnguyen]

Additional tried adding a sleep for 10 seconds which supports the "race condition" theory.

[jrasm91]

Updating tags on the assets triggers a sidecar write job, which persists the tag data to an xmp file. This is most likely happening around the same time as the storage template job. We can definitely work on improving this in the future, but it is a bit of a complicated problem and there are some overarching changes required to provide these types of guarantees moving forward. For now you can avoid this by simply pausing the storage template migration job while you upload.