Immich stops jobs when one single instance of the job fails

[CommanderRedYT]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

I have a huge library of files. With that size, it is of course possible that corrupt files exist.
There are for example videos with missing moov atom or images that are broken for some reason.

The problem now is that when I start the "Missing"-job for transcoding / Machine learning, the job queue gets instantly cleared after the first process dies (for example if ffprobe exits with non-zero code).

Immich should just skip over that file instead. Otherwise, the rest of the library is missing previews / transcodes etc.

The OS that Immich Server is running on

Archlinux

Version of Immich Server

v2.5.6

Version of Immich Mobile App

v2.5.6

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

-#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
      file: hwaccel.transcoding.yml
      service: nvenc # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${EXTERNAL_PATH}:/usr/src/app/external:ro
      - ${EXTERNAL_INSTA360}:/usr/src/app/insta360:ro
      - ${EXTERNAL_FAMILY}:/usr/src/app/family:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false
    user: 1002:1002

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: cuda # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:9@sha256:546304417feac0874c3dd576e0952c6bb8f06bb4093ea0c9ca303c73cf458f63
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      - pgdata:/var/lib/postgresql/data
    shm_size: 128mb
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
EXTERNAL_PATH=<redacted>
EXTERNAL_INSTA360=<redacted>
EXTERNAL_FAMILY=<redacted>


# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Vienna

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=<redacted>

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_HOST=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_PORT=5432

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Upload / have a broken file
2. Start manually processing them via the "Missing" button
3. See error in log
   ...

Relevant log output

immich_machine_learning  | [02/12/26 01:05:11] ERROR    Exception in ASGI application
immich_machine_learning  |
immich_machine_learning  |                              ╭─────── Traceback (most recent call last) ───────╮
immich_machine_learning  |                              │ /usr/src/immich_ml/main.py:186 in predict       │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   183 │   text: str | None = Form(default=None) │
immich_machine_learning  |                              │   184 ) -> Any:                                 │
immich_machine_learning  |                              │   185 │   if image is not None:                 │
immich_machine_learning  |                              │ ❱ 186 │   │   inputs: Image | str = await run(l │
immich_machine_learning  |                              │   187 │   elif text is not None:                │
immich_machine_learning  |                              │   188 │   │   inputs = text                     │
immich_machine_learning  |                              │   189 │   else:                                 │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/main.py:229 in run           │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   226 │   if thread_pool is None:               │
immich_machine_learning  |                              │   227 │   │   return func(*args, **kwargs)      │
immich_machine_learning  |                              │   228 │   partial_func = partial(func, *args, * │
immich_machine_learning  |                              │ ❱ 229 │   return await asyncio.get_running_loop │
immich_machine_learning  |                              │   230                                           │
immich_machine_learning  |                              │   231                                           │
immich_machine_learning  |                              │   232 async def load(model: InferenceModel) ->  │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/local/lib/python3.11/concurrent/futures/th │
immich_machine_learning  |                              │ read.py:58 in run                               │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/main.py:186 in <lambda>      │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   183 │   text: str | None = Form(default=None) │
immich_machine_learning  |                              │   184 ) -> Any:                                 │
immich_machine_learning  |                              │   185 │   if image is not None:                 │
immich_machine_learning  |                              │ ❱ 186 │   │   inputs: Image | str = await run(l │
immich_machine_learning  |                              │   187 │   elif text is not None:                │
immich_machine_learning  |                              │   188 │   │   inputs = text                     │
immich_machine_learning  |                              │   189 │   else:                                 │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/models/transforms.py:53 in   │
immich_machine_learning  |                              │ decode_pil                                      │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   50 def decode_pil(image_bytes: bytes | IO[byt │
immich_machine_learning  |                              │   51 │   if isinstance(image_bytes, Image.Image │
immich_machine_learning  |                              │   52 │   │   return image_bytes                 │
immich_machine_learning  |                              │ ❱ 53 │   image: Image.Image = Image.open(BytesI │
immich_machine_learning  |                              │      bytes) else image_bytes)                   │
immich_machine_learning  |                              │   54 │   image.load()                           │
immich_machine_learning  |                              │   55 │   if not image.mode == "RGB":            │
immich_machine_learning  |                              │   56 │   │   image = image.convert("RGB")       │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /opt/venv/lib/python3.11/site-packages/PIL/Imag │
immich_machine_learning  |                              │ e.py:3498 in open                               │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   3495 │   for message in warning_messages:     │
immich_machine_learning  |                              │   3496 │   │   warnings.warn(message)           │
immich_machine_learning  |                              │   3497 │   msg = "cannot identify image file %r │
immich_machine_learning  |                              │ ❱ 3498 │   raise UnidentifiedImageError(msg)    │
immich_machine_learning  |                              │   3499                                          │
immich_machine_learning  |                              │   3500                                          │
immich_machine_learning  |                              │   3501 #                                        │
immich_machine_learning  |                              ╰─────────────────────────────────────────────────╯
immich_machine_learning  |                              UnidentifiedImageError: cannot identify image file
immich_machine_learning  |                              <_io.BytesIO object at 0x7a6e70b12520>
immich_server            | [Nest] 51  - 02/12/2026, 2:05:11 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed with status 500: Internal Server Error
immich_server            | [Nest] 51  - 02/12/2026, 2:05:11 AM     LOG [Microservices:MachineLearningRepository] Machine learning server became unhealthy (http://immich-machine-learning:3003).
immich_server            | [Nest] 51  - 02/12/2026, 2:05:11 AM   ERROR [Microservices:{"id":"3336dbeb-10ca-4b3e-8aab-5cf5e9ba43be"}] Unable to run job handler (Ocr): Error: Machine learning request '{"ocr":{"detection":{"modelName":"PP-OCRv5_mobile","options":{"minScore":0.5,"maxResolution":736}},"recognition":{"modelName":"PP-OCRv5_mobile","options":{"minScore":0.8}}}}' failed for all URLs
immich_server            | [Nest] 51  - 02/12/2026, 2:15:55 AM   ERROR [Microservices:{"id":"200335ec-aa28-44f2-8bd1-2f15815e40b8"}] Unable to run job handler (AssetEncodeVideo): Error: ffprobe exited with code 1
immich_server            | ffprobe version 7.1.3-Jellyfin Copyright (c) 2007-2025 the FFmpeg developers
immich_server            |   built with gcc 14 (Debian 14.2.0-19)
immich_server            |   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
immich_server            |   libavutil      59. 39.100 / 59. 39.100
immich_server            |   libavcodec     61. 19.101 / 61. 19.101
immich_server            |   libavformat    61.  7.100 / 61.  7.100
immich_server            |   libavdevice    61.  3.100 / 61.  3.100
immich_server            |   libavfilter    10.  4.100 / 10.  4.100
immich_server            |   libswscale      8.  3.100 /  8.  3.100
immich_server            |   libswresample   5.  3.100 /  5.  3.100
immich_server            |   libpostproc    58.  3.100 / 58.  3.100
immich_server            | [mov,mp4,m4a,3gp,3g2,mj2 @ 0x5c5d60970640] moov atom not found
immich_server            | /usr/src/app/upload/library/.../foo.mp4: Invalid data found when processing input
immich_server            |
immich_server            | Error: ffprobe exited with code 1
immich_server            | ffprobe version 7.1.3-Jellyfin Copyright (c) 2007-2025 the FFmpeg developers
immich_server            |   built with gcc 14 (Debian 14.2.0-19)
immich_server            |   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
immich_server            |   libavutil      59. 39.100 / 59. 39.100
immich_server            |   libavcodec     61. 19.101 / 61. 19.101
immich_server            |   libavformat    61.  7.100 / 61.  7.100
immich_server            |   libavdevice    61.  3.100 / 61.  3.100
immich_server            |   libavfilter    10.  4.100 / 10.  4.100
immich_server            |   libswscale      8.  3.100 /  8.  3.100
immich_server            |   libswresample   5.  3.100 /  5.  3.100
immich_server            |   libpostproc    58.  3.100 / 58.  3.100
immich_server            | [mov,mp4,m4a,3gp,3g2,mj2 @ 0x5c5d60970640] moov atom not found
immich_server            | /usr/src/app/upload/library/.../foo.mp4: Invalid data found when processing input
immich_server            |
immich_server            |     at ChildProcess.<anonymous> (/usr/src/app/server/node_modules/.pnpm/fluent-ffmpeg@2.1.3/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
immich_server            |     at ChildProcess.emit (node:events:508:28)
immich_server            |     at ChildProcess._handle.onexit (node:internal/child_process:294:12)

Additional information

No response

[CommanderRedYT]

As a side note, when uploading new healthy media transcoding / ML still works. This only affects the "Missing" buttons

[bo0tzz]

This sounds very odd; there is no mechanism in the code that would do that.

[Arnie97]

I'm still bisecting to understand the exact change that caused the problem, but I'm sure this issue was not introduced until v2.0.

There were always some corrupted files in my large external library; and also some subdirectories that were deliberately set to a restrctive permission e.g. chmod 500 to prevent immich from scanning them. They did not block anything in v1.143.1, but now they block many job queues or even crash the server in v2.5.5.

[jrasm91]

Fixed in #26327