ColQwen on RunPOD. ValueError: The checkpoint you are trying to load has model type paligemma but Transformers does not recognize this architecture.

[simjak]

System Info

RunPod template

Information

• Docker + cli
• pip + cli
• pip + usage of Python interface

Tasks

• An officially supported CLI command
• My own modifications

Reproduction

Run Serverless template in RunPod https://huggingface.co/vidore/colqwen2-v1.0-merged

2025-01-02T14:07:41.995003415Z ValueError: The checkpoint you are trying to load has model type `paligemma` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
2025-01-02T14:07:52.132952285Z ==========
2025-01-02T14:07:52.132958838Z == CUDA ==
2025-01-02T14:07:52.133032294Z ==========
2025-01-02T14:07:52.136287256Z CUDA Version 12.1.0
2025-01-02T14:07:52.137018935Z Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2025-01-02T14:07:52.137771743Z This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2025-01-02T14:07:52.137774921Z By pulling and using the container, you accept the terms and conditions of this license:
2025-01-02T14:07:52.137777618Z https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2025-01-02T14:07:52.137792833Z A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2025-01-02T14:07:52.145053727Z *************************
2025-01-02T14:07:52.145074990Z ** DEPRECATION NOTICE! **
2025-01-02T14:07:52.145221016Z *************************
2025-01-02T14:07:52.145261791Z THIS IMAGE IS DEPRECATED and is scheduled for DELETION.
2025-01-02T14:07:52.145270512Z     https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/support-policy.md
2025-01-02T14:07:57.182379152Z INFO     2025-01-02 14:07:57,177 datasets INFO: PyTorch version     config.py:58
2025-01-02T14:07:57.182408359Z          2.4.0.dev20240518+cu121 available.
2025-01-02T14:07:57.792832161Z INFO     2025-01-02 14:07:57,790 infinity_emb INFO:           select_model.py:54
2025-01-02T14:07:57.792853568Z          model=`vidore/colpali-v1.2-merged` selected, using
2025-01-02T14:07:57.792858528Z          engine=`torch` and device=`None`
2025-01-02T14:07:57.997832110Z INFO     2025-01-02 14:07:57,996                      SentenceTransformer.py:113
2025-01-02T14:07:57.997862916Z          sentence_transformers.SentenceTransformer
2025-01-02T14:07:57.997866079Z          INFO: Load pretrained SentenceTransformer:
2025-01-02T14:07:57.997867864Z          vidore/colpali-v1.2-merged
2025-01-02T14:07:58.174784825Z WARNING  2025-01-02 14:07:58,172                     SentenceTransformer.py:1192
2025-01-02T14:07:58.174828174Z          sentence_transformers.SentenceTransformer
2025-01-02T14:07:58.174834844Z          WARNING: No sentence-transformers model
2025-01-02T14:07:58.174840934Z          found with name vidore/colpali-v1.2-merged.
2025-01-02T14:07:58.174847664Z          Creating a new one with MEAN pooling.
2025-01-02T14:07:58.175273522Z /usr/local/lib/python3.11/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2025-01-02T14:07:58.175307622Z   warnings.warn(
2025-01-02T14:07:58.341519674Z Traceback (most recent call last):
2025-01-02T14:07:58.341562024Z   File "/usr/local/lib/python3.11/dist-packages/transformers/models/auto/configuration_auto.py", line 945, in from_pretrained
2025-01-02T14:07:58.341580714Z     config_class = CONFIG_MAPPING[config_dict["model_type"]]

[michaelfeil]

Politely closing, as this issue should be opened in the Runpod serverless repo!

infinity can handle this already, but serverless reimplements the api itself and has no support for multimodal!