Error 804: forward compatibility was attempted on non supported HW

[Fmstrat]

Hi, getting this error when running in docker with latest openmmlab/lmdeploy container (sha256:3020bcad368e74a4b7e346c5fc991ed538e0776a30740819ab03cb6e7a55e728).

Other than the newer CUDA in the container (running Debian so 12.2 is latest in repo), everything lines up and should operate fine. Running ollama using the same docker configuration works just fine. I've of course tested the tried-and-true reboot, no luck.

Inside container nvidia-smi:

# nvidia-smi
Sun Jan 12 23:37:44 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.01             Driver Version: 535.216.01   CUDA Version: 12.4     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3070        On  | 00000000:0A:00.0  On |                  N/A |
|  0%   60C    P8              30W / 220W |    739MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

Outside container nvidia-smi (on host):

$ nvidia-smi
Sun Jan 12 18:37:48 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.01             Driver Version: 535.216.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3070        On  | 00000000:0A:00.0  On |                  N/A |
|  0%   61C    P5              36W / 220W |    739MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      3999      G   /usr/lib/xorg/Xorg                          388MiB |
|    0   N/A  N/A      4198      G   /usr/bin/gnome-shell                        121MiB |
|    0   N/A  N/A      4465      G   /usr/bin/nextcloud                            3MiB |
|    0   N/A  N/A     10760      G   ...erProcess --variations-seed-version       73MiB |
|    0   N/A  N/A     11849      G   /usr/lib/firefox-esr/firefox-esr            138MiB |
+---------------------------------------------------------------------------------------

Inside container lm-deploy check_env:

# lmdeploy check_env
/opt/py3/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
sys.platform: linux
Python: 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0]
CUDA available: False
MUSA available: False
numpy_random_seed: 2147483648
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.4.1+cu121
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

TorchVision: 0.19.1+cu121
LMDeploy: 0.6.5+af0fcf2
transformers: 4.47.1
gradio: 5.9.1
fastapi: 0.115.6
pydantic: 2.10.4
triton: 3.0.0
NVIDIA Topology: 
	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-15	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Inside container lmdeploy serve api_server Qwen/Qwen2.5-Coder-1.5B:

# lmdeploy serve api_server Qwen/Qwen2.5-Coder-1.5B
Fetching 10 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 213995.10it/s]
Traceback (most recent call last):
  File "/opt/py3/bin/lmdeploy", line 33, in <module>
    sys.exit(load_entry_point('lmdeploy', 'console_scripts', 'lmdeploy')())
  File "/opt/lmdeploy/lmdeploy/cli/entrypoint.py", line 42, in run
    args.run(args)
  File "/opt/lmdeploy/lmdeploy/cli/serve.py", line 299, in api_server
    else get_max_batch_size(args.device)
  File "/opt/lmdeploy/lmdeploy/utils.py", line 344, in get_max_batch_size
    device_name = torch.cuda.get_device_name(0).lower()
  File "/opt/py3/lib/python3.10/site-packages/torch/cuda/__init__.py", line 435, in get_device_name
    return get_device_properties(device).name
  File "/opt/py3/lib/python3.10/site-packages/torch/cuda/__init__.py", line 465, in get_device_properties
    _lazy_init()  # will define _get_device_properties
  File "/opt/py3/lib/python3.10/site-packages/torch/cuda/__init__.py", line 314, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW

docker-compose entry:

  ai-lmdeploy:
    image: openmmlab/lmdeploy@sha256:3020bcad368e74a4b7e346c5fc991ed538e0776a30740819ab03cb6e7a55e728 # [target: latest]
    container_name: ai-lmdeploy
    volumes:
      - /root/.cache/huggingface:/root/.cache/huggingface
    environment:
      HUGGING_FACE_HUB_TOKEN: <redacted>
    restart: unless-stopped
    ports:
      - 127.0.0.1:23333:23333
    networks:
      - ai
    # command: lmdeploy serve api_server Qwen/Qwen2.5-Coder-1.5B
    command: tail -F /dev/null
    # ipc: host
    # runtime: nvidia
    privileged: true
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

[lvhan028]

The error information indicates the following code suffered in your env.

import torch
device_name = torch.cuda.get_device_name(0).lower()

Please check if torch works in your env.

[Fmstrat]

Sure thing, and thank you for your help. This is using docker run:

$ docker run --runtime nvidia --gpus all --ipc=host --entrypoint sh --rm -ti openmmlab/lmdeploy:latest
# python
Python 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False
>>> 

If I give it full rights with privileged:

$ docker run --runtime nvidia --gpus all --ipc=host --entrypoint sh --rm -ti --privileged openmmlab/lmdeploy:latest
# python
Python 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
/opt/py3/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
False
>>>

To confirm docker was working OK, I built the following Dockerfile:

FROM nvidia/cuda:12.2.2-base-ubuntu22.04
RUN apt update
RUN apt install -y python3 pip
RUN pip3 install torch numpy

And:

$ docker build -t cudatest .
...
$ docker run --runtime nvidia --gpus all --ipc=host --entrypoint sh --rm -ti --privileged cudatest                                                                                                                                                                                                                                                      
# python3
Python 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>>

So everything seems good with Docker.

[Fmstrat]

Ok, I think I'm getting closer now. It looks like lmdeploy is built with CUDA 12.4, which isn't available in all LTS distros drivers. I'm going to see if I can find a driver set for Debian 12 that will be compatible with greater than 12.2.

[Fmstrat]

Well, I got CUDA lined up between them, but now it throws a different error:

# python
Python 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>>
# lmdeploy serve api_server Qwen/Qwen2.5-Coder-1.5B
Fetching 10 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 184771.10it/s]
[WARNING] gemm_config.in is not found; using default GEMM algo                                                                                                                                                                                                                                       
terminate called after throwing an instance of 'std::runtime_error'
  what():  [TM][ERROR] pointer_mapping_ does not have information of ptr at 0x46a6cbc00. Assertion fail: /opt/lmdeploy/src/turbomind/utils/allocator.h:284 

Aborted (core dumped)

[lvhan028]

I cannot reproduce the error. @irexyc, @lzhangzz any idea?

[irexyc]

@Fmstrat

What is the nvidia-cuda-runtime-cu12 version you installed?

You can check it by pip list | grep nvidia-cuda-runtime-cu12. If the version is above 12.2, you can try

pip uninstall nvidia-cuda-runtime-cu12
pip install nvidia-cuda-runtime-cu12==12.2.140

nvidia-cublas-cu12 and nvidia-curand-cu12 may also need to install the 12.2 version. You can search from pypi like https://pypi.org/project/nvidia-cublas-cu12/#history