[Bug]: crash on Triton backends when start vllm 0.6.1

[yejingfu]

Your current environment

Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.12.6 (main, Sep 10 2024, 00:05:17) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 4090

Nvidia driver version: 550.54.14
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] numpy==1.26.4
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] triton==3.0.0
[conda] Could not collect

Model Input Dumps

No response

🐛 Describe the bug

Traceback (most recent call last):
File "", line 189, in _run_module_as_main
File "", line 112, in _get_module_details
File "/usr/local/lib/python3.12/dist-packages/vllm/init.py", line 3, in
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 11, in
from vllm.config import (CacheConfig, ConfigFormat, DecodingConfig,
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 12, in
from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/init.py", line 3, in
from vllm.model_executor.sampling_metadata import (SamplingMetadata,
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/sampling_metadata.py", line 11, in
from vllm.triton_utils.sample import get_num_triton_sampler_splits
File "/usr/local/lib/python3.12/dist-packages/vllm/triton_utils/init.py", line 7, in
from vllm.triton_utils.custom_cache_manager import (
File "/usr/local/lib/python3.12/dist-packages/vllm/triton_utils/custom_cache_manager.py", line 3, in
from triton.runtime.cache import (FileCacheManager, default_cache_dir,
File "/usr/local/lib/python3.12/dist-packages/triton/init.py", line 8, in
from .runtime import (
File "/usr/local/lib/python3.12/dist-packages/triton/runtime/init.py", line 1, in
from .autotuner import (Autotuner, Config, Heuristics, autotune, heuristics)
File "/usr/local/lib/python3.12/dist-packages/triton/runtime/autotuner.py", line 9, in
from ..testing import do_bench, do_bench_cudagraph
File "/usr/local/lib/python3.12/dist-packages/triton/testing.py", line 7, in
from . import language as tl
File "/usr/local/lib/python3.12/dist-packages/triton/language/init.py", line 4, in
from . import math
File "/usr/local/lib/python3.12/dist-packages/triton/language/math.py", line 1, in
from . import core
File "/usr/local/lib/python3.12/dist-packages/triton/language/core.py", line 10, in
from ..runtime.jit import jit
File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 12, in
from ..runtime.driver import driver
File "/usr/local/lib/python3.12/dist-packages/triton/runtime/driver.py", line 1, in
from ..backends import backends
File "/usr/local/lib/python3.12/dist-packages/triton/backends/init.py", line 50, in
backends = _discover_backends()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/triton/backends/init.py", line 44, in _discover_backends
driver = _load_module(name, os.path.join(root, name, 'driver.py'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/triton/backends/init.py", line 12, in _load_module
spec.loader.exec_module(module)
File "/usr/local/lib/python3.12/dist-packages/triton/backends/nvidia/driver.py", line 7, in
from triton.runtime.build import _build
File "/usr/local/lib/python3.12/dist-packages/triton/runtime/build.py", line 8, in
import setuptools
File "/usr/lib/python3/dist-packages/setuptools/init.py", line 5, in
import distutils.core
ModuleNotFoundError: No module named 'distutils'

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[yejingfu]

It seems related to the docker image build script. I got this crash issue in my manually built docker image from Dockerfile in source code. But it works when I try the official prebuilt docker image: vllm/vllm-openai:v0.6.1.post2

[ywang96]

cc @simon-mo ^ seems to be the same error we ran into previously.

[simon-mo]

pip install -U setuptools worked for me

[yejingfu]

pip install -U setuptools works.
It's better to update the Dockerfile.

[ywang96]

Closing per answer from @simon-mo

[hmellor]

I'm reopening this as I've had the same error when building the Docker image from source with v0.6.1.post2.

As @yejingfu says, this needs to be fixed in the Dockerfile.

[hmellor]

It looks like this might have been fixed in #8510, but it does mean that the Docker image for v0.6.1.post2 cannot be built without modifying the Dockerfile and requirements.

Can we expect a v0.6.1.post3 release or shall we wait until v0.6.2?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!