Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: could not broadcast input array from shape (944,) into shape (512,) #9848

Open
1 task done
olegkhr opened this issue Oct 30, 2024 · 2 comments
Open
1 task done

[Bug]: could not broadcast input array from shape (944,) into shape (512,) #9848

olegkhr opened this issue Oct 30, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@olegkhr
Copy link

olegkhr commented Oct 30, 2024

Your current environment

The output of `python collect_env.py` 
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 9.2.0
Clang version: 3.4.2 (tags/RELEASE_34/dot2-final)
CMake version: version 3.24.2
Libc version: glibc-2.17

Python version: 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-3.10.0-957.27.2.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
GPU 4: Tesla V100-SXM2-32GB
GPU 5: Tesla V100-SXM2-32GB
GPU 6: Tesla V100-SXM2-32GB
GPU 7: Tesla V100-SXM2-32GB

Nvidia driver version: 530.30.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                64
On-line CPU(s) list:   0-63
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
Stepping:              7
CPU MHz:               1200.000
CPU max MHz:           2901.0000
CPU min MHz:           1200.0000
BogoMIPS:              5800.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              22528K
NUMA node0 CPU(s):     0-15,32-47
NUMA node1 CPU(s):     16-31,48-63
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.77
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.2
[pip3] triton==3.0.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.6.77                  pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.4.0                    pypi_0    pypi
[conda] torchvision               0.19.0                   pypi_0    pypi
[conda] transformers              4.45.2                   pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.3.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity
GPU0     X      NV1     NV1     NV2     NV2     SYS     SYS     SYS     0-15,32-47      0
GPU1    NV1      X      NV2     NV1     SYS     NV2     SYS     SYS     0-15,32-47      0
GPU2    NV1     NV2      X      NV2     SYS     SYS     NV1     SYS     0-15,32-47      0
GPU3    NV2     NV1     NV2      X      SYS     SYS     SYS     NV1     0-15,32-47      0
GPU4    NV2     SYS     SYS     SYS      X      NV1     NV1     NV2     16-31,48-63     1
GPU5    SYS     NV2     SYS     SYS     NV1      X      NV2     NV1     16-31,48-63     1
GPU6    SYS     SYS     NV1     SYS     NV1     NV2      X      NV2     16-31,48-63     1
GPU7    SYS     SYS     SYS     NV1     NV2     NV1     NV2      X      16-31,48-63     1

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Model Input Dumps

No response

🐛 Describe the bug

vllm serve "meta-llama/Meta-Llama-3.1-70B-Instruct" --port 7000 --max-num-seqs 64 --tensor-parallel-size=8 --max_model_len=32768 --distributed-executor-backend=mp --dtype=half

Log:
ERROR 10-30 11:32:21 engine.py:158] ValueError('could not broadcast input array from shape (944,) into shape (512,)')
ERROR 10-30 11:32:21 engine.py:158] Traceback (most recent call last):
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 156, in start
ERROR 10-30 11:32:21 engine.py:158] self.run_engine_loop()
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 219, in run_engine_loop
ERROR 10-30 11:32:21 engine.py:158] request_outputs = self.engine_step()
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 237, in engine_step
ERROR 10-30 11:32:21 engine.py:158] raise e
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 228, in engine_step
ERROR 10-30 11:32:21 engine.py:158] return self.engine.step()
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1389, in step
ERROR 10-30 11:32:21 engine.py:158] outputs = self.model_executor.execute_model(
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 82, in execute_model
ERROR 10-30 11:32:21 engine.py:158] driver_outputs = self._driver_execute_model(execute_model_req)
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 155, in _driver_execute_model
ERROR 10-30 11:32:21 engine.py:158] return self.driver_worker.execute_model(execute_model_req)
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 303, in execute_model
ERROR 10-30 11:32:21 engine.py:158] inputs = self.prepare_input(execute_model_req)
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 291, in prepare_input
ERROR 10-30 11:32:21 engine.py:158] return self._get_driver_input_and_broadcast(execute_model_req)
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 253, in _get_driver_input_and_broadcast
ERROR 10-30 11:32:21 engine.py:158] self.model_runner.prepare_model_input(
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1586, in prepare_model_input
ERROR 10-30 11:32:21 engine.py:158] model_input = self._prepare_model_input_tensors(
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1196, in _prepare_model_input_tensors
ERROR 10-30 11:32:21 engine.py:158] return builder.build() # type: ignore
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 867, in build
ERROR 10-30 11:32:21 engine.py:158] attn_metadata = self.attn_metadata_builder.build(
ERROR 10-30 11:32:21 engine.py:158] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/attention/backends/utils.py", line 215, in build
ERROR 10-30 11:32:21 engine.py:158] input_block_tables[i, :len(block_table)] = block_table
ERROR 10-30 11:32:21 engine.py:158] ValueError: could not broadcast input array from shape (944,) into shape (512,)
ERROR 10-30 11:32:21 serving_chat.py:603] Error in chat completion stream generator.
ERROR 10-30 11:32:21 serving_chat.py:603] Traceback (most recent call last):
ERROR 10-30 11:32:21 serving_chat.py:603] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 342, in chat_completion_stream_generator
ERROR 10-30 11:32:21 serving_chat.py:603] async for res in result_generator:
ERROR 10-30 11:32:21 serving_chat.py:603] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/utils.py", line 458, in iterate_with_cancellation
ERROR 10-30 11:32:21 serving_chat.py:603] item = await awaits[0]
ERROR 10-30 11:32:21 serving_chat.py:603] File "/home/user/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/multiprocessing/client.py", line 598, in _process_request
ERROR 10-30 11:32:21 serving_chat.py:603] raise request_output
ERROR 10-30 11:32:21 serving_chat.py:603] ValueError: could not broadcast input array from shape (944,) into shape (512,)
ERROR 10-30 11:32:22 multiproc_worker_utils.py:116] Worker VllmWorkerProcess pid 222405 died, exit code: -15
INFO 10-30 11:32:22 multiproc_worker_utils.py:120] Killing local vLLM worker processes
ERROR 10-30 11:32:30 client.py:250] TimeoutError('No heartbeat received from MQLLMEngine')
ERROR 10-30 11:32:30 client.py:250] NoneType: None
CRITICAL 10-30 11:32:31 launcher.py:99] MQLLMEngine is already dead, terminating server process

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@olegkhr olegkhr added the bug Something isn't working label Oct 30, 2024
@QwertyJack
Copy link
Contributor

The same here: ValueError: could not broadcast input array from shape (768,) into shape (512,

$ vllm serve /models/Qwen2.5-72B-Instruct-GPTQ-Int4 --served-model-name qwen2.5-72b --dtype half --kv-cache-dtype fp8

@FuryMartin FuryMartin mentioned this issue Oct 31, 2024
Closed
1 task
@olegkhr
Copy link
Author

olegkhr commented Oct 31, 2024

Workaround from this comment worked in my case:
vllm serve "meta-llama/Meta-Llama-3.1-70B-Instruct" --port 7000 --max-num-seqs 128 --tensor-parallel-size=8 --max_model_len=32768 --max-seq-len-to-capture=32768 --distributed-executor-backend=mp --dtype=half

@tjtanaa tjtanaa mentioned this issue Nov 4, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@QwertyJack @olegkhr