Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Fast inference with vLLM (Llama 2 13B)" example is broken #463

Closed
alex000kim opened this issue Oct 11, 2023 · 0 comments · Fixed by #465
Closed

"Fast inference with vLLM (Llama 2 13B)" example is broken #463

alex000kim opened this issue Oct 11, 2023 · 0 comments · Fixed by #465
Assignees
@irfansharif

Comments

@alex000kim
Copy link

As per https://modal.com/docs/guide/ex/vllm_inference, I ran:

git clone https://github.com/modal-labs/modal-examples
cd modal-examples
modal run 06_gpu_and_ml/vllm_inference.py

Here's the error that I got:

Downloading ray-2.7.1-cp38-cp38-manylinux2014_x86_64.whl (62.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.5/62.5 MB 178.5 MB/s eta 0:00:00
Downloading transformers-4.34.0-py3-none-any.whl (7.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.7/7.7 MB 163.4 MB/s eta 0:00:00
Downloading xformers-0.0.22-cp38-cp38-manylinux2014_x86_64.whl (211.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.6/211.6 MB 165.8 MB/s eta 0:00:00
Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.2/307.2 kB 204.4 MB/s eta 0:00:00
Downloading uvicorn-0.23.2-py3-none-any.whl (59 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.5/59.5 kB 177.2 MB/s eta 0:00:00
Downloading safetensors-0.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 197.8 MB/s eta 0:00:00
Downloading tokenizers-0.14.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 191.2 MB/s eta 0:00:00
Downloading huggingface_hub-0.17.3-py3-none-any.whl (295 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 295.0/295.0 kB 199.8 MB/s eta 0:00:00
Building wheels for collected packages: vllm
  Building wheel for vllm (pyproject.toml): started
  Building wheel for vllm (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [140 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-38
      creating build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/sequence.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/block.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/outputs.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/sampling_params.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/logger.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/utils.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/config.py -> build/lib.linux-x86_64-cpython-38/vllm
      creating build/lib.linux-x86_64-cpython-38/vllm/transformers_utils
      copying vllm/transformers_utils/tokenizer.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils
      copying vllm/transformers_utils/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils
      copying vllm/transformers_utils/config.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/input_metadata.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/weight_utils.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/model_loader.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/utils.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      creating build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/arg_utils.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/llm_engine.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/async_llm_engine.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/ray_utils.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      creating build/lib.linux-x86_64-cpython-38/vllm/worker
      copying vllm/worker/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/worker
      copying vllm/worker/worker.py -> build/lib.linux-x86_64-cpython-38/vllm/worker
      copying vllm/worker/cache_engine.py -> build/lib.linux-x86_64-cpython-38/vllm/worker
      creating build/lib.linux-x86_64-cpython-38/vllm/core
      copying vllm/core/scheduler.py -> build/lib.linux-x86_64-cpython-38/vllm/core
      copying vllm/core/policy.py -> build/lib.linux-x86_64-cpython-38/vllm/core
      copying vllm/core/block_manager.py -> build/lib.linux-x86_64-cpython-38/vllm/core
      copying vllm/core/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/core
      creating build/lib.linux-x86_64-cpython-38/vllm/entrypoints
      copying vllm/entrypoints/api_server.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints
      copying vllm/entrypoints/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints
      copying vllm/entrypoints/llm.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints
      creating build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/mpt.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/baichuan.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/qwen.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/falcon.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/attention.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/sampler.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/activation.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/layernorm.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils
      copying vllm/model_executor/parallel_utils/parallel_state.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils
      copying vllm/model_executor/parallel_utils/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/mpt.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/opt.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/internlm.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/gpt2.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/baichuan.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/gpt_j.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/gpt_neox.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/llama.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/qwen.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/bloom.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/falcon.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/gpt_bigcode.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/random.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/mappings.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/layers.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/utils.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      creating build/lib.linux-x86_64-cpython-38/vllm/entrypoints/openai
      copying vllm/entrypoints/openai/protocol.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints/openai
      copying vllm/entrypoints/openai/api_server.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints/openai
      copying vllm/entrypoints/openai/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints/openai
      running build_ext
      /tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: numpy.core.multiarray failed to import (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
      Traceback (most recent call last):
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 434, in build_wheel
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 145, in <module>
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 364, in run
          self.run_command("build")
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 88, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 525, in build_extensions
          _check_cuda_version(compiler_name, compiler_version)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 413, in _check_cuda_version
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (11.8) mismatches the version that was used to compile
      PyTorch (12.1). Please make sure to use the same CUDA versions.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for vllm
Failed to build vllm
ERROR: Could not build wheels for vllm, which is required to install pyproject.toml-based projects
Terminating task due to error: failed to run builder command "python -m pip install typing-extensions==4.5.0 'vllm @ git+https://github.com/vllm-project/vllm.git@805de738f618f8b47ab0d450423d23db1e636fa2' "

Caused by:
    container exit status: 1
Runner failed with exception: task exited with failure, status = exit status: 1
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/alexkim/mambaforge/bin/modal:8 in <module>                                                │
│                                                                                                  │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/__main__.py:9 in main               │
│                                                                                                  │
│    8 │   setup_rich_traceback()                                                                  │
│ ❱  9 │   entrypoint_cli()                                                                        │
│   10                                                                                             │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:1157 in __call__            │
│                                                                                                  │
│   1156 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1157 │   │   return self.main(*args, **kwargs)                                                 │
│   1158                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/typer/core.py:778 in main                 │
│                                                                                                  │
│   777 │   ) -> Any:                                                                              │
│ ❱ 778 │   │   return _main(                                                                      │
│   779 │   │   │   self,                                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/typer/core.py:216 in _main                │
│                                                                                                  │
│   215 │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                       │
│ ❱ 216 │   │   │   │   rv = self.invoke(ctx)                                                      │
│   217 │   │   │   │   if not standalone_mode:                                                    │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:1688 in invoke              │
│                                                                                                  │
│   1687 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1689                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:1688 in invoke              │
│                                                                                                  │
│   1687 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1689                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:1434 in invoke              │
│                                                                                                  │
│   1433 │   │   if self.callback is not None:                                                     │
│ ❱ 1434 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1435                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:783 in invoke               │
│                                                                                                  │
│    782 │   │   │   with ctx:                                                                     │
│ ❱  783 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    784                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/decorators.py:33 in new_func        │
│                                                                                                  │
│    32 │   def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R":                            │
│ ❱  33 │   │   return f(get_current_context(), *args, **kwargs)                                   │
│    34                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/cli/run.py:145 in f                 │
│                                                                                                  │
│   144 │   │                                                                                      │
│ ❱ 145 │   │   with run_stub(                                                                     │
│   146 │   │   │   stub,                                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/synchronicity/synchronizer.py:497 in      │
│ proxy_method                                                                                     │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/synchronicity/combined_types.py:26 in     │
│ __call__                                                                                         │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/contextlib.py:199 in __aenter__                         │
│                                                                                                  │
│   198 │   │   try:                                                                               │
│ ❱ 199 │   │   │   return await anext(self.gen)                                                   │
│   200 │   │   except StopAsyncIteration:                                                         │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/runner.py:88 in _run_stub           │
│                                                                                                  │
│    87 │   │   │   # Create all members                                                           │
│ ❱  88 │   │   │   await app._create_all_objects(                                                 │
│    89 │   │   │   │   stub._blueprint, post_init_state, environment_name, shell=shell, output_   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/app.py:103 in _create_all_objects   │
│                                                                                                  │
│   102 │   │   │   │   existing_object_id = tag_to_object_id.get(tag)                             │
│ ❱ 103 │   │   │   │   await resolver.load(obj, existing_object_id)                               │
│   104 │   │   │   │   self._tag_to_object_id[tag] = obj.object_id                                │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:126 in load            │
│                                                                                                  │
│   125 │   │                                                                                      │
│ ❱ 126 │   │   return await cached_future                                                         │
│   127                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:102 in loader          │
│                                                                                                  │
│   101 │   │   │   async def loader():                                                            │
│ ❱ 102 │   │   │   │   await obj._load(obj, self, existing_object_id)                             │
│   103 │   │   │   │   if existing_object_id is not None and obj.object_id != existing_object_i   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/image.py:176 in _load               │
│                                                                                                  │
│    175 │   │   │   for image in base_images.values():                                            │
│ ❱  176 │   │   │   │   base_image_ids.append((await resolver.load(image)).object_id)             │
│    177 │   │   │   base_images_pb2s = [                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:126 in load            │
│                                                                                                  │
│   125 │   │                                                                                      │
│ ❱ 126 │   │   return await cached_future                                                         │
│   127                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:102 in loader          │
│                                                                                                  │
│   101 │   │   │   async def loader():                                                            │
│ ❱ 102 │   │   │   │   await obj._load(obj, self, existing_object_id)                             │
│   103 │   │   │   │   if existing_object_id is not None and obj.object_id != existing_object_i   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/image.py:176 in _load               │
│                                                                                                  │
│    175 │   │   │   for image in base_images.values():                                            │
│ ❱  176 │   │   │   │   base_image_ids.append((await resolver.load(image)).object_id)             │
│    177 │   │   │   base_images_pb2s = [                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:126 in load            │
│                                                                                                  │
│   125 │   │                                                                                      │
│ ❱ 126 │   │   return await cached_future                                                         │
│   127                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:102 in loader          │
│                                                                                                  │
│   101 │   │   │   async def loader():                                                            │
│ ❱ 102 │   │   │   │   await obj._load(obj, self, existing_object_id)                             │
│   103 │   │   │   │   if existing_object_id is not None and obj.object_id != existing_object_i   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/image.py:176 in _load               │
│                                                                                                  │
│    175 │   │   │   for image in base_images.values():                                            │
│ ❱  176 │   │   │   │   base_image_ids.append((await resolver.load(image)).object_id)             │
│    177 │   │   │   base_images_pb2s = [                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:126 in load            │
│                                                                                                  │
│   125 │   │                                                                                      │
│ ❱ 126 │   │   return await cached_future                                                         │
│   127                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:102 in loader          │
│                                                                                                  │
│   101 │   │   │   async def loader():                                                            │
│ ❱ 102 │   │   │   │   await obj._load(obj, self, existing_object_id)                             │
│   103 │   │   │   │   if existing_object_id is not None and obj.object_id != existing_object_i   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/image.py:296 in _load               │
│                                                                                                  │
│    295 │   │   │   if result.status == api_pb2.GenericResult.GENERIC_STATUS_FAILURE:             │
│ ❱  296 │   │   │   │   raise RemoteError(f"Image build for {image_id} failed with the exception  │
│    297 │   │   │   elif result.status == api_pb2.GenericResult.GENERIC_STATUS_TERMINATED:        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RemoteError: Image build for im-bvJc9XyO2U9rSetK1p4yUT failed with the exception:
task exited with failure, status = exit status: 1
@irfansharif irfansharif self-assigned this Oct 12, 2023
irfansharif added a commit that referenced this issue Oct 13, 2023
@irfansharif
Fix vLLM example
10e445a 
Fixes #463. Pytorch 2.1.0 (https://github.com/pytorch/pytorch/releases/tag/v2.1.0)
was just released just last week, and it's built using CUDA 12.1. The
image we're using uses CUDA 11.8, as recommended by vLLM. Previously
vLLM specified a dependency on torch>=2.0.0, and picked up this 2.1.0
version. That was pinned back to 2.0.1 in
vllm-project/vllm#1290. When picking up that SHA
however, we ran into what vllm-project/vllm#1239
fixes. So for now point to temporary fork with that fix.
@irfansharif irfansharif mentioned this issue Oct 13, 2023
Merged
irfansharif added a commit that referenced this issue Oct 14, 2023
@irfansharif
Fix vLLM example (#465)
b0f9499 
Fixes #463. Pytorch 2.1.0 (https://github.com/pytorch/pytorch/releases/tag/v2.1.0)
was just released just last week, and it's built using CUDA 12.1. The
image we're using uses CUDA 11.8, as recommended by vLLM. Previously
vLLM specified a dependency on torch>=2.0.0, and picked up this 2.1.0
version. That was pinned back to 2.0.1 in
vllm-project/vllm#1290. When picking up that SHA
however, we ran into what vllm-project/vllm#1239
fixes. So for now point to temporary fork with that fix.
gongy pushed a commit that referenced this issue Jan 5, 2024
@irfansharif @gongy
Fix vLLM example (#465)
97b45fa 
Fixes #463. Pytorch 2.1.0 (https://github.com/pytorch/pytorch/releases/tag/v2.1.0)
was just released just last week, and it's built using CUDA 12.1. The
image we're using uses CUDA 11.8, as recommended by vLLM. Previously
vLLM specified a dependency on torch>=2.0.0, and picked up this 2.1.0
version. That was pinned back to 2.0.1 in
vllm-project/vllm#1290. When picking up that SHA
however, we ran into what vllm-project/vllm#1239
fixes. So for now point to temporary fork with that fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix vLLM example modal-labs/modal-examples
2 participants
@irfansharif @alex000kim