Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Openvino model executor #1

Closed
wants to merge 107 commits into from
Closed

Openvino model executor #1

wants to merge 107 commits into from

Conversation

ilya-lavrenov
Copy link
Owner

@ilya-lavrenov ilya-lavrenov commented Mar 12, 2024 •
edited
Loading

Install:

# Note: install openvino with pytorch_module_extension first!
cd vllm
export PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu"
VLLM_OPENVINO=1 pip install -e .

Run sample via vLLM modeling:

python3 examples/offline_inference.py

Run via optimum modeling:

VLLM_OPENVINO_OPTIMUM=1 python3 examples/offline_inference.py

zhuohan123 and others added 28 commits March 5, 2024 06:27
@zhuohan123
Add distributed model executor abstraction
d8c0998
@zhuohan123
fix
16de289
@zhuohan123
fix
15a1fe7
@zhuohan123
Merge branch 'main' into add-executor-abstraction
ac2e888
@zhuohan123
format
675190d
@zhuohan123
health check
2592130
@zhuohan123
pull out common functionalities and fix tests
e381ca3
@zhuohan123
Merge branch 'main' into add-executor-abstraction
3bdda0b
@zhuohan123
fix lora test
c348371
@zhuohan123
Fix style
002c67f
@zhuohan123
fix review comments
198e794
@zhuohan123
rename
f82841b
@zhuohan123
Merge branch 'main' into add-executor-abstraction
390dbaf
@zhuohan123
Add base class
ebcd813
@zhuohan123
refactor async executors
fe2ef93
@zhuohan123
fix async style
89e0cac
@zhuohan123
Merge branch 'add-executor-abstraction' of https://github.com/vllm-pr…
22ee8ca 
…oject/vllm into add-executor-abstraction
@zhuohan123
lazy import
1c77da8
@zhuohan123
Merge branch 'main' into add-executor-abstraction
4b4206d
@ilya-lavrenov
Generic changes ported from 1028
f4f0162
@ilya-lavrenov
Added OpenVINO backend
1ee42ca
@ilya-lavrenov
Added OpenVINO worker
261f23d
@ilya-lavrenov
Reverted some changes
9d0a984
@ilya-lavrenov
Added OpenVINO cache engine
42833a9
@ilya-lavrenov
Merge remote-tracking branch 'vllm/main' into openvino-model-executor
30345f0
@ilya-lavrenov
Restore cuda memory profiling
d946943
@ilya-lavrenov
Compare with HuggingFace
894bae9
@ilya-lavrenov
Moved patching to get_model, removed custom InputMetadata
228e3c0
@ilya-lavrenov ilya-lavrenov force-pushed the openvino-model-executor branch from 735fcf4 to 228cbf7 Compare March 12, 2024 20:49
ilya-lavrenov and others added 28 commits April 4, 2024 13:16
@ilya-lavrenov
Merge pull request #23 from slyalin/paged_attention_in_openvino
dbed638 
Use PagedAttentionExtension from OV without contrib dependency
@slyalin
Disable weight compression on optimum-intel path if model is being co…
0acb46c 
…nverted on-the-fly (not from IR).
@ilya-lavrenov
Disable NPU merged to OV master recently
90b8aca
@ilya-lavrenov
Merge pull request #26 from ilya-lavrenov/disable-npu
818e384 
Disable NPU merged to OV master recently
@mzegla
Merge branch 'openvino-model-executor' into produce_artifacts
22d0f2b
@mzegla
revert contrib building
26f5b28
@mzegla
add bitsandbytes to requirements, use fixed vllm version in the client
601115c
@mzegla
relax openvino requirement to support latest master
2547b12
@ilya-lavrenov
Merge pull request #27 from mzegla/missing_req
948137a 
Add bitsandbytes to requirements and use fixed vllm version in the client
@slyalin
Enable int8 weight compression via env var
0bb4a52
@slyalin
Merge remote-tracking branch 'lavrenov/openvino-model-executor' into …
786c6e5 
…disable_int8
@ilya-lavrenov
Update optimum-intel
ea934ee
@ilya-lavrenov
Merge pull request #29 from ilya-lavrenov/update-optimum
f73cfd2 
Update optimum-intel
@slyalin
Describe weights comression option in the documentation
02a108a
@ilya-lavrenov
Merge pull request #25 from slyalin/disable_int8
f35263f 
Disable weight compression on optimum-intel conversion path
@ilya-lavrenov
Merge pull request #20 from mzegla/produce_artifacts
3570043 
Produce artifacts for bare metal installation in Dockerfile.openvino
@luo-cheng2021
int8 kvcache support
f2f839a
@luo-cheng2021
use 'VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8' instead of 'kv_cache_dt…
378de66 
…ype=u8' to enable u8 kvcache
@luo-cheng2021
add description for 'VLLM_OPENVINO_CPU_KV_CACHE_PRECISION'
14ea134
@ilya-lavrenov
Merge pull request #28 from luo-cheng2021/luocheng/pa-kv-u8
307a6d1 
[CPU] PagedAttention support u8 kvcache
@slyalin
Merge branch 'openvino-model-executor' into transformers_4_39
fc6302a
@ilya-lavrenov
Revert "Produce artifacts for bare metal installation in Dockerfile.o…
30ea687 
…penvino"
@ilya-lavrenov
Merge pull request #30 from ilya-lavrenov/revert-20-produce_artifacts
388450f 
Revert "Produce artifacts for bare metal installation in Dockerfile.openvino"
@ilya-lavrenov
Merge pull request #12 from slyalin/transformers_4_39
d848897 
Transformers 4.39
@slyalin
Enabled int8 weights by default for performnace benchmarking purposes
4931727
@ilya-lavrenov
Merge pull request #31 from slyalin/int8_enabled_by_default
469a4d0 
Enabled int8 weights by default
@luo-cheng2021
comment for u8 kvcache layout
560c2ce
@ilya-lavrenov
Merge pull request #32 from luo-cheng2021/luocheng/pa-kv-u8-desc
2e5648a 
[CPU] Add comment for u8 kvcache layout
@ilya-lavrenov ilya-lavrenov deleted the openvino-model-executor branch October 9, 2024 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ilya-lavrenov @zhuohan123 @slyalin @luo-cheng2021 @mzegla