Code

Support FP8-E5M2 KV Cache#2279

Code

Merged

zhuohan123 merged 43 commits intovllm-project:mainvllm-project/vllm:mainfrom zhaoyang-star:fp8_cachezhaoyang-star/vllm:fp8_cacheCopy head branch name to clipboardJan 29, 2024

+912-196

Commits

Commits on Dec 27, 2023

• test_cache.py passed

  

  zhaoyang

  committed
  0ac4ba1

  Copy full SHA for 0ac4ba1

• format

  

  zhaoyang-star

  committed
  63ec85b

  Copy full SHA for 63ec85b

• fix compiling warning

  

  zhaoyang-star

  committed
  f98c816

  Copy full SHA for f98c816

Commits on Dec 28, 2023

• fix fp8x4 -> float4

  

  zhaoyang-star

  committed
  9881221

  Copy full SHA for 9881221

Commits on Dec 29, 2023

• test_attention pass

  

  zhaoyang-star

  committed
  850137b

  Copy full SHA for 850137b

• fix typo

  

  zhaoyang-star

  committed
  a852f54

  Copy full SHA for a852f54

• fix typo

  

  zhaoyang-star

  committed
  7be2ed4

  Copy full SHA for 7be2ed4

• add latency & throughput benchmark

  

  zhaoyang-star

  committed
  4bcf15c

  Copy full SHA for 4bcf15c

• fix benchmark_latency.py

  

  zhaoyang-star

  committed
  1a13c5a

  Copy full SHA for 1a13c5a

Commits on Jan 3, 2024

• Merge branch 'main' into fp8_cache

  

  zhaoyang-star

  committed
  5cdb619

  Copy full SHA for 5cdb619

• fix copy_blocks

  

  zhaoyang-star

  committed
  82516df

  Copy full SHA for 82516df

• use VLLM_DISPATCH_CASE_FLOATING_BYTE_TYPES

  

  zhaoyang-star

  committed
  556e5b2

  Copy full SHA for 556e5b2

Commits on Jan 12, 2024

• add default behavior in description

  

  zhaoyang-star

  committed
  c67277b

  Copy full SHA for c67277b

Commits on Jan 13, 2024

• grace code regarding to comments

  

  zhaoyang-star

  committed
  c3760f8

  Copy full SHA for c3760f8

• add namespace fp8_e5m2_unscaled

  

  zhaoyang-star

  committed
  7bae850

  Copy full SHA for 7bae850

Commits on Jan 15, 2024

• change interface

  

  zhaoyang-star

  committed
  525003b

  Copy full SHA for 525003b

• solve conflict

  

  zhaoyang-star

  committed
  537b5a7

  Copy full SHA for 537b5a7

• fix tp

  

  zhaoyang-star

  committed
  7e837dd

  Copy full SHA for 7e837dd

• print log.info

  

  zhaoyang-star

  committed
  fe5f053

  Copy full SHA for fe5f053

Commits on Jan 16, 2024

• add log and raise error on amd gpu

  

  zhaoyang-star

  committed
  58d9817

  Copy full SHA for 58d9817

Commits on Jan 18, 2024

• fix none error

  

  zhaoyang-star

  committed
  dddd6eb

  Copy full SHA for dddd6eb

• do not use VLLM_LDG

  

  zhaoyang-star

  committed
  a61d828

  Copy full SHA for a61d828

Commits on Jan 19, 2024

• fix mirror typo

  

  zhaoyang-star

  committed
  1cb7af6

  Copy full SHA for 1cb7af6

• fix compiler error on lower cc

  

  zhaoyang-star

  committed
  589297a

  Copy full SHA for 589297a

• fix unittest error

  

  zhaoyang-star

  committed
  6223984

  Copy full SHA for 6223984

• fix utest

  

  zhaoyang-star

  committed
  4f85f9b

  Copy full SHA for 4f85f9b

Commits on Jan 20, 2024

• Merge branch 'main' into fp8_cache

  

  zhaoyang-star

  committed
  0ff1d14

  Copy full SHA for 0ff1d14

Commits on Jan 21, 2024

• fix tp error

  

  zhaoyang-star

  committed
  b4db831

  Copy full SHA for b4db831

Commits on Jan 24, 2024

• more clear, tp error

  

  zhaoyang-star

  committed
  3072560

  Copy full SHA for 3072560

• fix conflict

  

  zhaoyang-star

  committed
  d837bbb

  Copy full SHA for d837bbb

• avoid compile fp8 when cuda version is lower than 11.8

  

  zhaoyang-star

  committed
  b493300

  Copy full SHA for b493300

Commits on Jan 25, 2024

• fix ut

  

  zhaoyang-star

  committed
  5461bd6

  Copy full SHA for 5461bd6

• fix typo

  

  zhaoyang-star

  committed
  f66fb4e

  Copy full SHA for f66fb4e

• loose NUM_BLOCKS in test_attention

  

  zhaoyang-star

  committed
  7e5d61b

  Copy full SHA for 7e5d61b

• update test_cache

  

  zhaoyang-star

  committed
  b4aedf5

  Copy full SHA for b4aedf5

• fix yapf

  

  zhaoyang-star

  committed
  eac2720

  Copy full SHA for eac2720

Commits on Jan 26, 2024

• loose num_blocks

  

  zhaoyang-star

  committed
  bb6cc13

  Copy full SHA for bb6cc13

• fix stuck when tp>1

  

  zhaoyang-star

  committed
  455d0b5

  Copy full SHA for 455d0b5

Commits on Jan 27, 2024

• update

  

  zhaoyang-star

  committed
  dbd464c

  Copy full SHA for dbd464c

• rename: create_kv_caches -> create_kv_caches_with_random

  

  zhaoyang-star

  committed
  11411e1

  Copy full SHA for 11411e1

Commits on Jan 28, 2024

• remove cache_dtype_str

  

  zhaoyang-star

  committed
  4945577

  Copy full SHA for 4945577

• add co-author

  Show description for b52e702
  

  zhaoyang-star

  and

  zhuohan123

  committed
  b52e702

  Copy full SHA for b52e702

• Merge branch 'main' into fp8_cache

  zhuohan123

  committed
  fee9a13

  Copy full SHA for fee9a13