[Kernel] Dynamic Per-Token Activation Quantization#5037

Merged

WoosukKwon merged 80 commits intovllm-project:mainfrom neuralmagic:dyn-per-token

Jun 7, 2024

+440-76

Commits on Apr 30, 2024

Commits on May 1, 2024

Merge branch 'main' into ds-quant
dsikka
authored
format fix post rebase
dsikka
committed
lazy import CompressedTensorsW8A8StaticTensor (#220 )

varun-sundar-rabindranath
and
Varun Sundar Rabindranath
authored
lazy cutlass_gemm_dq import (#221 )

varun-sundar-rabindranath
and
Varun Sundar Rabindranath
authored
fix asm
Varun Sundar Rabindranath
committed

Commits on May 2, 2024

update shape change
dsikka
committed
add todo
dsikka
committed
Rename quant_per_tensor -> static_scaled_int8_quant
Varun Sundar Rabindranath
committed
Remove cruft
Varun Sundar Rabindranath
committed

Commits on May 14, 2024

Merge branch 'main' into ds-quant
dsikka
authored
fixes : typo
Varun Sundar Rabindranath
committed

Commits on May 15, 2024

py-cutlass temporary hack for num_prompts==1
Varun Sundar Rabindranath
committed
yapf
Varun Sundar Rabindranath
committed

Commits on May 16, 2024

add test_int8_quant
Varun Sundar Rabindranath
committed

Commits on May 17, 2024

call cpp cutlass
Varun Sundar Rabindranath
committed
Merge branch 'main' into ds-quant
dsikka
authored
remove cutlass py interface
Varun Sundar Rabindranath
committed
format.sh
Varun Sundar Rabindranath
committed
remove fake-quant
Varun Sundar Rabindranath
committed
add compressed tensors test
dsikka
committed
remove torch.int8
dsikka
committed
format
dsikka
committed

Commits on May 20, 2024

Commits on May 21, 2024

PR comments
dsikka
committed
Fix scales/zero-points device allocation
Varun Sundar Rabindranath
committed
ruff
Varun Sundar Rabindranath
committed
add better comments
Varun Sundar Rabindranath
committed

Commits on May 22, 2024

add comment
dsikka
committed
Merge branch 'main' into ds-quant
dsikka
authored
clang format
dsikka
committed
clang format again
dsikka
committed
address PR comments
Varun Sundar Rabindranath
committed
clang-format
Varun Sundar Rabindranath
committed

Commits on May 23, 2024

remove layer name
dsikka
committed
remove unused import
dsikka
committed
remove parent name
dsikka
committed
Fix rounding
Varun Sundar Rabindranath
committed
comment
Varun Sundar Rabindranath
committed
cruft
Varun Sundar Rabindranath
committed
yapf
Varun Sundar Rabindranath
committed
remove unquantized check
dsikka
committed

Commits on May 24, 2024

update parsing to use compressed-tensors; add dynamic per token parsing case; make ignore list handling more robust
dsikka
committed
add dynamic quantization arg, fill out create_weights/apply
dsikka
committed
Add quant_per_token kernels

Varun Sundar Rabindranath
authored and
dsikka
committed
make changes to config parsing based on sparseml updates; test dynamic per token model
dsikka
committed
fix shape for cutlass issues
dsikka
committed
remove dicts; use quantization args directly
dsikka
committed
update compressed-tensors; add docstring
dsikka
committed
Dyn per token varun cleanup (#227 )

authored and
dsikka
committed
add test_int8_quant

Varun Sundar Rabindranath
authored and
dsikka
committed
remove fake quant
dsikka
committed
Merge branch 'main' into dyn-per-token
dsikka
authored
format
dsikka
committed
combine static and dynamic quant computation
Varun Sundar Rabindranath
committed
TORCH_CHECK and nits
Varun Sundar Rabindranath
committed
use Union
Varun Sundar Rabindranath
committed
clang-format
Varun Sundar Rabindranath
committed
fix typo
Varun Sundar Rabindranath
committed
isort
Varun Sundar Rabindranath
committed

Commits on May 28, 2024

update test case
dsikka
committed
fix isort
dsikka
committed

Commits on May 29, 2024

store input scales in gpu
Varun Sundar Rabindranath
committed

Commits on Jun 5, 2024

Merge branch 'main' into dyn-per-token
dsikka
authored

Commits on Jun 6, 2024

tensor device location fixes
Varun Sundar Rabindranath
committed
format.sh
Varun Sundar Rabindranath
committed
remove compressed tensors
dsikka
committed
format fix
dsikka
committed
add comments; some clean-up
dsikka
committed
review comments
Varun Sundar Rabindranath
committed
review comments and const correctness
Varun Sundar Rabindranath
committed
format.sh
Varun Sundar Rabindranath
committed

Commits on Jun 7, 2024

nit fixes
dsikka
committed