Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Refactor & Add Quant Integration #10

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open

[WIP] Refactor & Add Quant Integration #10

wants to merge 19 commits into from

Conversation

shadowpa0327
Copy link
Owner

@shadowpa0327 shadowpa0327 commented Nov 8, 2024 •
edited
Loading

[WIP] What this PR do?

  1. Refactor code structure
  • Move kernels & customized attention module implementation from kernels/ into palu
  1. Add cuda implementation for quantization integration
  2. Add unit tests for each kernel & customized attention modules.
  3. Add quantization cache object

Todos

  • Tests for quantization integration
  • Re-test decomposition & searching to ensure functionality after refactor
  • Modify triton kernels to support recomputation with arbitrary sequence length

@shadowpa0327
refactor module structure
70a77d2
@shadowpa0327
add palu's backend (customized kernel wrapper), and quantization util…
a26414f 
…s for KV-Cache managements
@shadowpa0327
[NFC] code cleanup
cd46da9
@shadowpa0327
add cuda kernel impl for quantization integration
64d5e03
@shadowpa0327
remove duplicated code
55269a3
@shadowpa0327
code cleanup
e72bf05
@shadowpa0327
refactor directory structure. Move csrc into palu/
2d2fa08
@shadowpa0327
add test for recompute & quant kernels
a00462a
@shadowpa0327
refactor palu_attention_module & add test
c82eb05
@shadowpa0327
support arbitary seq_len of recompute kernel
49944d3
@shadowpa0327
fix bug in kernels
f24ddef
@shadowpa0327
remove redundant print
050b321
@shadowpa0327
fix typo
addd454
@shadowpa0327
fix bug when their is no residual
0a00a1c
@shadowpa0327
code cleanup and fix typo
9bf2f77
@shadowpa0327
add testcase in bgemm, for testing correctness for matrix shape happe…
06e1d41 
…n during prefilling
@shadowpa0327
add test for palu_attention with value 4-bit quant
83501b2
@shadowpa0327
code cleanup
31c70c9
@shadowpa0327
update test
4cfca28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@shadowpa0327