Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support in-situ conversion of GPTQ model to marlin format (naive GPTQ kernel also supported). #92

Merged
merged 27 commits into from
Oct 21, 2024
Merged

Support in-situ conversion of GPTQ model to marlin format (naive GPTQ kernel also supported). #92

merged 27 commits into from
Oct 21, 2024

Conversation

guoqingbao
Copy link
Collaborator

For quantized 4-bit GPTQ model:

cargo run --release -- --port 2000 --weight-path /home/mistral_7b-int4/ mistral --quant marlin

It performs in-situ conversion of GPTQ model to marlin format during model loading.

Please note:

Marlin format in-situ conversion only support 4-bit GPTQ (with sym=True, groupsize=128 or -1, desc_act=False).

@guoqingbao
Support in-situ quantization
899515a
@guoqingbao
Typo fix
6e791f5
@guoqingbao
Cargo fmt
504398d
@guoqingbao
Optimize quantized matmul in batch processing & update Q4K results
a3e1fc4
@guoqingbao
Merge branch 'master' into develop
7309f55
@guoqingbao
Fix bug for non-stream response
80f56ae
@guoqingbao
Ask users to provide huggingface token if no token cached and passed …
bd476d3 
…to the program.
@guoqingbao
No crash when both hidden_act and hidden_activation are set for gemma…
afb50f3 
… model
@guoqingbao
Print the number of decoded tokens for each request
616ffc6
@guoqingbao
Merge branch 'master' into develop
573a61a
@guoqingbao
Restore previous bug fix
360a227
@guoqingbao
Support softcapping (Gemma-2 models)
a33884f
@guoqingbao
Merge branch 'master' into develop
f3b1a7d
@guoqingbao
Update lib.rs
761067e
@guoqingbao
Fix Gemma-2 multiple eos/bos ids
ff84499
@guoqingbao
Custom benchmark with parameters
2c81291
@guoqingbao
Mention arguments for benchmark.py
221eace
@guoqingbao
Tweak
08f9491
@guoqingbao
Support GPTQ/Marlin format quantization (4bit weight, f16 input)
e23d8ae
@guoqingbao
Merge branch 'master' into develop
d4239ef
@guoqingbao
Support bf16 inputs for GPTQ/Marlin format quantization
d40c2b0
@guoqingbao
Merge branch 'develop' into develop
2d5b452
@guoqingbao
Merge remote-tracking branch 'eric/master' into develop
b7c6cd1
@guoqingbao
Merge remote-tracking branch master into develop
4b2fe7d
@guoqingbao
Add an example for marlin format conversion & update results
6078814
@guoqingbao
Typo fix
08b5e8a
@guoqingbao
Support in-situ conversion of GPTQ model to marlin format (naive GPTQ…
30a5040 
… kernel also supported).
@guoqingbao guoqingbao merged commit 3ebf3b8 into master Oct 21, 2024
6 checks passed
guoqingbao added a commit that referenced this pull request Dec 26, 2024
@guoqingbao
Squashed commit of the following:
786f140 
commit 8059379
Author: Guoqing Bao <topon@outlook.com>
Date:   Thu Nov 21 18:07:34 2024 +0800

    Update batched results

commit 4cfe1ff
Author: Guoqing Bao <topon@outlook.com>
Date:   Thu Nov 21 17:54:12 2024 +0800

    Batch sampling for argmax strategy under no repetition penalty

commit dfce2ea
Author: Guoqing Bao <topon@outlook.com>
Date:   Thu Nov 21 15:35:22 2024 +0800

    Optimize token tensor padding & tweak weight path

commit 3ebf3b8
Author: Guoqing Bao <topon@outlook.com>
Date:   Mon Oct 21 18:24:45 2024 +0800

    Support in-situ conversion of GPTQ model to marlin format (naive GPTQ kernel also supported). (#92)

    Support in-situ conversion of GPTQ model to marlin format (naive GPTQ kernel also supported).

commit 46b10ad
Author: Guoqing Bao <topon@outlook.com>
Date:   Wed Oct 16 17:48:54 2024 +0800

    Add an example for marlin format conversion & update results (#91)

    Add an example for marlin format conversion & update results
guoqingbao added a commit that referenced this pull request Dec 26, 2024
@guoqingbao
Squashed commit of the following:
02e426f 
commit 8059379
Author: Guoqing Bao <topon@outlook.com>
Date:   Thu Nov 21 18:07:34 2024 +0800

    Update batched results

commit 4cfe1ff
Author: Guoqing Bao <topon@outlook.com>
Date:   Thu Nov 21 17:54:12 2024 +0800

    Batch sampling for argmax strategy under no repetition penalty

commit dfce2ea
Author: Guoqing Bao <topon@outlook.com>
Date:   Thu Nov 21 15:35:22 2024 +0800

    Optimize token tensor padding & tweak weight path

commit 3ebf3b8
Author: Guoqing Bao <topon@outlook.com>
Date:   Mon Oct 21 18:24:45 2024 +0800

    Support in-situ conversion of GPTQ model to marlin format (naive GPTQ kernel also supported). (#92)

    Support in-situ conversion of GPTQ model to marlin format (naive GPTQ kernel also supported).

commit 46b10ad
Author: Guoqing Bao <topon@outlook.com>
Date:   Wed Oct 16 17:48:54 2024 +0800

    Add an example for marlin format conversion & update results (#91)

    Add an example for marlin format conversion & update results
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@guoqingbao