Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LLaMa3.1 chat completion #67

Merged
merged 2 commits into from
Jul 26, 2024
Merged

LLaMa3.1 chat completion #67

merged 2 commits into from
Jul 26, 2024

Conversation

guoqingbao
Copy link
Collaborator

Support LLaMa3.1 chat completion by:

  1. Add supports for LLaMa3.1 model config.
  2. Add supports for LLaMa3.1 chat template.

Tested case:

cargo run --release -- --port 2000 --weight-path /home/Meta-Llama-3.1-8B-Instruct/ llama3 --repeat-last-n 64

Give 65 tokens/s on A100 (BF16).

@guoqingbao guoqingbao merged commit e922750 into master Jul 26, 2024
4 of 5 checks passed
@guoqingbao guoqingbao mentioned this pull request Jul 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@guoqingbao