Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feat] Add window attention for gemma-2 #1056

Merged
merged 4 commits into from
Aug 14, 2024
Merged

[Feat] Add window attention for gemma-2 #1056

merged 4 commits into from
Aug 14, 2024

Conversation

Ying1123
Copy link
Member

@Ying1123 Ying1123 commented Aug 12, 2024 •
edited
Loading

DO NOT turn on auto-merge.
I'll merge it manually.

@Ying1123 Ying1123 changed the title feat: Add window attention for gemma-2 [Feat] Add window attention for gemma-2 Aug 12, 2024
@zhyncs
Copy link
Member

zhyncs commented Aug 12, 2024

@Ying1123 We can temporarily change DEFAULT_MODEL_NAME_FOR_TEST to gemma to trigger the CI test, and then change it back to Llama after verification.

@Ying1123 Ying1123 force-pushed the ying-window branch 3 times, most recently from 7ac08ad to 0ad6781 Compare August 12, 2024 12:38
@zhyncs zhyncs added the enhancement New feature or request label Aug 12, 2024
NihalPotdar
NihalPotdar reviewed Aug 13, 2024
View reviewed changes
Copy link

@NihalPotdar NihalPotdar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried out the engine with the gemma-2-2b, 9b, and 27b models, and looks good. Left some quick comments. Overall lgtm!

python/sglang/srt/layers/radix_attention.py Outdated Show resolved Hide resolved
python/sglang/srt/model_executor/model_runner.py Outdated Show resolved Hide resolved
@Ying1123 Ying1123 force-pushed the ying-window branch 7 times, most recently from 06d3450 to f0f9941 Compare August 13, 2024 22:04
@Ying1123 Ying1123 force-pushed the ying-window branch 2 times, most recently from 224b293 to 749a8ff Compare August 13, 2024 22:41
@Ying1123 Ying1123 merged commit 0909bb0 into main Aug 14, 2024
2 of 6 checks passed
@Ying1123 Ying1123 deleted the ying-window branch August 14, 2024 00:01
merrymercy
merrymercy reviewed Aug 14, 2024
View reviewed changes
python/sglang/srt/server_args.py
self.disable_radix_cache = True
self.disable_regex_jump_forward = True
self.disable_flashinfer = False
self.disable_cuda_graph = True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cuda graph should be turned on

@Ying1123 Ying1123 mentioned this pull request Aug 14, 2024
Open
32 tasks
hnyls2002 pushed a commit that referenced this pull request Aug 14, 2024
@Ying1123 @hnyls2002
[Feat] Add window attention for gemma-2 (#1056)
ceb8a61
@Ying1123 Ying1123 mentioned this pull request Aug 14, 2024
Merged
@Ying1123
Copy link
Member Author

Additional comments addressed in #1090

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@merrymercy merrymercy merrymercy approved these changes

@NihalPotdar NihalPotdar NihalPotdar left review comments

@zhyncs zhyncs zhyncs left review comments

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Ying1123 @zhyncs @merrymercy @NihalPotdar