USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
-
Updated
Dec 17, 2024 - Python
#
deepspeed-ulysses
Here are 2 public repositories matching this topic...
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
pytorch multi-modal gemma pipeline-parallelism transformers-models tensor-parallelism llava llm-training internlm flash-attention zero3 llm-framework sequence-parallelism internlm2 ring-attention deepspeed-ulysses llama3 910b
-
Updated
Dec 20, 2024 - Python
Improve this page
Add a description, image, and links to the deepspeed-ulysses topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the deepspeed-ulysses topic, visit your repo's landing page and select "manage topics."