[Templates] Remove unnecessary requirements and intro flash attn #42169
Conversation
|
I didn't run the lora part. But for 7B, 13B, and 70B full parameter fine-tuning, I can run the template code without these 2 dependencies. I didn't compare the performances, etc. Someone should confirm whether they are needed:
|
|
Thanks. |
ArturNiederfahrenhorst
requested review from
a team,
justinvyu,
matthewdeng and
sofianhnaide
as code owners
There was a problem hiding this comment.
we should pin deepspeed to the right version as well.
| sentencepiece==0.1.99 \ | ||
| "urllib3<1.27" \ | ||
| git+https://github.com/huggingface/transformers.git@d0c1aeb \ | ||
| git+https://github.com/huggingface/peft.git@08368a1fba16de09756f067637ff326c71598fb3 | ||
| pip3 install -U flash-attn --no-build-isolation |
There was a problem hiding this comment.
can we pin a version please?
| peft==0.7.0 | ||
| flash-attn --global-option="--no-build-isolation" |
ArturNiederfahrenhorst
changed the title
Signed-off-by: Artur Niederfahrenhorst <attaismyname@googlemail.com>
ArturNiederfahrenhorst
force-pushed
the
finetuningtemplatedeps
branch
from
908bb26 to
bfaf295
Compare
ArturNiederfahrenhorst
force-pushed
the
finetuningtemplatedeps
branch
from
1ec1f03 to
4868268
Compare
|
I requested review by @justinvyu , who'll review today. |
| @@ -1,4 +1,4 @@ | |||
| deepspeed | |||
There was a problem hiding this comment.
does this template work on 0.12.3? Last time we tried it used to have some problems above 0.10.3. Am I remembering a different context?
There was a problem hiding this comment.
0.12.3 is the one that we install in the ray release byod. So I'd expect it to work.
| @@ -10,6 +10,7 @@ RUN pip install --upgrade pip | |||
| RUN pip install -U -r requirements.txt | |||
| RUN pip install torch==2.1.1 --index-url https://download.pytorch.org/whl/cu121 | |||
| RUN pip uninstall bitsandbytes -y | |||
| RUN pip install flash-attn==2.4.2 --global-option="--no-build-isolation" | |||
There was a problem hiding this comment.
what's the purpose of this --no-build-isolation?
There was a problem hiding this comment.
Honestly, no clue. It's the recommended way of installing flash attention according to their github repo.
ArturNiederfahrenhorst
force-pushed
the
finetuningtemplatedeps
branch
from
a2aa28f to
28f22e9
Compare
|
Release tests succeeded on AWS. |
ArturNiederfahrenhorst
merged commit 53e9b33
into
ray-project:master
Why are these changes needed?
As @scottsun94 noted, the requirements that this PR removes are not needed to execute the llm finetuning template.
This PR also adds flash attention to the template and pins deepspeed.