update docs #1602

felipemello1 · 2024-09-16T21:20:27Z

Context

What is the purpose of this PR? Is it to

add a new feature
fix a bug
update tests and/or documentation
other (please add here)

Update:

Recipe overview
LoRA
torchtune.models

Removed unnecessary paragraphs so its more lean

Test plan

Built the docs after the changes and double checked

pytorch-bot · 2024-09-16T21:20:30Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1602

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d5d5986 with merge base 6b55c1d ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

codecov-commenter · 2024-09-16T22:10:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 26.81%. Comparing base (6b55c1d) to head (0423644).

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1602       +/-   ##
===========================================
- Coverage   72.27%   26.81%   -45.46%     
===========================================
  Files         290      290               
  Lines       14552    14570       +18     
===========================================
- Hits        10517     3907     -6610     
- Misses       4035    10663     +6628

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

docs/source/api_ref_models.rst

ebsmothers · 2024-09-17T03:19:01Z

docs/source/api_ref_models.rst


 mistral
 -------

 All models from `Mistral AI family <https://mistral.ai/technology/#models>`_.

-Request Access on `Hugging Face <https://huggingface.co/mistralai/Mistral-7B-v0.3>`__.
+Important: You need to request access on `Hugging Face <https://huggingface.co/mistralai/Mistral-7B-v0.1>`__ to download this model.


Where did we land on this? Why not just use 0.2 if the arch is the same?

i can run it and see if loss goes down, but thats not the most robust test

Eh np, fine to just keep it as is for now

ebsmothers · 2024-09-17T03:19:56Z

docs/source/api_ref_models.rst

-.. autosummary::
-    :toctree: generated/
-    :nosignatures:
+.. .. autosummary::


So these we will just expose later on?

I think it makes sense. Joe was against it in the past, and Philip suggested the same. While we are not ready for MM, it doesnt make much sense to expose it, IMO

ebsmothers · 2024-09-17T03:20:43Z

docs/source/recipes/lora_finetune_single_device.rst

@@ -5,11 +5,10 @@ LoRA Single Device Finetuning
 =============================

 This recipe supports finetuning on next-token prediction tasks using parameter efficient fine-tuning techniques (PEFT)
-such as `LoRA <https://arxiv.org/abs/2106.09685>`_ and `QLoRA <https://arxiv.org/abs/2305.14314>`_.  These techniques
+such as :ref:`glossary_lora` and :ref:`glossary_qlora`. These techniques


nit: why not keep the links to the original papers here?

because we have pages for them, explaining the details, how to use it, the these pages have hte link for the papers.

ebsmothers · 2024-09-17T03:21:54Z

docs/source/recipes/lora_finetune_single_device.rst


 For a deeper understanding of the different levers you can pull when using this recipe,
 see our documentation for the different PEFT training paradigms we support:

 * :ref:`glossary_lora`
 * :ref:`glossary_qlora`

-Many of our other memory optimization features can be used in this recipe, too:


Nit would at least keep this first sentence as a lead-in the the "You can learn more about all of our memory optimization features". Otherwise I agree that it's nice to drop the full list here though

docs/source/recipes/recipes_overview.rst

ebsmothers

Left a bunch of small comments, but no major concerns

Co-authored-by: ebsmothers <ebs@meta.com>

docs/source/recipes/recipes_overview.rst

SalmanMohammadi · 2024-09-17T08:39:20Z

docs/source/recipes/recipes_overview.rst

-using a variety of :ref:`memory optimization features <memory_optimization_overview_label>`. Our  fine-tuning recipes support all of our models and all our dataset types.
-This includes continued pre-training, and various supervised funetuning paradigms, which can be customized through our datasets. Check out our
-:ref:`dataset tutorial <dataset_tutorial_label>` for more information.
+Our recipes include:


Oh Felipe you minimalist

docs/source/recipes/recipes_overview.rst

SalmanMohammadi · 2024-09-17T14:42:17Z

LGTM, some small nits pls address.

Co-authored-by: Salman Mohammadi <salman.mohammadi@outlook.com>

Co-authored-by: ebsmothers <ebs@meta.com>

Co-authored-by: Felipe Mello <felipemello@fb.com> Co-authored-by: ebsmothers <ebs@meta.com> Co-authored-by: Salman Mohammadi <salman.mohammadi@outlook.com>

update recipe overview

29c7668

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 16, 2024

Felipe Mello added 2 commits September 16, 2024 14:21

small str update

44ed73a

uodated lora finetune tutorial

02afc5d

Felipe Mello and others added 5 commits September 16, 2024 17:34

update models doc

ca50a91

405 render

587759b

add param

c303b58

remove clip

df9ef2e

Merge branch 'main' into update_docs

0423644

felipemello1 changed the base branch from main to release/0.3.0 September 17, 2024 00:46

felipemello1 changed the base branch from release/0.3.0 to main September 17, 2024 00:46

smal corrections

4dd400b

felipemello1 changed the title ~~WIP update docs overview~~ update docs Sep 17, 2024

felipemello1 marked this pull request as ready for review September 17, 2024 01:56

ebsmothers reviewed Sep 17, 2024

View reviewed changes

docs/source/api_ref_models.rst Outdated Show resolved Hide resolved

ebsmothers reviewed Sep 17, 2024

View reviewed changes

docs/source/api_ref_models.rst Outdated Show resolved Hide resolved

ebsmothers reviewed Sep 17, 2024

View reviewed changes

docs/source/api_ref_models.rst Outdated Show resolved Hide resolved

ebsmothers reviewed Sep 17, 2024

View reviewed changes

docs/source/recipes/recipes_overview.rst Outdated Show resolved Hide resolved

ebsmothers approved these changes Sep 17, 2024

View reviewed changes

Update docs/source/api_ref_models.rst

941250e

Co-authored-by: ebsmothers <ebs@meta.com>

SalmanMohammadi reviewed Sep 17, 2024

View reviewed changes

docs/source/recipes/recipes_overview.rst Outdated Show resolved Hide resolved

SalmanMohammadi approved these changes Sep 17, 2024

View reviewed changes

felipemello1 and others added 7 commits September 17, 2024 10:50

Update docs/source/recipes/recipes_overview.rst

48e4680

Co-authored-by: Salman Mohammadi <salman.mohammadi@outlook.com>

Update docs/source/recipes/recipes_overview.rst

48abd60

Co-authored-by: Salman Mohammadi <salman.mohammadi@outlook.com>

Update docs/source/recipes/recipes_overview.rst

c7b3984

Co-authored-by: Salman Mohammadi <salman.mohammadi@outlook.com>

Update docs/source/recipes/recipes_overview.rst

536f20f

Co-authored-by: ebsmothers <ebs@meta.com>

Update docs/source/api_ref_models.rst

94e71f8

Co-authored-by: ebsmothers <ebs@meta.com>

Update docs/source/api_ref_models.rst

f4360d0

Co-authored-by: ebsmothers <ebs@meta.com>

add line back

d5d5986

felipemello1 merged commit 74b99fe into pytorch:main Sep 17, 2024
17 checks passed

felipemello1 deleted the update_docs branch September 17, 2024 18:26

ebsmothers added a commit that referenced this pull request Sep 17, 2024

update docs (#1602)

34acc16

Co-authored-by: Felipe Mello <felipemello@fb.com> Co-authored-by: ebsmothers <ebs@meta.com> Co-authored-by: Salman Mohammadi <salman.mohammadi@outlook.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update docs #1602

update docs #1602

felipemello1 commented Sep 16, 2024 •

edited

Loading

pytorch-bot bot commented Sep 16, 2024 •

edited

Loading

codecov-commenter commented Sep 16, 2024 •

edited

Loading

ebsmothers Sep 17, 2024

felipemello1 Sep 17, 2024

ebsmothers Sep 17, 2024

ebsmothers Sep 17, 2024

felipemello1 Sep 17, 2024

ebsmothers Sep 17, 2024

felipemello1 Sep 17, 2024

ebsmothers Sep 17, 2024

ebsmothers left a comment

SalmanMohammadi Sep 17, 2024

SalmanMohammadi commented Sep 17, 2024

update docs #1602

update docs #1602

Conversation

felipemello1 commented Sep 16, 2024 • edited Loading

Context

Test plan

pytorch-bot bot commented Sep 16, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1602

✅ No Failures

codecov-commenter commented Sep 16, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebsmothers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SalmanMohammadi commented Sep 17, 2024

felipemello1 commented Sep 16, 2024 •

edited

Loading

pytorch-bot bot commented Sep 16, 2024 •

edited

Loading

codecov-commenter commented Sep 16, 2024 •

edited

Loading