Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ORPO Trainer + support HF metrics directly from chunked loss functions + fixes to avoid torch compile recompilations #429

Merged
merged 12 commits into from
Dec 6, 2024

Conversation

shivam15s
Copy link
Collaborator

@shivam15s shivam15s commented Dec 6, 2024

Summary

This PR adds support for the following:

  1. LigerORPOTrainer: a wrapper on top of HuggingFace ORPO Trainer to use LigerORPOLoss module.
  2. We also provide an example for using LigerORPOTrainer in examples/alignment/run_orpo.py
  3. Change FusedLinearPreference base class' forward function to return additional metrics to align our implementation with HF ORPO Trainer
  4. Additional refactor to avoid torch compile recompilations -- accumulate_chunk function now calls accumulate_helper which is torch compiled solely and input_chunk/target_chunk/target dimension 1 (seq len) is explicitly marked as dynamic to avoid recompilations

Testing Done

  • Hardware Type:
  • run make test to ensure correctness
  • run make checkstyle to ensure code style
  • run make test-convergence to ensure convergence

@shivam15s shivam15s force-pushed the shisahni/orpo_integrate branch from 984c5b5 to 3f378d1 Compare December 6, 2024 01:35
Copy link
Collaborator

@ByronHsu ByronHsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome work!!

@ByronHsu ByronHsu merged commit 6cb0018 into main Dec 6, 2024
3 checks passed
@ByronHsu ByronHsu deleted the shisahni/orpo_integrate branch December 6, 2024 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants